I’ve got some server maintenance at midnight tonight. In the mean-time I’m up here at work twiddling my thumbs with three hours to go. Yeah, there’s work things I could be doing, but I’ve already put in more than a full day and I’ve got a cold. :-p
I like Jekyll. Mostly. It’s just that the whole pull, type-out-an-awkward-filename, write some YAML, write some textile, commit, push dance is enough of a speed bump that I don’t post enough. At least that’s my excuse.
So I’m going to try my hand at whipping up a blog. There’s lots of things I could write it in. Node.js is something I’ve been meaning to give a go. ngx_openresty looks really interesting. But for tonight I’ll see what I can accomplish in three hours and stick with Ruby. I could go with Harbor, but I like Sinatra’s router, and I think this is something I can manage in a single “controller” so I’m going to give that a shot. I could persist to PostgreSQL and Sequel which is always a winning combination. But I think I’ll go with Redis this time around. It’s fast, and ought to suit the blog-post format well. At least initially. Not sure of the best fit, but I’ll try a key for each post, and a List to track/retrieve them.
Follow along here, and with any luck I’ll have something in the next few hours. :-)
I’ve got a disused Linode instance already, so if it works out, I’ll deploy, update DNS and we’ll see how it takes.
‘Tis Bash really. Because it’s the default, or easy to get on FreeBSD, OpenBSD, OSX, Ubuntu and Solaris.
So how many lines is it?
18,865
Yes. I wrote this post just for that. Vanity. Sue me. ;-)
It’s a big collection of scripts to automate server configuration. It spans the gamut from compiling NGinx, to configuring LACP on FreeBSD, to generating HAProxy entries, to cloning git repositories and mounting NFS shares. Pretty much anything you’d need to web-servers, load-balance them, setup replicated database servers, etc is in there.
Because they’re complex. :-) Force yourself to learn shell-scripting, and soon enough you can fire off a script like the Github Migration in the previous post in a few minutes, saving days of manual labor when you’re migrating over 100 projects. You might be surprised at just how often you use it. I couldn’t imagine doing half of what I used to do manually or with Ruby now.
I mean, Ruby is great in many many ways. But it’s no substitute for being able to fire off a quick for-loop right where your cursor is now.
So if you decide to tackle one new language this year, and you spend a fair about of time on *nix or *bsd like systems, I heartily recommend Bash.
By the way, I named that system’s administration project after this beer. Gubna. ;-)
There’s probably thousands of ways to do this out there, but here’s mine. A quick little shell script I wrote up to migrate a repository from a local gem-server up to a Github Organization, including your branches and tags. If you give it your login details it’ll create the remote repository if it doesn’t already exist.
Once you tweak the exported variables at the top, you call it like so: moveit my_repository_name
Just a quick note. You might sometimes find yourself wanting to filter a list of files on your system. We’ll use a clone of this blog for an example. You could use grep:
Regexps are obviously very powerful, and I often use something similar in my own scripts.
For simple cases like this however, you might want to get familiar with the built-in Bash wildcards:
Here we’re going to go over the basics of getting the latest SmartOS release (January 12, 2012 as of this post) running. Here’s a screen-cast for the video inclined, but I’ll also provide all the tweaky bits with some more detail below so you can follow along at home.
You’ll need an IntelCPU with the VT-x extensions assuming you want to use KVM virtualization on SmartOS as it doesn’t support AMD CPUs. You’ll get VT-x support on any 2nd Gen i3 processor, the i5 or i7 processors, or a Nehalem or later Xeon. This should cover most any Intel system you might have at your office as long as it’s not more than a couple years old. You can find out for sure on Intel’s Virtualization Technology List page.
If you’re not already familiar with ZFS, I’d suggest downloading the latest FreeBSD, running it under VirtualBox, and playing around with it there. For 99% of what you’ll be doing, aside from standard device-name conventions for disks, it’ll be identical to what you’ll be doing in SmartOS. You might also want to buff up on the Wikipedia article on ZFS.
For everyone else, on with the show!
Single Disk Configurations
First, an issue with SmartOS configuration common to all installations. No swap is created. If you create a zones/swap device, it’ll be used automatically at boot. The device should be a zvol, the same size as your physical memory. So in my case:
The primarycache setting is your ZFS Adaptive Read Cache (in-memory cache). It obviously doesn’t make sense to cache data stored on a swap device since if you had memory to cache it, you’d be better served using that memory for the data directory instead of pushing it out to disk.
The secondarycache is the ZFS L2ARC, ie: a cache device (SSD) attached to the pool. Again, caching swap doesn’t really make much sense, so make sure that it’s not.
I’ve seen it mentioned on IRC that disabling the caches for swap is automatic, but without confirmation, so no harm in making sure.
Multiple Disk Configurations
Now we move onto multi-disk configurations. If this is you, then hold off on adding the swap above. We’ll get around to it in a bit after a few other things. The SmartOS configuration script has a bug that ignores extra disks you pass it. It’ll only use the first.
On our staging system we have a JBOD with 24 drives on it, and 3 drives in the server itself. We want to use that JBOD for our zones zpool. Because you can’t remove a top-level vdev in ZFS, you’re in a bit of a pickle though. If you use one of those disks right off the bat, you won’t be able to get the end result/configuration you want. Unless the disk you use is ultimately intended to be a spare as you can add/remove those at will.
So if you’ve already configured SmartOS, you want a redundant multi-disk configuration (mirrored, raidz, etc) and you don’t have a spare disk, this is probably where you stop and go dig up some old disks you can temporarily use while we shuffle things around.
If you’ve already configured your install, and you chose one of the disks you ultimately want to use in your pool, then you’ll need to delete it, and reconfigure it to use one of your spare disks instead.
The easiest way to blow the pool away so you can get back to the configuration is to run the format command, and blow away the partition on the disk with fdisk.
Making Things Complicated: As an alternative to the fdisk/reboot, you could swap out the disk by adding your spare as a mirror, then removing the original disk like so:
Once that’s done and your running setup is on a spare disk, we can continue.
First up, you can’t configure a dump device on your pool if:
It’s going to include log or cache devices
If it’s going to span multiple raidz vdevs
So you’re going to want to dedicate a raw physical device. As an aside, because of an issue in a critical service (vmadmd) when you have more than one zpool, you can’t put the dump device on a secondary pool. You’re going to want to make sure your system only has the “zones” pool when all is said and done, so that’s why we use a raw device here.
You can’t use this device for anything now else mind you. So in the multi-disk setup you need:
The initial disk you’ll install to (this could be a device you don’t intend to use, or it could ultimately be intended to be add to the pool as a spare)
The disk for the dump device
The disk(s) for your multi-device “zones” zpool
Now we’ll create the multi-device zpool that we’ll eventually turn into our “zones” zpool:
Note, this is a JBOD in my case. Each of these is a 1TB 7200RPM disk, so this will create about 15TB of usable space after subtracting the spares (3TB) and parity (6TB).
It’s really a bit of a silly setup for a production VM host.
If you need that much storage for guest content, it should probably be on NFS or something so you can eventually have redundant, load-balanced guests. If you needed it for performance, you’d use smaller, higher-RPM disks in conjunction with SSDs.
I’m just using this setup here because this is a host for our non-production staging sites. By comparison the production setup consists of twelve 10,000RPM 300GB disks (two of those reserved for spares, and the other ten split into two raidz sets), two mirrored 50GB SSDs for log devices, and one 100GB SSD as a L2ARC cache device. A lot less storage space, but a lot more performance (and guest content is on NFS anyway, this is just for OS installs, swap, logs, etc). When you’re running a lot of VMs, disk IO performance is going to be pretty high up there when it comes to keeping things snappy.
So now we have our “backup” zpool. Let’s replicate our data to it:
Now that our dump device is off of the “zones” pool, and we’ve backed up our install, we can delete it:
Finally, we’ll export our “backup” pool, and reimport it with the “zones” name:
If the disk for the original configure-time “zones” pool was intended as a spare, and you had left it off earlier in the zpool create backup since it was in-use, now is when you’d add it back:
Now we’ll loop back around to the issue we mentioned at the beginning, and create the swap device that the configuration script missed creating for us:
Finally your zpool is how you wanted it. All disks where they belong, and a dedicated dump device humming along. Go ahead and reboot now to ensure everything’s properly mounted, and afterwards ensure that swap -l lists your swap device and dumpadm displays your dedicated device.
If you run a zfs list you’ll probably notice that the backup retained a zones/dump file-system. That’s no longer used since we moved it to a dedicated device, so let’s do a little house-cleaning and delete it:
Ultimately we’ve ended up with something like this:
Now you’re ready to rock, creating your own base image, and provisioning new guests. You’ll find help on that in the SmartOS wiki article: How to create a Virtual Machine in SmartOS.