Building a FreeBSD Build System

Building a FreeBSD Build System

by Bjorn Nelson
04/13/2006

Updating software--many fear this task. It risks interruption of service. It introduces a big, gray unknown. Even with a test system and usage reports, it may bring doom to your applications. With great confidence and trust in his abilities, the system administrator must tread down this uncertain path. Sometimes this requires persuading, as many people don't see the need. Trying to quantify the risk of a compromise is not easy, especially for those who have never experienced an attack. The process of rebuilding software can sometimes be lengthy, and merging configuration files can be tedious. It's a thankless task, and not many users will recognize the effort.

Fortunately, FreeBSD has a pretty good track record of not breaking during updates--or maybe I underestimate my sysadmin skills. Unfortunately, keeping FreeBSD current is a very time-consuming job. I started seeing how daunting this is while watching our 20-plus FreeBSD servers age and fall into legacy mode. I noticed that Red Hat took very little time to stay up-to-date, and I thought there had to be a way to do something similar on FreeBSD.

I set out on a search for what was available. I tried binary updating from bsdupdates.com, which was pretty easy to set up and was great for security updates, but it didn't really help with updating configuration files--and upgrades cost money. I then attempted radmind, but quite honestly, waiting for it to traverse my machine and build an image didn't give me the sense of time savings I wanted.

Eventually I put it on hold for year until I could find an official document or article describing best practices and procedures for updating a bunch of servers. Then, while glancing at the FreeBSD Handbook, I found what I had been looking for: tracking for multiple machines. I read about it and rejoiced. With great guidance and direction, I proceeded to build the ultimate updater: the FreeBSD build system.

The section in the handbook is a great start, but it will leave you with a system that is far from ultimate status. Never fear; when you finish this article, you will have an unbeatable update system. Even mergemaster will work faster. You will have an update system in which a machine update/upgrade will take less than 10 minutes.

To set up a FreeBSD build system, you need three components. A build server is the first requirement. It should be either a fairly beefy uniprocessor or a lesser SMP-based machine. The second component is a staging server, which is basically a test machine where you can test the build without potentially destroying a production box. This doesn't have to be a machine with much fanfare, but it should be as close as possible to the rest of your machines to ensure an accurate test platform. The third component, called the build set, consists of all the clients to which you want to install the updates. These are your production machines.
Build Server

Most of the scripts I've included here have optimizations for parallel processing, so it would be wise to use a build server that supports SMP of some form. The hardware I used is a quad PIII 700MHz with 4GB of RAM (although it would probably run fine on 512MB, 1.7GB inactive and 1.6GB free on normal load). My buildworld/buildkernel commands take not much more than 45 minutes each.
Configure make

One nice aspect to running a source-based operating system is the ability to configure all compiles to follow the same behavior. The BSD make uses a centralized /etc/make.conf file that needs to match on all the machines that will use the build server. For my build server I set:

NO_PROFILE=true
USA_RESIDENT=true

This is a fairly small, centralized make configuration. Most people disable parts of the core system such as X and games. For the most part I do too, but sometimes you will run across a requirement that needs things like X. I haven't run into any harm building more than what the build server requires, and restricting the build set by enabling variables such as NO_X or NO_GAMES on the clients. As always, test on the staging server before installing on the build set.
Synchronize the source

Next, you need a way to synchronize usr/src with the latest code from an official FreeBSD repository on a regular schedule. Many methods of doing this are documented in the handbook. I used cvsup. Set it up by installing it from ports:

% cd /usr/ports/net/cvsup && make install

Then add a command to your /etc/crontab, which will start this process every morning:

# cvsup local sources and ports
0 7 * * * root /usr/local/bin/cvsup -g -L 2 /etc/cvsupfile
>> /var/log/cvsup-client.out

Obviously, the schedule you use is your decision. You may want to slow the interval from daily to weekly or monthly to allow yourself more time for testing and to ensure that you can update all your machines using the same update baseline.

The next step is to build your /etc/cvsupfile. I set my host to localhost because I use a local cvsup-mirror:

*default host=localhost
*default prefix=/usr
*default base=/usr/local/etc/cvsup
*default release=cvstag=RELENG_6_0
*default delete use-rel-suffix

src-all
*default tag=.
ports-all
doc-all

I track the RELENG_6_0 tag, which gives me security updates and bug fixes for 6.0. Then, when a later version comes out, I will switch on my own schedule and start updating all our servers.

Add to your /etc/make.conf:

SUP_UPDATE=true
SUP=/usr/local/bin/cvsup
SUPFLAGS=-g -L 2
SUPFILE=/etc/cvsupfile
PORTSSUPFILE=/etc/cvsupfile
DOCSUPFILE=/etc/cvsupfile

and start a synchronization:

% cd /usr/ports && make update

Export the source

The next step is to allow the clients to access to your sources and built binaries. Although I use NFS here, an interesting alternative for an internet-wide system would be to export over FTP.

Click here to find out more!

For NFS, add to your /etc/exports file:

/usr/src -ro -mapall=nobody
/usr/obj -ro -mapall=nobody
/usr/ports -ro -mapall=nobody

Enable NFS by adding to /etc/rc.conf:

rpcbind_enable="YES"
nfs_server_enable="YES"
mountd_flags="-r"

Then start NFS by running /etc/rc.d/nfsd start.
Build the binaries

Now that your synchronized code is accessible remotely, you can start to build it. Normally, you can just cd to /usr/src and perform the make buildkernel and installkernel, but because you are now building for multiple machines, you may want to make it institute a locking mechanism as well as make it a little more efficient. I am including a shell script called build_all.sh that will disable make in /usr/src while a build is under way. It will also disable make installworld or installkernel outright if there is an error in the build process. For finesse, it even prints out a message when you run make to help you diagnose the problem.

Something else I found that traverses the border of the unsupported is the ability to build multiple make buildkernels simultaneously. While this might provide only a small benefit for uniprocessor machines, it's a huge benefit for a multiprocessor one. The build system will even let you know if an error occurred in building a particular kernel and prevent you from hosing your machine. Nonetheless, always test on your staging server before touching production.

To use these features, copy build_all.sh to your build server and put it in your crontab:

# build sources
0 10 * * * root ~root/bin/build_all.sh 4

The argument you give build_all.sh will be passed through to -j during the buildworld stage.
/usr/ports Power

Although I won't go into much depth on improving the spectacular ports system, there are a couple of tweaks you can now do to make it even more powerful and faster.

The first benefit is that you can keep an internal distfiles mirror. This reduces the time required in installing from ports. I have included another script called fetch_distfiles.sh, which will go through and run make fetch on every port in the system. You may want to modify the SUBPROC and CONCURRENTFETCH variables to suit your hardware, but basically they set a high-water mark for parallel process and parallel fetches you want to run simultaneously. Be forewarned that if you set these variables too high, you will come in to work to a build server running at the speed of hot buttered rum. The best strategy is to start low and turn up from there.

The second benefit is that you can start making local ports. You can keep a meta port that just builds all the fun packages you normally install, or build a port to install that anonymous binary application you have to run on all your servers.

The only change you need to make to a local port is to its makefile to be sure it doesn't get confused:

CATEGORIES=local
VALID_CATEGORIES+= ${CATEGORIES}

This will allow you to keep your port in new category called local.

Next, add the category to /usr/ports/.cvsignore so that it doesn't get accidentally erased on the next cvsup update. I have included a makefile for a local port I made called rcs_mergemaster.
Add the clients

The procedures for adding clients and the staging server are the same. Because the total update procedure takes about 5 minutes, it doesn't hurt to just run a test install on the staging server first, before the day starts.
Mount the remote filesystem

The first step is to mount the filesystems exported by the build server. Add to your rc.conf:

nfs_client_enable="YES"

Then to your /etc/fstab to mount the share automatically on boot:

build-ports:/usr/ports /usr/ports nfs ro,intr,bg 0 0
build-src:/usr/src /usr/src nfs ro,intr,bg 0 0
build-obj:/usr/obj /usr/obj nfs ro,intr,bg 0 0

Using the intr option allows you to recover from a NFS hang by sending an interrupt signal to the process trying to access the remote system. The bg option speeds up and prevents a hang during boot. The ro option is just a safeguard to verify the filesystem mounts read-only.

Next, mount the new filesystem entries in /etc/fstab:

Click here to find out more!

% mount -a

Next, configure your clients' /etc/make.conf, which should match the build server pretty closely. My typical make.conf is:

NO_X=true
WITHOUT_X11=true
NO_GAMES=true
NO_PROFILE=true
USA_RESIDENT=true

Because you won't be able to write to the /usr/ports mount, add a line so that you can still build ports locally:

WRKDIRPREFIX=/usr/work

To configuring which kernel to use, add the line:

KERNCONF=GENERIC

There's a distinction to consider here. You will definitely want to set different values for the KERNCONF variable between your build server and your clients. You must add every kernel to the KERNCONF variable on the build server that you want to be available to your build set. On every client in your build set, you need to set only one kernel in the KERNCONF variable.

If you have portupgrade installed, add the following to /usr/local/etc/pkgtools.conf; otherwise, it will fail:

ENV['PORTS_INDEX'] = '/usr/work'
ENV['PORTS_DBDIR'] = '/usr/work'

RCS and mergemaster

The biggest trick with the biggest time benefit is to configure mergemaster to be a fast process. The maintainer of mergemaster has gone to great lengths to keep it very hands-on, so that new users don't run into trouble or misunderstand what it does while it runs. While this step of the build system removes some of the hands-on requirements, I have built in some safety nets that will keep you from falling too hard in the event of a mistake.

I set the MM_PRE_COMPARE_SCRIPT hook to mergemaster by creating an /etc/mergemaster.rc file:

AUTO_INSTALL=yes
PRESERVE_FILES=yes
PRESERVE_FILES_DIR=/var/tmp/mergemaster/preserved-files-`date +%y%m%d-%H%M%S`
IGNORE_MOTD=yes
MM_PRE_COMPARE_SCRIPT=/usr/src/precompare_mm.sh

mergemaster will automatically find and source this file by default. This rc file tells mergemaster to call a script called /usr/src/precompare_mm.sh. I put this into /usr/src so that any client using the build system can also source it. One problem with this is that I cannot guarantee that the FreeBSD organization won't horribly disfigure or abuse the src folder. It's under its control, so to make this setup more solid, it's best to keep the script in another location and copy it to /usr/src nightly, or make it a local package and install it to all the clients.

The next step is to put your configs under RCS. The first safety net of this script is that if the /etc/RCS directory does not exist, it won't do anything different from what mergemaster already does. Once you create the /etc/RCS directory, you activate it. Be sure to put any and all config files you have modified under RCS with:

% ci -l -t-Import filename

I've modified a few config files under /etc:

crontab
fstab
group
hosts
inetd.conf
make.conf
master.passwd
motd
newsyslog.conf
ntp.conf
ntp.drift
profile
rc.conf
resolv.conf
services
shells
syslog.conf

If you want to keep your SSH host keys, add:

ssh/ssh_host_key
ssh/ssh_host_key.pub
ssh/ssh_host_rsa_key
ssh/ssh_host_rsa_key.pub

mergemaster will update some files under boot, root, var, and of course etc. Test this on your staging server. After running it, you can always recover from the directory you set for the PRESERVE_FILES_DIR in the mergemaster.rc file. (Everything replaced automatically will go into a subfolder called rcs_mergemaster.)

This addition does two things. First, it installs any file that is included with the latest release as long as the file it replaces does not have a corresponding RCS entry. If it does, it will assume you have updated the file and will continue with the standard mergemaster method of showing you the differences and offering to merge. The second addition, or side benefit, is that you now can keep your config files in version control. No longer do you need to keep 100 dated or initialed backups before making a change. Just check it in with a lock, and write a short message about your change when you are done modifying it.

The only problem is you now have to convert all your coworkers to the RCS way. There is a way around that as well. I have included a script called rcs_checkin.sh that you can run at night; it will check in while locking any files that have already been placed under RCS. Just set the directory it should search under for its argument.
Update Procedure

Now it's time to reap the reward of the ultimate build system. What complicated and wizardly commands do you need to know? Just the familiar old procedure (which you should of course test on the staging server first):

% mergemaster -p
% make installkernel
% reboot
% make installworld
% mergemaster
% reboot

You have now completed your first update on the ultimate updater. Keep in mind that you still need to follow the upgrade guidelines. For instance, upgrading from 4.11 to 6.0 requires an intermediate upgrade to 5.3. Also, you may want to look to such projects as clusterssh to perform multiple upgrades/updates at once or cfengine to automate the builds to run.
Bibliography

Meyer, Mike. "Tracking for Multiple Machines," FreeBSD Handbook, 2005 (accessed January 13, 2006).

Thanks goes to the mailing lists of talk.nycbug.org, freebsd-ports.freebsd.org, and freebsd-arch.freebsd.org.

Bjorn Nelson built a FreeBSD build system while working as a system administrator at Baruch College.

Ref :: http://www.onlamp.com/pub/a/bsd/2006/04/13/freebsd-build-system.html
http://www.onlamp.com/pub/a/bsd/2006/04/13/freebsd-build-system.html?page=2
http://www.onlamp.com/pub/a/bsd/2006/04/13/freebsd-build-system.html?page=3

---