Neat way to prevent multiple instances of a script

Sometimes you need to ensure that you can never have more than one instance of a script running at the same time. This is especially important with scripts that modifies files, and if the script runs for longer than a fraction of a second it becomes more critical. Also if you have many people administrating a system, it becomes more important to ensure they don't step on each other's toes.

Basically the problem is known as the multiple writers problem, and it is solved by something called "semaphores". "Semaphores" is a feature implemented in the kernel with the purpose of providing a way to guarantee that a piece of code be made "mutually exclusive". I don't want to go into this, it is already properly explained on many websites, but have a look at the Wikipedia article if you are interested in the topic.

One technique to get around the problem of multiple instances of a script is to use the existence of a specific file somewhere to flag other instances of the script that there is already a running instance. The "touch" command does not complain about existing files, so you need to to check first whether the file exists already, exit if it does, and create it otherwise. For example

if [ -f /tmp/already_running ] then echo Can not continue - the lock file already exists! echo If you are sure that no other instance of this script echo is running, delete the file /tmp/already_running and try again. exit 1 else touch /tmp/already_running fi ....

Near the end of this script it is common to delete the file for the next use. This, however, is not the best solution.

If two instances of the script were started at nearly the same instant, what could happen is that instance 1 checks the file, finds it does not exist, but then gets kicked off the CPU so that instance 2 can run. Instance 2 then checks the lock file and also sees it is OK to continue, and then creates the lock. Instance 1 eventually gets CPU time again and, having already previously checked the lock file, believes it is safe to continue running.

This is known as a race condition, and is by definition what Semaphores are meant to prevent. But semaphores are not easily accessible in scripts, or so it might seem.

Now I know you are asking "but what is the chance of such a precice timing of sceduled cpu time to cause this kind of race condition". Yes, the chances are probably low, especially if you are the only person using a specific script. However there is a proper way of checking that we are the only instance running, and it is even simpler to implement thatn the check-file-touch-file method!

The secret lies in the mkdir command.

if ! mkdir /tmp/already_running then echo Can not continue - the lock file already exists! echo If you are sure that no other instance of this script echo is running, delete the file /tmp/already_running and try again. exit 1 fi ....

mkdir will automatically use the kernel built-in semaphores during the actual process of creating the entry in the file system. It will fail if the directory exists, and on succesfull return, the lock will be in place already, so no extra commands are needed to complete the mutex locking process.

The second part of this is to automate the release of the lock when the script exists. Typically you want the lock to be released even if someone kills the script, press Ctrl-C, or if it terminates normally or on an error.

This is done by means of a EXIT trap, and the format of using traps in bourne shell variants is:

trap "do-something-here" EXIT

This trap must be set AFTER obtaining the lock, otherwise a second instance of the script will "inadvertendly" remove the lock obtained by the first instance of the script (because the new instance will basically remove the lock which it was not able to obtain when it exists on not being unable to obtain the lock.

You obviously don't have to have an if-then-fi to print a message to the user - if you are the only person using a script, you can simplify the checking of the lock as follow:

MUTEX_LOCK=/tmp/myscript_already_running mkdir $MUTEX_LOCK || exit 1 trap "rmdir $MUTEX_LOCK" EXIT

With the above you will simply have an error message from mkdir which you need to interpret as "the script is already running", eg:

mkdir: Failed to make directory "/tmp/myscript_already_running"; File exists

Using this technique, a whole script might look like this:

#!/bin/ksh # This is myscript v1.0. #Set up the running environment MUTEX_LOCK=/tmp/myscript_already_running ... if ! mkdir $MUTEX_LOCK then echo Can not continue - the lock file already exists! echo If you are sure that no other instance of this script echo is running, delete the file $MUTEX_LOCK and try again. exit 1 fi trap "rmdir $MUTEX_LOCK" EXIT ... # Doing work which requires only one instance of the script to be running ... # THE END

Note there is no "remove lock" statement at the end of the script. This is handled by the trap, which executes on any exit, except of course a kill -9.

Using a kill -9 should in any case only ever be used as a last resort, because it does not allow the program to clean up after itself.

Some days I'm glad that I'm not a network administrator

Automating the system identification for a Solaris zone to speed up zone deployment

Recapping, I demonstrated how to create a basic Solaris zone from scratch. Then I showed how to use ZFS snapshots to add the ability to “reset” a zone to a clean state, and how to speed up the definition step by exporting a zone configuration file and then using this as a template for defining zones.

This can save a considerable amount of time with complex zones. The other two steps of creating a zone, namely installing it (populating it with files) and setting it up by completing the system identification during the first boot can also be improved one, the first by using the zoneadm “clone” feature, and the second by using a pre-defined sysidcfg file (and maybe a few other tweaks) injected into the zone file system before it is booted the first time.

This blog entry talks about the second of these.

The sysidcfg file is simply a text file with lines specifying the values for the various options. This file is placed in the zone's /etc directory in its root file system, before it is booted. Then during boot-up, the file is read and any specified values prompted, while any missing items will be prompted for as per normal.

The items that can be set are as follow:

Item	Variable Name	Description of Values
Security Policy	security_policy	Kerberos or NONE. If set to “kerberos”, additional properties can be set. If not specified, a value will be prompted.
Name Service	name_service	NIS, NIS+, LDAP, DNS, NONE. Some additional properties are available when using NIS, NIS+, LDAP or DNS. If not specified, you will be prompted for the appropriate value(s).
NFSv4 Domain Name	nfs4_domain	Specify either the keyword “dynamic”, or provide the value to be used for the NFS4 domain name as a Fully Qualified domain name. If not specified, you will be prompted for the appropriate value(s).
Region and Time zone	timezone	Ether give the time zone from /usr/share/lib/zoneinfo/* or else specify a GMT-offset value. If not specified, you will be prompted for this information.
Terminal Type	terminal	Specify the TERM type, eg vt100. If not specified, you will be prompted for this value.
Locale	system_locale	Specify a locale, eg C, such as found from /usr/lib/locale. If not specified, you will be prompted for this value.
Root Password	root_password	The Encrypted root password. To get this, the easiest is to make a dummy user, set its password to what you want, and then copy the encrypted value from the /etc/shadow file. Other options include writing a little perl script or C program to produce the encrypted version of a password. If not specified, you will be prompted during the first boot.
Network Settings	network_interface	Except for the hostname, these are normally obtained from the zone definition. It can be specified here to override those values, but will not be prompted if not specified.

Note: It is entirely possible to use sys-unconfig in a zone. Doing so will have a similar effect to running sys-unconfig on a global zone or normal Solaris system: The zone will halt and on the next boot you will be presented with prompts for the system identification items. Be Aware that sys-unconfig also removes the zone's root key, and a new one will be generated on the first boot after the system was un-configured.

Something else to note is that a zone's “hostname” and “zone name” does not have to be the same. If you do keep it the same, there will be less opportunity for confusion. While the other network settings for a zone is obtained automatically from the zone's definition, the hostname will still be prompted. To eliminate this prompt, include a network settings section in the zone's sysidcfg file.

Some items available in the sysidcfg file for a normal system can not be set during a zone's system identification as it relies on configuring the kernel and a zone does not have its own dedicated kernel. These include items like power management and the Date and Time, including a Time-server.

An example of a basic sysidcfg file might look like this:

bash-3.2# cat sysidcfg nfs4_domain=dynamic security_policy=NONE timezone=Africa/Johannesburg terminal=vt100 system_locale=C name_service=NONE network_interface=PRIMARY {hostname=ziggy.mydomain}

In the above example the keyword PRIMARY is used to automatically select the only interface configured on this zone. This effectively allows for setting the zone name in the sysidcfg template with minimal fuss. It is of course also possible to use the interface name.

If any of the options are omitted from the file, those items will be prompted for in the usual manner. I did not specify the root login password, so that will be the only item which will be prompted for during the boot up process.

To test this, do the following:

1. Define the zone (using zonecfg)

2. Install the zone (using zoneadm -z zonename install

3. Copy the sysidcfg file to the zone's etc file, eg
   cp sysidcfg.template /export/zones/zonename/root/etc/sysidcfg

4. Boot the zone and connect to its console, eg
   zoneadm -z zonename boot; zlogin -C zonename

And voila! Now you can automate the zone definition and the zone's system identification. In the next part I'll show how to speed up the Installation step.

First steps to cloning a Solaris Zone

Today I want to just mention a few concepts that I've been deliberately neglecting - some of the ideas I mentioned in my 15-minute-to-your-first-zone guide and in my post about Automating Zone creation, and that I will still be posting about in the next few posts are based on this.

Firstly Sun has cleverly integrated Zone management with the ZFS file system.

1. If the parent directory of a Zone's root is on a ZFS file system, then zoneadm will create a new ZFS file system for the rootpath of the zone.

2. Stopping and starting the zone will mount and unmount the zone's root file system

3. Cloning a zone by means of the zoneadm utility will automatically use a ZFS snapshot to create a ZFS clone which will be mounted on the rootpath of the new zone.

This is even more interesting because prior to release 11/06 of Solaris 10, running a zone with its rootpath on a ZFS file system was an unsupported configuration.

The second thing is that resources which are only needed while a zone is running, is created and destroyed dynamically when the zone is started or halted. In particular this applies to network interfaces and loopback file systems. When you start up a zone, you will notice new entries were created for its interfaces, and new file systems were mounted. The file systems which are so managed are normally hidden from df in the global zone, but shows up when you run df with the new -Z switch.

The next concept is that of how zlogin works. You can think of zlogin as a kind of "su" command, but in stead of running a commands under a different userid, it runs a command in a different zone. The default command which it runs is a shell. You can also compare is to using ssh or rexec to run a specified command somewhere else, though there is no networking involved.

The concept that every process has got a UID and GID which controls its access to files and system calls in extended in Solaris by a new field storing the process' Zone-ID. This, together with Process Permission flags and a chroot is essentially what zones are, but more on that later.

Using zlogin to run a command or create a new shell in a zone will create a wtmpx login record having zone:global as the origin of the session.

zlogin with the -C option depends on the zone's console device, and creates a wtmpx entry in the zone with the console recorded as the origin of the login session.

A few things which I consider to be good Zone management habbits:

1. Keep an entry for each zone's IP address in the global zone's hosts file, and maintain the /etc/inet/netmasks file with entries for all the subnets you will be using.

2. If you put each zone in its own file system then they can not all "fill up" at the same time. With ZFS file systems, this requires that you set quotas and/or reservations on the zones' root file systems.

Finally a very simple yet important and eventually very powerful concept, namely exporting and importing of zone configurations.

I want to demonstate this using as an example the configuration from an existing zone. First we export it and store it in a text file, like this

globalzone # zonecfg -z myfirstzone export > /tmp/zone_config.txt

Have a look at the file ...

globalzone # cat /tmp/zone_config.txt

create -b

set zonepath=/export/zones/myfirstzone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.131

set physical=e1000g0

end

Now just make a few small changes to the file - specifically we update the zonepath and the IP address

globalzone # sed '

/zonepath/ s/myfirstzone/firstclone/;

/address=/ s/100.131/100.132/

' /tmp/zone_config.txt > /tmp/clone_config.txt

Of course you could use your favourite text editor to do that, but using sed is just so sexy.

globalzone # cat /tmp/clone_config.txt

create -b

set zonepath=/export/home/zones/firstclone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.132

set physical=e1000g0

end

We will feed this config file into zonecfg. Of course you could just manually set each of those entries, but zone configurations can easily get complex - you may have multiple network interfaces, many file systems, and several other non-default settings like resource controls, something I'll get to in due course.

Right now what I have on my system (the "before" picture)

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

Creating the zone config based on this:

globalzone # zonecfg -z firstclone -f /tmp/zone_config.txt

Then the "after" picture showing the new zone configured...

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

- firstclone configured /export/zones/firstclone native shared

All that remains is to populate this new "cloned" zone with user-land bits...

globalzone # timex zoneadm -z firstclone install

A ZFS file system has been created for this zone.

Preparing to install zone <firstclone>.

Creating list of files to copy from the global zone.

Copying <188162> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <1307> packages on the zone.

Initialized <1307> packages on zone.

Zone <firstclone> is initialized.

Installation of <1> packages was skipped.

The file </export/zones/firstclone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

real 28:13.55

user 4:46.13

sys 7:18.22

And we're done!

Over 28 minutes – that is still much to slow. In the next post I will use this concept and add to it to take “cloning” to the next level - It should not take more than a few seconds.

15 minutes to your first Solaris zone

So you got OpenSolaris on your local machine and you would like to try out Zones. This tutorial will give you a 15 minute by-example guide to setting up your first zone, and after that explain a few things hands-on, showing you where what is.

There are a few things to understand about Zones before we start:

First thing is the concept of the Global Zone - the "base" operating system instance is the Global zone and all other "local zones" run under the control of the global zone.

Secondly you get three types of local Zones, being Full root zones, Sparse root zones, and Branded Zones. These differ in what gets copied into the zone's root file system, as well as in how they start up. Branded Zones, which starts up using custom scripts and provides special hooks into the operating system, can emulate a different kernel, but that is a topic for another day.

A Full root zone gets a complete copy of all the Solaris installed package files, which needs about 5 GB of storage. A Sparse zone only gets its own /etc and /dev, which saves on space but it uses a read-only loop-back mounted copy of other directories, such as /opt and /usr. Copying all the files into a Full root zone takes some 10 minutes longer than setting up a sparse zone and uses 5 GB more, but because /usr and /opt is writable it results in a more flexible (read more useful) example environment.

Normally before you set up a zone you would need to do thorough planning, taking care of a things like disk space/file systems, CPU and memory resource allocations, maybe even special boot options. But for a quick start lets just assume defaults for most things.

You do need to work out a few details before you start. 1. Select a name for the zone. This can also be the "hostname" for the virtual OS instance in the zone. 2. Select an IP address and identify on which interface it will live. 3. Identify a spot on your file system where the Zone root will be installed.

Lets call the zone "myfirstzone", let it use IP address 192.168.24.131 on interface e1000g0, and give it a directory /export/zones/myfirstzone

mkdir -m 0700 /export/zones/myfirstzone

Yes, you guessed right, later we can add more zones under /export/zones. Be aware, if you don't restrict the permissions on the directory, the zone installation later on will fail with a security error.

The commands for working with zones are:

zonecfg - Change a zone's configuration / setup. This will create and/or modify the xml base configuration of a zone. zoneadm - Manipulate running zones, eg rebooting and showing their status. zlogin - Connect to a zone (Create a shell session in a zone or connect to the zone's console)

Without further ado, creating the zone using zonecfg zonecfg -z myfirstzone zonecfg:myfirstzone> create zonecfg:myfirstzone> set zonepath=/export/zones/myfirstzone zonecfg:myfirstzone> add net zonecfg:myfirstzone:net> set address=192.168.24.131 zonecfg:myfirstzone:net> set physical=e1000g0 zonecfg:myfirstzone:net> end zonecfg:myfirstzone> commit zonecfg:myfirstzone> exit bash-3.2#

The basic Zone has now been set up. We can view it using this command zonecfg -z myfirstzone export create -b set zonepath=/export/zones/myfirstzone set autoboot=false set ip-type=shared add net set address=192.168.24.131 set physical=e1000g0 end

Now to populate it with Solaris packages/files, very simply use the zoneadm install command. This process takes a good 15 minutes or more, and you should see output like this:

bash-3.2# zoneadm -z myfirstzone install Preparing to install zone . Creating list of files to copy from the global zone. Copying <187209> files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <1282> packages on the zone. Initialized <1282> packages on zone. Zone is initialized. Installation of <1> packages was skipped. The file contains a log of the zone installation.

The install process is highly disk IO intensive. If you're building a zone on the same disk from which the system is running, the resulting disk contention can cause it to run quite a bit longer, up to an hour on systems with slow disks.

Use this time to scan through the man pages for the zone commands. There is also an introductory man page, i.e zones(5).

Also while this install is running use the time to look at the output from these commands: zonecfg -z myfirstzone info

zoneadm list -vc

Once the installation process finishes, you can boot the zone and log in. During the first login, you need to login on the zone console and provide the system identification information prompted.

zoneadm -z myfirstzone boot; zlogin -C myfirstzone [Connected to zone 'myfirstzone' console] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone ... ... ... System identification is completed. rebooting system due to change(s) in /etc/default/init [NOTICE: Zone rebooting] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone Reading ZFS config: done. zoneone console login:

Now log in as root with the password you just set, and run a few commands in the zone; - Do a ps -ef in the zone, and another one in the Global zone, and compare the output. - Also compare the output from a few other commands, particularly df -h, uptime, zonename, hostname, ifconfig, and uptime. On the ifconfig -a command output, note the zone identifier for each interface, as well as the multiple loopback identifiers.

For now I will leave you to marvel at the differences and similarities, but I will point out one process running in the global zone, namely zoneadmd. It manages the zone environment, ensures that its file systems and network interfaces are mounted and plumbed, etc. There will be an instance of it for every running non-Global zone.

Disconnecting from the zone Console requires that you enter the "disconnect" escape sequence, by default ~. (A tilde followed by a full stop) If you're already working remotely using something like SSH, use ~~. - The extra tilde character will inform SSH that it should not itself act on the disconnect command, but rather pass it on down the server. It is also possible to change your escape sequences, but I'll reserve that for a small blog article on another day. Note that disconnecting the console is NOT the same as logging out of the zones - processes like prstat will continue to run on the zone's console and you can reconnect later by again by running "zlogin -C myfirstzone"

Now that the zone is running, lets do a few interesting things with it.

First, in the Global Zone, have another look at what you see from zoneadm list -vc. Other interesting command are "prstat -Z", as well as "df -Z"

Next try out zlogin sans the -C option which would put you on the console. Use exit, logout, or ^D to disconnect.

Create a new user in the zone zlogin myfirstzone mkdir /export/home useradd -d /export/home/joe -m joe passwd joe

Use SSH to login to the zone from another machine, using the zone's IP address. Confirm where you logged in to using hostname, and zonename.

Now lets start an FTP server in the zone: zlogin myfirstzone svcadm enable ftpd

zlogin as used above runs the specified command directly in the zone. You can check the results using by ftp-ing to the zone, though you need to use a user like joe created above, which can login via the network.

Finally, lets add some disk space to our existing zone. This could have been done during the configuration process, but that would have defeated the purpose of showing how to add something to a zone.

I don't have an extra disk device, so I'll use a loop-back (lofi) device. cd /export/zones mkfile 1g fakedev_for_myfirstzone lofiadm -a `pwd`/fakedev_for_myfirstzone newfs /dev/lofi/1 /dev/rlofi/1: Unable to find Media type. Proceeding with system determined parameters. newfs: construct a new file system /dev/rlofi/1: (y/n)? y /dev/rlofi/1: 2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors 1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632, 2083232, 2092832

There are at least 3 ways in which this file system can be added into the domain. I will mount it in the global zone, then loop it into "myfirstzone" (This allws you to mount it into multiple zones should you so wish.)

mkdir mountpoint_for_myfirstzone_space1 mount /dev/lofi/1 `pwd`/mountpoint_for_myfirstzone_space1 zonecfg -z myfirstzone zonecfg:myfirstzone> add fs zonecfg:myfirstzone:fs> set type=lofs zonecfg:myfirstzone:fs> set special=/export/zones/mountpoint_for_myfirstzone_space1 zonecfg:myfirstzone:fs> set dir=/space1 zonecfg:myfirstzone:fs> end zonecfg:myfirstzone> verify zonecfg:myfirstzone> confirm zonecfg:myfirstzone> exit

Make the change take effect zoneadm -z myfirstzone reboot

Log back into the zone, and look at the output of df again. Compare this with the output from df in the global-zone ...

I'll post a follow-up article explaining some of the more advanced zone configuration topics, but hopefully you'll be impressed with how easy this all is. If you want to know why zones work, thing chroot jail combined with ppriv's (man privileges)

Have faith in Linux

Linux does not have an inherent right to exist. Now that may sound like blasphemy to many Linux aficionados, but let me explain.

I've just read one too many posts in Linux support forums where a new user coming from a Windows background have a problem with something that worked or were easy to do under Windows, and seems impossible or hard to do under Linux. Often the person is taking a slightly accusatory tone, maybe because they have been fooled into thinking everything will be plain-sailing. The unfortunate all-to-often response that I see is that "Linux is not trying to compete with Windows".

A statement like that is a wimpish cop-out, and is wrong on so many levels it isn't even funny. People who read this response gets the message that Linux cannot compete with Windows. Before you write this again, go read one of the many "Why Linux is Better than Windows" pages, and think about that statement again.

Of course Linux is competing with Windows: A computer runs only one operating system at a time, either Windows or Linux or some other operating system (barring virtualization of course).

Importantly, without users using, and therefore computers running Linux, there is no Linux. Developers need motivation to write drivers and programs, and with no users, much of this motivation disappears, and Linux disappears into the annals of time as a futile though interesting exercise in creating an alternative to Windows.

So without Users, there is No Linux. Linux' right to exist resides in its user base, and every user who goes back to Windows because of insensitive answers is one less reason to develop Linux. You may consider each sensitive, well-thought-out, positive, encouraging answer to a frustrated users' plea for help to be an investment into the user base which will ensure that tomorrow you still HAVE a Linux platform.

Maybe we should take a page out of the fanatic "fanboy" Linux followers' book and become a bit more aggressive in our approach to promoting and backing Linux as a better-than-Windows platform.

People will always compare their experience of Linux with what they are used to getting from Windows. Unfortunately human nature have us focus on the negatives, and this means people overlook the great features of Linux and open source software because of a few small troubles.

I find there are two things that need to be said when new Linux users are frustrated with the platform.

• One, remind the person that Linux development is behind on some areas due to restrictions, such as DRM and copyrights existing in the Windows software, which slows down or outright prevents development of software for Linux. At the same time remind the user that in many more areas Linux is far ahead of its Windows competition.

• And second: encourage the person to continue looking for a solution - New programs and packages appear every day, so a newly hatched solution might exist but still be unknown to most people.

So go and arm yourself with some real information on why people should switch completely to Linux so that you don't come across as a fanboy, and stop wimping out when Linux does fall short of its competition.

A file server for home (Part 2)

In this second part of "A file server for home" I will introduce a few factors which I am currently considering in making the decisions about what my file server will eventually look like. Monetary constraints: That money would be a factor is so obvious it is almost overlooked. It is the single most important contributor in any business decision, and while one might be tempted to think that power users have a large budget for computer spend, there exist a certain culture among many hackers, particularly in the "over-clocking" community, in which one try to make the most of the least through being clever in stead of by throwing money at the problem. While I am not into “over-clocking”, I can still respect the art from the sideline. I am looking for a cost-effective way to achieve resilience and stability, and the principle remains: I will try to build the best file server possible while spending as little as possible through (hopefully) being clever with configuration decisions. With this in mind I am going to try to do two things: I will attempt to reuse what I have, and when I cannot help it, I will buy components with an upgrade path and maximal life expectancy. Physical Constraints: One of the implied ideas of a file server is that it is going to be used to house a number of disk drives in the chassis. Keeping in mind that I want to reuse some of my existing hard drives and want future expansion capability, it is clear that the chassis must not be too limited in terms of the number of drives that it supports. I had a look at the cases that I have lying around in my garage, particularly a lovely, if somewhat older, full tower case. Unfortunately none of them are ATX form factor compatible, and I will have to use a motherboard which supports SATA hard drives, which implies that the motherboard will be fairly new and therefore will be an ATX form factor board, which effectively eliminates the use of any of the unused cases that I already have: Hence the need for a new chassis. I started searching and reading reviews on web sites, and one thing that frustrated me was the limited space for installing hard drives in many cases - Most top out at 8, sometimes no more than six drives are supported, and that is counting the bays to be used for CD/DVD and floppy-disk drives. Only the very expensive cases support ten or more disk drives. Additionally, I hoped to find a case in which the disk drives are side-accessible so that I would be able to service drives without having to resort to removing the graphics card, etc. As luck would have it I found someone who wanted to sell a case because it was too large and too heavy to carry around to LAN parties. The case in question turned out to be the Thermaltake VA8000. With a native capability to house 14 drives and the ability to be extended further by using special drive bay modules, it was an almost perfect fit to my requirements. On checking the condition of the case I found that there was some scratches on the side panel, the keys are missing and the front doors would no longer latch into the closed position. These are all really minor issues, and while cleaning the case I discovered that the little magnets meant to keep the doors from swinging open had just fallen out and were stuck to metal parts elsewhere in the chassis, so that was easily sorted with a bit of crazy-glue. What is more is that the case's monstrous size makes it easy to access the drives even with a bunch of cards installed, despite the fact that the drives are not side accessible, so this chassis is going to form the perfect basis for my file server. In fact this case deserves a full review on its own, so watch this space for an upcoming post on this topic. I still need to finalize some of the other components for this server, but currently the items that I am considering are the following:

• Motherboard: I have a Gigabyte GA-K8NXP-SLI motherboard which I am currently using as my desktop "workstation". The SLI part is way overkill for a server, but this motherboard sports no less than eight SATA ports and two IDE controllers, for a maximum of 12 drives.
• CPU and Memory: Seeing that I will need to replace the motherboard in my desktop and the new motherboard is unlikely to use the same socket type, it means that my existing processor and memory will be freed up, and will serve just fine in the server. Therefore the server will inherit my existing AMD Athlon 64 3000+ processor and 2 GB of DDR400 memory.
• Graphics card: This motherboard does not come with an on-board graphics card, so I will have to provide one. My existing card is a Gigabyte GV-3D1 dual 6600GT, which would be totally overkill for any server, but might not be compatible with whatever new motherboard I eventually get for my desktop. I will test the GV-3D1 on my new motherboard and if it works it will stay in my desktop and I will just use an old PCI graphics card in the server. Otherwise I will use it in the file server purely to prevent it from being wasted, and then I would need to start evaluating what to buy for my desktop, though that is an entirely different discussion in itself.
• DVD-Writer: The drive I have right now is a very capable LG Lightscribe, but is slower on some media formats, particularly the DVD "minus" variety of disks. I will buy an extra DVD-writer, and then decide which one of the two goes into my desktop, and put the other one in the server. This is not something that needs to be decided now, but it will depend on what my usage pattern turns out to be: Will I be writing more disks on my desktop or will I need it more often in the server to make backups...?
• Tape Drive: I am hoping to find a tape drive to ease the backup process, ideally either an LTO 3 or LTO 4 drive. This is something that I will add to the file server when I find a suitable device.
• Hard Drives: The server will start off by inheriting the six disk drives currently in my desktop system, these being: an 80 GB IDE drive, a 320 GB IDE drive and four 160 GB SATA drives. More drives will be added over time as and when required.
• Power Supply: Hard drives do not actually use huge amounts of power. It is typical for a drive to use around the 10W mark, while during peak usage or start up they drives can use up to about 30W each. Hard disk spin up is staggered, and if I assume that out of about 10 potential drives I am unlikely to often have more than one drive operating at peak performance at any one point in time, with maybe two more drives operating at around the 20W mark and the rest mostly idle at 10W, I will realize around (7*10) + (2*20) + 30 = 140 W for the disk drives. The graphics card could potentially peak at anything from 20W for an old PCI card to around 100W if I were to use the GV-3D1 card (I don't expect to ever see the graphics card in the server peak but I will use peak values here just to be safe). The CPU, RAM and Motherboard chip set adds another 150W during peak usage, bringing the total to around 400W. This is peak usage, but a 500W power supply gives me some head-room for movement and in any case you do not want to operate at the limit of a power supply's capacity as it is a likely source of instability. A good power supply is something that I will need to purchase.

The Gigabyte motherboard has got one caveat that I need to mention: Four of the SATA ports are provided by a Silicon Image SI3114 controller, which does something non-standard to the disks connected to it. What I have found is that with drives that I have previously used on the SI3114 the partition table is not recognized when you try to use them on a different SATA controller. This also happens if I try to use one of these drives in an external USB enclosure like the NexStar III SATA enclosure that I have. On the other hand, disks which were previously partitioned while connected on the nVidia chipset's built-in SATA controller is recognized on other motherboards and in my external enclosure without any problems. The reason why I mention this is that if the motherboard ever should fail, I will be in trouble because these controllers are not present on to many motherboards, and so, unless I can get another motherboard with the same type of controller, I would have a hard time reading the data from the disks that were connected to the SI3114 controller. I must admit that this is not something that I have been able to verify yet, but there is this risk. Note: All of the motherboard SATA ports are set to operate in JBOD mode, which allows me to select how I want to set up the RAID in software, and also allows me to change the RAID layout later. I also still have to make a final decision on what operating system to use. I would be a poor Solaris fan if I did not at least consider Solaris as an option, but in my experience it is just more suited to a server environment where the applications you want to use are unlikely to change much and all well supported under Solaris. OpenSolaris offers me the same capability to set up zones and ZFS file systems, but many applications are easier to get to run under OpenSolaris. Seeing that I will want to use the server for several other purposes besides simple file sharing and because I have not yet identified the actual software that I will use, I will eliminate Solaris at this point just because it will be too restrictive in future. If file serving was the only purpose, then it would have been perfect. I will also eliminate Windows because it is not free, too difficult to configure, doesn’t have enough free software, and is just generally a pain to work with. If I were to use Linux I would have trouble getting ZFS to work - the Linux ZFS port is not nearly stable enough for my liking. This means I will probably have to use OpenSolaris. The decision to use OpenSolaris is not yet cast in stone, but it looks like the right choice because I will be able to set up Zones/Containers and I will be able to use ZFS features like snapshots and raid-z. The four 160 GB disks that I have will nicely make up one raid-z "stripe". For the rest of the disks I will need to find an optimal way to configure them, but I still need to investigate the optimal way to set up the disk. I am also considering is to get the free Foundation Suite Basic disk management software as I know it very well and it too provides simple and online management of disk space. In part three of this series I will go in depth in how I selected an operating system for this file server.

Choosing between Solaris, Linux and MS-Windows

Today I will take this very contentious, highly flame-war-provoking issue head on. Many people are likely to disagree with me, but that is OK - opinions are a big part of what makes the Internet tick. Naturally you need to start off by considering your specific requirements and the resources that you have available: Many RISC based computers support only a very specific Unix operating system, while many consumer devices such as web-cams and WiFi adapters are difficult if not impossible to get to work with anything other than MS Windows. It will make sense to take stock of your existing skills to support that environment when choosing an operating system for a server application if multiple options exist. To start off with I would like to group operating systems into three categories the way I see it. These are intentionally vague, and operating systems can migrate between the categories. The categories are:

• The End-User operating system: The question "Does it do the job?" comes to mind. I will include MS Windows and MacOS into this category. For reasons I will get into soon, this includes the Server, Pro, and Home editions of MS Windows.
• The Power-user operating system: "I'll hack it until it works the way I want it to work". This category without any doubt includes Linux.
• The server operating system: This is for specifically supported hardware running specifically supported applications, and includes all the Uni'ces, in particular Solaris because it is available for free and on non-RISC hardware, and probably FreeBSD as well.

I'll discuss each of these categories a bit more in depth, taking them in the opposite order. In the enterprise data center we usually find many large, multi-processor servers designed and ratified to run specific operating systems, which in turn are ratified and supported for use with specific applications and databases. Vendors in this environment sell solutions where all the components making up the whole have been preselected, and include support for their solutions under a contract which often limits your freedom, for example by limiting you to specific versions of a specific operating system. In this environment, going with a supported operating system is the only sensible choice, and the big Unix vendors all have long lists of applications that can be run on their systems. The benefits of going with these types of solutions lies in that the hardware is intelligent about the operating system, being able to halt the operating system without terminating it, allowing hardware to be re-configured while the operating system is running, is able to take crash-dumps when the system hangs and presents a uniform device tree of the installed hardware to the operating system. This all provides a high level of stability and supportability, something which comes at a price, but worth it when the service delivery to your clients depends on the systems running without interruptions. That is not to say that Linux and MS Windows servers do not have a place in the data center - the exact same rule holds true: Select the platform for which your application is supported - whether it is a mainframe or a network vendor's appliance with an embedded firewall and authentication service. With Linux-based servers I often notice that the applications prefer an environment where a high degree of customizability is provided, and with Windows servers I often find that the applications have a keep-it-simple nature, at least as far as the server configuration is concerned. Linux fits squarely into the Power-user operating system category. Features like the ability of the user to re-compile the operating system to eke out an extra ounce of performance, directly modify the source code to get a specific result, the ability to change and/or replace any single component with one of your own choosing, and a wonderful following of supporters eager to help out one another truly makes this operating system into a power-user's heaven. I find myself more productive when working on Linux: this is without a doubt because I have it set up exactly the way I like it and I have a huge collection of software tools installed and configured to help me maintain the system. Also something that helps is that Linux is much more transparent in reporting what is going on "under the hood" - You can obtain accurate log files of system events, examine what program processes are doing in fine detail, and you have accurate reports of the state of every single component that the operating system controls. This all makes it easier to track down the "root cause" of problems, and in my experience with Linux, using the computer is less of a hit-and-miss affair - if it works a certain way today, it will work the same way tomorrow and the day after. Things like program crashes are much more easily reproducible, and as a result, easier to resolve. All of this requires a certain level of commitment and even enthusiasm to learn more about the workings of the computer and the operating system though, and that is probably not for everyone. The other options available to "power users" with PCs are basically FreeBSD, Solaris, Open Solaris, and Windows. Windows is sometimes not transparent enough and things do not always work the way Microsoft says it does. Solaris' hardware support is still somewhat limited, making it a difficult to justify choice. Open Solaris is maybe still too much of a newcomer to be judged fairly, tough good work is being done, albeit slowly, by projects such as Nexenta GNU Solaris, an OpenSolaris distribution providing the GNU tools in stead of the Solaris defaults. The End-user category are generally for people who don't care that their operating system limits their ability to troubleshoot faults, learn more about the computer or replace sub-components of the operating system. Many of these are company PCs used to run a selection of applications needed to run their business, the support and development of which companies have invested a lot of time and money into. 20 years ago Microsoft was probably deliberately lax in prosecuting pirates of their operating systems and application software. Most of these "pirates" were kids who were learning how computers worked, and by the time they made it into the workforce, Windows was what they expected on their computers and in the data centers. Today this large following of Windows users equates to a workforce who will pretty much need to be retrained if one suddenly wanted to switch to a different operating system. You could argue that the training is not so hard - many applications are web based these days, sending an email remains pretty much the same, etc, but this only holds true in theory: The moment many users are faced with a new login screen that looks different, they feel lost. Suddenly the user name field becomes case sensitive, and your lost user turns into a frustrated user who hasn't even had a chance to try the applications yet. Worse, every application's menus are laid out differently, and that is only once they learn which application does what. Space characters in file names suddenly cause havoc, and the slash in directory paths is turned around. To the end user, it feels as if nothing works the way they are expecting it to work and it takes months before they once more become productive. I must agree with A Russel Jones who concludes that for people who don't want to have to consider whether their applications will work with the window manager that they have installed, learn how to enter commands in terminal, or why security is a good thing when it seems to just make life harder, a Windows based PC is often the best option. Linux can be carefully configured to simulate this in a controlled environment where an IT department makes all the design decisions, installs all the software, and makes sure that the all of the hardware in use is supported - when you have people with the necessary skills. Setting up such an ideal configuration is something that will take time and effort, and that means there is a price tag attached, and this could be more expensive than just running Windows. In the end business decisions always come back to the numbers: What are the benefits (over the alternative), and how does the long-term costs of the various options compare. The Pro and Server editions of MS Windows have the same basic limitations that the Home edition have: The server edition is no more stable, powerful, transparent, manageable, or secure than the Home edition, and this puts it into the "End-user" category. It also fits in with the "keep-it-simple" tendency of this category, even if it is installed on a data center rack mounted server which never crashes. The power user is probably the best off: They have the most freedom to choose what operating system and applications they want to run, how they want to configure their computers, and can usually make their own choices about what programs to use. This is basically because they are able to support themselves when things stop working. So basically it boils down to the age-old adage: Use the right tool for the job. I suspect that you will find that nine times out of ten the categories I described here will hold true though. Disclaimer: This article necessarily relies heavily on generalization. Please keep that in mind if you decide to comment.