Making the most of Solaris Man pages

Solaris man pages (manual pages) are well written, consistent, complete, and generally a great source of information. Here are a few tips to help you get the most out of them. Of course this applies to all Solaris derivatives, including proper Solaris, OpenSolaris and other Open Solaris distributions like Solaris Express, Belenix and Nexenta. It even applies to other Unix flavors like BSD and AIX, and Unix derivatives like Linux.

The simplest (and most obvious) way to use the manual pages are to enter "man command" and then read the entire page.

My first hint however is to change the "PAGER" environment variable. I suggest you set it like this in your .profile file:

PAGER="less -iMsq"; export PAGER

(or for csh and its family members, use setenv PAGER "less -iMsq")

The reasons for this are

a) "less" supports more useful options than does "more". In particular, it supports highlighting, as well as all of the below!

b) The "-i" causes less to ignore case in searches. A definite advantage because you can simply enter /user and it will find the string "user" even when capitalized for use at the start of a sentence.

c) The -M causes "less" to show a more verbose status at the bottom of the screen.

d) -q ... Stop irritating me.

e) -s to "squeeze" blank lines (because otherwise less will "format" pages with blank spaces, as if it is a printed page)

f) You can return to the first line (top) of the page by taping "1" followed by "G"

g) less is more.

The PAGER environment variable is automatically used by the man command, so you can now test it and will find that you can scroll up and down in man pages. To find a word in the man page, tap the forward-slash / followed by the string to search for, and the word wherever it is found will be highlighted in inverse text.

Note: less does have some limitations over more, but these do not apply to viewing man pages. For the curious, the limitation is in the handling of control characters in the input file. In particular, more does a better job of displaying "captured" sessions recorded from a system console or from a shell using the "script" command. Regardless, I solve these by reading capture files using cat -nvet | less -iMq (Note - I like seeing line numbers)

Second Hint: Generate the Manual Pages keyword index database (The so-called windex or apropos information). To do this, you need to run the following command once.

$ catman -w

This will run for a while as it generates and stores the man page keyword index for future use.

Once it completes, the man command's -k option and the "whatis" command will work. This allows you to find what you need much easier. For example to find all man pages for tools, commands and drivers related to Wifi and Wireless networking, you can use

$ man -k wifi wireless

Third hint: Some man pages are not automatically searched or found. In older (proper) Solaris releases, the man command has got a "default list" of locations where to find man pages. In Recent Open Solaris releases, man constructs a list of locations to search. The man manual page states:

  Search Path
     Before searching for a given name, man constructs a list  of
     candidate directories and sections. man searches for name in
     the directories specified by the MANPATH  environment  vari-
     able.

     In the absence of MANPATH, man constructs  its  search  path
     based  upon the PATH environment variable, primarily by sub-
     stituting man for the last component of  the  PATH  element.
     Special  provisions  are added to account for unique charac-
     teristics  of   directories   such   as   /sbin,   /usr/ucb,
     /usr/xpg4/bin, and others. If the file argument contains a /
     character, the dirname portion of the argument  is  used  in
     place of PATH elements to construct the search path.

Note: The (currently apparently undocumented) -p option to man will show the effective man search path.

As implied, you can set in MANPATH a list of directories containing man pages you use regularly. It is also possible to overide the man search path using the man command's -M option. This is particularly true for packages that install under /usr/local, /opt/SUNW...., etc. So to view the man page for "explorer" you would run:

man -M /opt/SUNWexplo/man explorer

(Assuming you have SUNWexplorer installed)

Finally, it helps to understand the structure of a man page. The individual manual pages are divided up into a common set of sections, and knowing them you will often skip straight to a specific (sub)section in the page when looking for specific information.

Some of these subsections are optional, but the names used are consistent. Here is a quick summary of some important sections.

NAME	What this page is about. This is the "whatis" information.
SYNOPSIS	The summary, eg how to use a command.
DESCRIPTION	A more complete description of what this component is.
OPTIONS, OPERANDS and USAGE	These section details how a command is used, eg what command-line options are available for a specific command.
ENVIRONMENT VARIABLES	As the name suggests, this section details Environment variables which affects the behavior of the command.
EXIT STATUS	As the name implies, it explains the possible exit status values. Useful to know that the section exists though.
FILES	A list of files which are relevant to the command or subsystem. In particular here you will find out where a program keeps its configuration files. For a good example, see man dumpadm.
EXAMPLES	Often a great place to skip to when a manual page is complicated.
ATTRIBUTES	An often overlooked section, it explains the status of the item. For commands it tells you which packages the files are in. This section of man pages are fully explained in the "attributes" manual page.
SEE ALSO	One of the most useful parts of man pages, this gives you hints about other pages which are related to the same topic.
NOTES	Various Additional notes.

Other sections, such as SECURITY, ERRORS, SUBCOMMANDS, etc exists in some man pages, but the above are probably the most useful sections.

The entire collection of man pages are divided up into manual sections, such as Section 1 - User Commands, Section 1M - Maintenance commands for sys-admins, Section 3C, the C programming reference information and section 7 - information about device drivers. This is not hugely important, except that you need to understand that some pages occur in more than one section. For example "read" or "signal" By default, man will display the first manual page which matches the file name. If you want to see one of the other pages you need to specify the specific manual section explicitly.

For example

$ man -f signal signal signal (3c) - simplified signal management for application processes signal signal (3ucb) - simplified software signal facilities signal signal.h (3head) - base signals

We can see that file "signal" has got manual entries in sections 3c (C programming reference), 3ucb (UCB/Posix version), and 3head (the Headers section)

Simply entering ||man signal|| will show the man page from "3C". To see the man page in the "3head" section, enter either

$ man -s 3head signal

or

$ man signal.3head

I often end up using the second form simply because after viewing a man page I often see that there is another related man page from the SEE ALSO section. I then exit the man page, recall the previous command, and append .section to the command line, which is to say that it is just a matter of convenience.

One last tip: There are online versions of the manual pages at http://docs.sun.com.

You don't need to learn every command and every option by heart. Knowing that you have manual pages, and that you can look up the information you need quickly by looking at related commands from SEE ALSO, and from the keyword index using man -k, and then being able to quickly search through man pages for the right section or keyword absolutely WILL make you a clever, more efficient and over all better sysadmin!

The Pupil will surpass the tutor

Linux is an attempt at making a free clone of Unix. Initially it aimed to be Unix compatible, though I feel that goal has become less and less important as Linux grew in maturity.

Now all of a sudden we have a complete turn-about as the big Unices want to be like Linux! - Linux is attractive for a variety of reasons, including a fast, well refined kernel, lots of readily available and free applications, good support and, because of these, a growing and loyal following. The utilities available with most Linux distributions are based on the core utilities found in the big Unices and have a large collection of new additions, all working together in a more or less coherent way to build a usable platform.

Nowadays many new Unix administrators have at least some Linux experience and with this background can be easily frustrated when looking for Linux specific utilities (Where is top in Solaris). End users would like to see the applications that used on Linux to run on Unix. And the ability to run Linux on cheap and cheerful PC hardware does not detract from Linux's popularity by any means.

So Sun Microsystems, just like IBM with AIX, finds itself looking at Linux to see what this platform is doing right to make it sucesfull. To me this, more than anything else, is proof that Linux finally grew up.

I expect that a leap-frog game will emmerge between Linux and Unix, particularly Solaris, where the two are competing with innovative features to be the platform of choice for both datacenter and desktop applications.

Congratulations to the Linux community on a job well done.

Sharing a ZFS pool between Linux and Solaris

If you are multi-booting between Linux and Solaris (and others like FreeBSD, OpenBSD and Mac OS X, I expect) you will sooner or later encounter the problem of how to share disk space between the operating systems. FAT32 is not satisfactory due to its lack of POSIX features, in particular file ownership and access modes, not to mention its sub par performance. ext2/3 is not an option because you only get read-only support for it in Solaris, and similarly UFS enjoys only read-only support in Linux. The whole situation is rather depressing.

Enter ZFS.

This all started because I discovered that I can have a ZFS root file system without having to install OpenSolaris. The trick as some of you may know, is to select "Solaris Express" from the first menu on booting the install disk, and then select one of the two "Interactive Text" options from the next menu. This puts you back into 1984 in terms of installers, but you get the option of using ZFS for root!

Note: It might be possible to do this with the default installer, but on my computer the installer just would not run (I got some daft error about fonts and mouse themes). With a ZFS root, the Swap and Dump automatically goes onto dedicated vdevs, and you save a lot in terms of pre-allocated space.

I have of course used ZFS on my laptop previously as a test, but the benefits were limited by the fact that I still had "slices" for the OS and a small ZFS pool on a spare slice.

I'm not sure which build of Nevada first introduced the ZFS root option in the installer, but it is available in build 90 at least.

My choice of Linux distribution is Ubuntu 8.04. The steps to setting up a ZFS pool shared across operating systems are as follow:

1. Select a Partitioning scheme with minimal space allocated to each of Ubuntu and Nevada.
I decided to put Ubuntu in an Extended partition with a 10 GB Logical Partition for the OS, /var and /home, and a 1 GB Logical partition for Swap.
For Solaris I allocated a 24 GB primary partition to become the ZFS root pool, which includes Swap, Dump, OS and Live-upgrade space.
The balance of the 100 GB disk will be shared between Ubuntu and Solaris using ZFS.

Note: Linux and Solaris has got some different views on how disk partitioning works.
Due to historical reasons, in particular due to compatibility with Solaris on SPARC hardware, Solaris slices live in a single primary partition with an identifier of 0x82 (SOLARIS) or 0xbf (SOLARIS2) somewhat like how logical fdisk partitions live inside an "extended partition".

2. Install Ubuntu first, creating only the partitions for it. Remember to not have any external drives connected as it can screw up the order in which drives are detected and as a result bugger up the Grub menu list.

During the installation you create an Admin user. This will eventually in the future become a "backup" admin user.

3. Reboot and load patches/updates, and backup the Grub /boot/grub/menu.lst file to an external media such as a USB thumb drive for easy access. The Ubuntu Grub does not understand ZFS, so you need to use Nevada's Grub to manage the multi-booting.

4. Also set Ubuntu to use the hardware clock as local time in stead of UTC. (This is what Solaris uses) To do this change UTC=yes to UTC=no in /etc/default/rcS, then reboot.

5. Install Nevada. Use either of the Interactive Text installer options, but for simplicity's sake specify the system as non-networked.

6. Reboot and create a user for every-day use, and add this user to the "Primary Administrator" role using usermod -P "Primary Administrator" <userid>

7. Add the Ubuntu Grub entries you saved in step 3 to the end of the Nevada grub menu.lst file. This will be stored in /boot/grub/menu.lst (The default pool name is rpool)

8. Reboot back into Ubuntu, then follow the Linux ZFS-FUSE instalation instructions to get ZFS-FUSE installed. I used the trunk to get the latest ZFS updates from Opensolaris.org included. Also see this Ubuntu Wiki page, and Ralf Hildebrand's blog for more info.

For reference, this is the procedure I used

apt-get install mercurial build-essential scons libfuse-dev libaio-dev devscripts build-essential zlib1g-dev cd ~ hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk cd trunk/src scons sudo scons install

9. Create an fdisk partition for the shared ZFS pool using the remaining disk space. I used a primary partition and set the identifier to W95 FAT32, though this is probably unimportant.

10. While still running running Ubuntu, create a ZFS pool on this new fdisk partition using a command like this:

sudo /usr/local/sbin/zfs-fuse sudo /usr/local/sbin/zpool create -m /shared SHARED

I like to give my ZFS pools names in all-capitals, purely because it makes the ZFS pool devices stand out better in the output from df and mount.

WARNING: I found that if I created the ZFS pool under Solaris, it refused to import into Ubuntu, but if I created it under Linux it imports/exports just fine in both directions. Both pools are created as version 10 pools, so the reason for this is not obvious. If you do decide to experiment with creating the pool under Solaris, when you want to realy get rid of the pool you will discover you need to dd zeros over the pool before creating it again, otherwise the condition remains unchanged despite destroying and re-creating the pool. If you do experiment with this please do share your results!

11. Export the ZFS pool using

/usr/local/sbin/zpool export SHARED

12. Reboot into Nevada and import the pool using

/usr/local/sbin/zpool import SHARED

Note: If you forget to export before you shut down, you will need to add -f to force the import after booting into the other OS.

At this point I just sat there and stared in wonder at how well it actually works. There is beauty in finally seeing this working!

13. Create some init.d / rc scripts to automate the import/export on shutdown/startup.

14. Now you can start customizing both operating environments. You may want to setup Automatic network configuration by enabling the SMF for NWAM in Solaris, eg by doing:

pfexec svcadm disable physical:default pfexec svcadm enable physical:nwam

I'm looking forward to testing Live Upgrade on my setup with ZFS root, and to getting a shared home directory to work well for both Solaris and Ubuntu. I have created a login ID with the same gid/uid and a home directory under the shared ZFS pool, but after a few changes it got broken under Ubuntu, probably due to subtle differences in how Gnome/Desktop config items are stored and/or expected.

Despite my initial sceptism about FUSE, it is actually quite functional. All-in-all I love being able to share a file system, well, many files systems, between the two operating environments!

15 minutes to your first Solaris zone

So you got OpenSolaris on your local machine and you would like to try out Zones. This tutorial will give you a 15 minute by-example guide to setting up your first zone, and after that explain a few things hands-on, showing you where what is.

There are a few things to understand about Zones before we start:

First thing is the concept of the Global Zone - the "base" operating system instance is the Global zone and all other "local zones" run under the control of the global zone.

Secondly you get three types of local Zones, being Full root zones, Sparse root zones, and Branded Zones. These differ in what gets copied into the zone's root file system, as well as in how they start up. Branded Zones, which starts up using custom scripts and provides special hooks into the operating system, can emulate a different kernel, but that is a topic for another day.

A Full root zone gets a complete copy of all the Solaris installed package files, which needs about 5 GB of storage. A Sparse zone only gets its own /etc and /dev, which saves on space but it uses a read-only loop-back mounted copy of other directories, such as /opt and /usr. Copying all the files into a Full root zone takes some 10 minutes longer than setting up a sparse zone and uses 5 GB more, but because /usr and /opt is writable it results in a more flexible (read more useful) example environment.

Normally before you set up a zone you would need to do thorough planning, taking care of a things like disk space/file systems, CPU and memory resource allocations, maybe even special boot options. But for a quick start lets just assume defaults for most things.

You do need to work out a few details before you start. 1. Select a name for the zone. This can also be the "hostname" for the virtual OS instance in the zone. 2. Select an IP address and identify on which interface it will live. 3. Identify a spot on your file system where the Zone root will be installed.

Lets call the zone "myfirstzone", let it use IP address 192.168.24.131 on interface e1000g0, and give it a directory /export/zones/myfirstzone

mkdir -m 0700 /export/zones/myfirstzone

Yes, you guessed right, later we can add more zones under /export/zones. Be aware, if you don't restrict the permissions on the directory, the zone installation later on will fail with a security error.

The commands for working with zones are:

zonecfg - Change a zone's configuration / setup. This will create and/or modify the xml base configuration of a zone. zoneadm - Manipulate running zones, eg rebooting and showing their status. zlogin - Connect to a zone (Create a shell session in a zone or connect to the zone's console)

Without further ado, creating the zone using zonecfg zonecfg -z myfirstzone zonecfg:myfirstzone> create zonecfg:myfirstzone> set zonepath=/export/zones/myfirstzone zonecfg:myfirstzone> add net zonecfg:myfirstzone:net> set address=192.168.24.131 zonecfg:myfirstzone:net> set physical=e1000g0 zonecfg:myfirstzone:net> end zonecfg:myfirstzone> commit zonecfg:myfirstzone> exit bash-3.2#

The basic Zone has now been set up. We can view it using this command zonecfg -z myfirstzone export create -b set zonepath=/export/zones/myfirstzone set autoboot=false set ip-type=shared add net set address=192.168.24.131 set physical=e1000g0 end

Now to populate it with Solaris packages/files, very simply use the zoneadm install command. This process takes a good 15 minutes or more, and you should see output like this:

bash-3.2# zoneadm -z myfirstzone install Preparing to install zone . Creating list of files to copy from the global zone. Copying <187209> files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <1282> packages on the zone. Initialized <1282> packages on zone. Zone is initialized. Installation of <1> packages was skipped. The file contains a log of the zone installation.

The install process is highly disk IO intensive. If you're building a zone on the same disk from which the system is running, the resulting disk contention can cause it to run quite a bit longer, up to an hour on systems with slow disks.

Use this time to scan through the man pages for the zone commands. There is also an introductory man page, i.e zones(5).

Also while this install is running use the time to look at the output from these commands: zonecfg -z myfirstzone info

zoneadm list -vc

Once the installation process finishes, you can boot the zone and log in. During the first login, you need to login on the zone console and provide the system identification information prompted.

zoneadm -z myfirstzone boot; zlogin -C myfirstzone [Connected to zone 'myfirstzone' console] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone ... ... ... System identification is completed. rebooting system due to change(s) in /etc/default/init [NOTICE: Zone rebooting] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone Reading ZFS config: done. zoneone console login:

Now log in as root with the password you just set, and run a few commands in the zone; - Do a ps -ef in the zone, and another one in the Global zone, and compare the output. - Also compare the output from a few other commands, particularly df -h, uptime, zonename, hostname, ifconfig, and uptime. On the ifconfig -a command output, note the zone identifier for each interface, as well as the multiple loopback identifiers.

For now I will leave you to marvel at the differences and similarities, but I will point out one process running in the global zone, namely zoneadmd. It manages the zone environment, ensures that its file systems and network interfaces are mounted and plumbed, etc. There will be an instance of it for every running non-Global zone.

Disconnecting from the zone Console requires that you enter the "disconnect" escape sequence, by default ~. (A tilde followed by a full stop) If you're already working remotely using something like SSH, use ~~. - The extra tilde character will inform SSH that it should not itself act on the disconnect command, but rather pass it on down the server. It is also possible to change your escape sequences, but I'll reserve that for a small blog article on another day. Note that disconnecting the console is NOT the same as logging out of the zones - processes like prstat will continue to run on the zone's console and you can reconnect later by again by running "zlogin -C myfirstzone"

Now that the zone is running, lets do a few interesting things with it.

First, in the Global Zone, have another look at what you see from zoneadm list -vc. Other interesting command are "prstat -Z", as well as "df -Z"

Next try out zlogin sans the -C option which would put you on the console. Use exit, logout, or ^D to disconnect.

Create a new user in the zone zlogin myfirstzone mkdir /export/home useradd -d /export/home/joe -m joe passwd joe

Use SSH to login to the zone from another machine, using the zone's IP address. Confirm where you logged in to using hostname, and zonename.

Now lets start an FTP server in the zone: zlogin myfirstzone svcadm enable ftpd

zlogin as used above runs the specified command directly in the zone. You can check the results using by ftp-ing to the zone, though you need to use a user like joe created above, which can login via the network.

Finally, lets add some disk space to our existing zone. This could have been done during the configuration process, but that would have defeated the purpose of showing how to add something to a zone.

I don't have an extra disk device, so I'll use a loop-back (lofi) device. cd /export/zones mkfile 1g fakedev_for_myfirstzone lofiadm -a `pwd`/fakedev_for_myfirstzone newfs /dev/lofi/1 /dev/rlofi/1: Unable to find Media type. Proceeding with system determined parameters. newfs: construct a new file system /dev/rlofi/1: (y/n)? y /dev/rlofi/1: 2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors 1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632, 2083232, 2092832

There are at least 3 ways in which this file system can be added into the domain. I will mount it in the global zone, then loop it into "myfirstzone" (This allws you to mount it into multiple zones should you so wish.)

mkdir mountpoint_for_myfirstzone_space1 mount /dev/lofi/1 `pwd`/mountpoint_for_myfirstzone_space1 zonecfg -z myfirstzone zonecfg:myfirstzone> add fs zonecfg:myfirstzone:fs> set type=lofs zonecfg:myfirstzone:fs> set special=/export/zones/mountpoint_for_myfirstzone_space1 zonecfg:myfirstzone:fs> set dir=/space1 zonecfg:myfirstzone:fs> end zonecfg:myfirstzone> verify zonecfg:myfirstzone> confirm zonecfg:myfirstzone> exit

Make the change take effect zoneadm -z myfirstzone reboot

Log back into the zone, and look at the output of df again. Compare this with the output from df in the global-zone ...

I'll post a follow-up article explaining some of the more advanced zone configuration topics, but hopefully you'll be impressed with how easy this all is. If you want to know why zones work, thing chroot jail combined with ppriv's (man privileges)

Is [insert_OS_here] ready for the desktop

What a daft question. Yet I've heard it repeated so many times that I almost get tempted to answer it.

Who's desktop, that is my counter question. Of course it utterly depends on what you want to use your desktop for, and more than that, HOW you want to employ your desktop. For some people no computer will be able to replace what they do on a drafting board, with an air brush, by means of filters and lenses, or using their lathe or whatever else. Those are probably the real artists.

The rest of us content ourselves with playing at it using whatever software we can find that best meets our needs. I loudly proclaim that don't have a single MS Windows file anywhere on my computers, but I secretly reserve the right to open a remote desktop onto my wife's PC from time to time and there run my favorite home floor planning program. (I have yet to find a floor planning/home design program which runs on anything other than MS Windows). By which I mean that each operating system has got some gaps and some areas where it excels.

I run Nevada on my work laptop, because it is the only operating system that is ready for the desktop. Qualifier: When at a customer's site I often need to be able to run core dumps, crash dumps, and explorers though analysis tools, jumpstart servers, and perform a few other miscellaneous tasks which are much harder to do on any other operating system.

On the other hand Linux is the only operating system that I can qualify as desktop ready when it comes to my normal desktop PC. On this particular system I try out new programs, fiddle with kernel options, play with hardware bits, do some (amateurish) video and photo editing, etc. I'm sure I could do most of that on [Open]Solaris or even Windows, but I'm not that masochistic. Linux just makes it easy for the enthusiast to get under the bonnet of the OS and scratch around. After a discussion with a colleague a few days ago I must say that this is not always a benefit (Linux by its very nature makes it difficult to support in a mission critical environment)

But the point I am trying to make is that you need to decide what you want to do, and how you want to go about doing it. Then next you need to consider whether there are any trade offs. Can you live with your Linux system not supporting your webcam, can you live with the limited selection of software applications available under Solaris, or can you accept Windows' inherent security and stability issues.

So next time you're tempted to ask whether your operating system is "ready for the desktop", please do qualify the question with what you use your "desktop" for.

Have faith in Linux

Linux does not have an inherent right to exist. Now that may sound like blasphemy to many Linux aficionados, but let me explain.

I've just read one too many posts in Linux support forums where a new user coming from a Windows background have a problem with something that worked or were easy to do under Windows, and seems impossible or hard to do under Linux. Often the person is taking a slightly accusatory tone, maybe because they have been fooled into thinking everything will be plain-sailing. The unfortunate all-to-often response that I see is that "Linux is not trying to compete with Windows".

A statement like that is a wimpish cop-out, and is wrong on so many levels it isn't even funny. People who read this response gets the message that Linux cannot compete with Windows. Before you write this again, go read one of the many "Why Linux is Better than Windows" pages, and think about that statement again.

Of course Linux is competing with Windows: A computer runs only one operating system at a time, either Windows or Linux or some other operating system (barring virtualization of course).

Importantly, without users using, and therefore computers running Linux, there is no Linux. Developers need motivation to write drivers and programs, and with no users, much of this motivation disappears, and Linux disappears into the annals of time as a futile though interesting exercise in creating an alternative to Windows.

So without Users, there is No Linux. Linux' right to exist resides in its user base, and every user who goes back to Windows because of insensitive answers is one less reason to develop Linux. You may consider each sensitive, well-thought-out, positive, encouraging answer to a frustrated users' plea for help to be an investment into the user base which will ensure that tomorrow you still HAVE a Linux platform.

Maybe we should take a page out of the fanatic "fanboy" Linux followers' book and become a bit more aggressive in our approach to promoting and backing Linux as a better-than-Windows platform.

People will always compare their experience of Linux with what they are used to getting from Windows. Unfortunately human nature have us focus on the negatives, and this means people overlook the great features of Linux and open source software because of a few small troubles.

I find there are two things that need to be said when new Linux users are frustrated with the platform.

• One, remind the person that Linux development is behind on some areas due to restrictions, such as DRM and copyrights existing in the Windows software, which slows down or outright prevents development of software for Linux. At the same time remind the user that in many more areas Linux is far ahead of its Windows competition.

• And second: encourage the person to continue looking for a solution - New programs and packages appear every day, so a newly hatched solution might exist but still be unknown to most people.

So go and arm yourself with some real information on why people should switch completely to Linux so that you don't come across as a fanboy, and stop wimping out when Linux does fall short of its competition.

A file server for home (Part 2)

In this second part of "A file server for home" I will introduce a few factors which I am currently considering in making the decisions about what my file server will eventually look like. Monetary constraints: That money would be a factor is so obvious it is almost overlooked. It is the single most important contributor in any business decision, and while one might be tempted to think that power users have a large budget for computer spend, there exist a certain culture among many hackers, particularly in the "over-clocking" community, in which one try to make the most of the least through being clever in stead of by throwing money at the problem. While I am not into “over-clocking”, I can still respect the art from the sideline. I am looking for a cost-effective way to achieve resilience and stability, and the principle remains: I will try to build the best file server possible while spending as little as possible through (hopefully) being clever with configuration decisions. With this in mind I am going to try to do two things: I will attempt to reuse what I have, and when I cannot help it, I will buy components with an upgrade path and maximal life expectancy. Physical Constraints: One of the implied ideas of a file server is that it is going to be used to house a number of disk drives in the chassis. Keeping in mind that I want to reuse some of my existing hard drives and want future expansion capability, it is clear that the chassis must not be too limited in terms of the number of drives that it supports. I had a look at the cases that I have lying around in my garage, particularly a lovely, if somewhat older, full tower case. Unfortunately none of them are ATX form factor compatible, and I will have to use a motherboard which supports SATA hard drives, which implies that the motherboard will be fairly new and therefore will be an ATX form factor board, which effectively eliminates the use of any of the unused cases that I already have: Hence the need for a new chassis. I started searching and reading reviews on web sites, and one thing that frustrated me was the limited space for installing hard drives in many cases - Most top out at 8, sometimes no more than six drives are supported, and that is counting the bays to be used for CD/DVD and floppy-disk drives. Only the very expensive cases support ten or more disk drives. Additionally, I hoped to find a case in which the disk drives are side-accessible so that I would be able to service drives without having to resort to removing the graphics card, etc. As luck would have it I found someone who wanted to sell a case because it was too large and too heavy to carry around to LAN parties. The case in question turned out to be the Thermaltake VA8000. With a native capability to house 14 drives and the ability to be extended further by using special drive bay modules, it was an almost perfect fit to my requirements. On checking the condition of the case I found that there was some scratches on the side panel, the keys are missing and the front doors would no longer latch into the closed position. These are all really minor issues, and while cleaning the case I discovered that the little magnets meant to keep the doors from swinging open had just fallen out and were stuck to metal parts elsewhere in the chassis, so that was easily sorted with a bit of crazy-glue. What is more is that the case's monstrous size makes it easy to access the drives even with a bunch of cards installed, despite the fact that the drives are not side accessible, so this chassis is going to form the perfect basis for my file server. In fact this case deserves a full review on its own, so watch this space for an upcoming post on this topic. I still need to finalize some of the other components for this server, but currently the items that I am considering are the following:

• Motherboard: I have a Gigabyte GA-K8NXP-SLI motherboard which I am currently using as my desktop "workstation". The SLI part is way overkill for a server, but this motherboard sports no less than eight SATA ports and two IDE controllers, for a maximum of 12 drives.
• CPU and Memory: Seeing that I will need to replace the motherboard in my desktop and the new motherboard is unlikely to use the same socket type, it means that my existing processor and memory will be freed up, and will serve just fine in the server. Therefore the server will inherit my existing AMD Athlon 64 3000+ processor and 2 GB of DDR400 memory.
• Graphics card: This motherboard does not come with an on-board graphics card, so I will have to provide one. My existing card is a Gigabyte GV-3D1 dual 6600GT, which would be totally overkill for any server, but might not be compatible with whatever new motherboard I eventually get for my desktop. I will test the GV-3D1 on my new motherboard and if it works it will stay in my desktop and I will just use an old PCI graphics card in the server. Otherwise I will use it in the file server purely to prevent it from being wasted, and then I would need to start evaluating what to buy for my desktop, though that is an entirely different discussion in itself.
• DVD-Writer: The drive I have right now is a very capable LG Lightscribe, but is slower on some media formats, particularly the DVD "minus" variety of disks. I will buy an extra DVD-writer, and then decide which one of the two goes into my desktop, and put the other one in the server. This is not something that needs to be decided now, but it will depend on what my usage pattern turns out to be: Will I be writing more disks on my desktop or will I need it more often in the server to make backups...?
• Tape Drive: I am hoping to find a tape drive to ease the backup process, ideally either an LTO 3 or LTO 4 drive. This is something that I will add to the file server when I find a suitable device.
• Hard Drives: The server will start off by inheriting the six disk drives currently in my desktop system, these being: an 80 GB IDE drive, a 320 GB IDE drive and four 160 GB SATA drives. More drives will be added over time as and when required.
• Power Supply: Hard drives do not actually use huge amounts of power. It is typical for a drive to use around the 10W mark, while during peak usage or start up they drives can use up to about 30W each. Hard disk spin up is staggered, and if I assume that out of about 10 potential drives I am unlikely to often have more than one drive operating at peak performance at any one point in time, with maybe two more drives operating at around the 20W mark and the rest mostly idle at 10W, I will realize around (7*10) + (2*20) + 30 = 140 W for the disk drives. The graphics card could potentially peak at anything from 20W for an old PCI card to around 100W if I were to use the GV-3D1 card (I don't expect to ever see the graphics card in the server peak but I will use peak values here just to be safe). The CPU, RAM and Motherboard chip set adds another 150W during peak usage, bringing the total to around 400W. This is peak usage, but a 500W power supply gives me some head-room for movement and in any case you do not want to operate at the limit of a power supply's capacity as it is a likely source of instability. A good power supply is something that I will need to purchase.

The Gigabyte motherboard has got one caveat that I need to mention: Four of the SATA ports are provided by a Silicon Image SI3114 controller, which does something non-standard to the disks connected to it. What I have found is that with drives that I have previously used on the SI3114 the partition table is not recognized when you try to use them on a different SATA controller. This also happens if I try to use one of these drives in an external USB enclosure like the NexStar III SATA enclosure that I have. On the other hand, disks which were previously partitioned while connected on the nVidia chipset's built-in SATA controller is recognized on other motherboards and in my external enclosure without any problems. The reason why I mention this is that if the motherboard ever should fail, I will be in trouble because these controllers are not present on to many motherboards, and so, unless I can get another motherboard with the same type of controller, I would have a hard time reading the data from the disks that were connected to the SI3114 controller. I must admit that this is not something that I have been able to verify yet, but there is this risk. Note: All of the motherboard SATA ports are set to operate in JBOD mode, which allows me to select how I want to set up the RAID in software, and also allows me to change the RAID layout later. I also still have to make a final decision on what operating system to use. I would be a poor Solaris fan if I did not at least consider Solaris as an option, but in my experience it is just more suited to a server environment where the applications you want to use are unlikely to change much and all well supported under Solaris. OpenSolaris offers me the same capability to set up zones and ZFS file systems, but many applications are easier to get to run under OpenSolaris. Seeing that I will want to use the server for several other purposes besides simple file sharing and because I have not yet identified the actual software that I will use, I will eliminate Solaris at this point just because it will be too restrictive in future. If file serving was the only purpose, then it would have been perfect. I will also eliminate Windows because it is not free, too difficult to configure, doesn’t have enough free software, and is just generally a pain to work with. If I were to use Linux I would have trouble getting ZFS to work - the Linux ZFS port is not nearly stable enough for my liking. This means I will probably have to use OpenSolaris. The decision to use OpenSolaris is not yet cast in stone, but it looks like the right choice because I will be able to set up Zones/Containers and I will be able to use ZFS features like snapshots and raid-z. The four 160 GB disks that I have will nicely make up one raid-z "stripe". For the rest of the disks I will need to find an optimal way to configure them, but I still need to investigate the optimal way to set up the disk. I am also considering is to get the free Foundation Suite Basic disk management software as I know it very well and it too provides simple and online management of disk space. In part three of this series I will go in depth in how I selected an operating system for this file server.

Choosing between Solaris, Linux and MS-Windows

Today I will take this very contentious, highly flame-war-provoking issue head on. Many people are likely to disagree with me, but that is OK - opinions are a big part of what makes the Internet tick. Naturally you need to start off by considering your specific requirements and the resources that you have available: Many RISC based computers support only a very specific Unix operating system, while many consumer devices such as web-cams and WiFi adapters are difficult if not impossible to get to work with anything other than MS Windows. It will make sense to take stock of your existing skills to support that environment when choosing an operating system for a server application if multiple options exist. To start off with I would like to group operating systems into three categories the way I see it. These are intentionally vague, and operating systems can migrate between the categories. The categories are:

• The End-User operating system: The question "Does it do the job?" comes to mind. I will include MS Windows and MacOS into this category. For reasons I will get into soon, this includes the Server, Pro, and Home editions of MS Windows.
• The Power-user operating system: "I'll hack it until it works the way I want it to work". This category without any doubt includes Linux.
• The server operating system: This is for specifically supported hardware running specifically supported applications, and includes all the Uni'ces, in particular Solaris because it is available for free and on non-RISC hardware, and probably FreeBSD as well.

I'll discuss each of these categories a bit more in depth, taking them in the opposite order. In the enterprise data center we usually find many large, multi-processor servers designed and ratified to run specific operating systems, which in turn are ratified and supported for use with specific applications and databases. Vendors in this environment sell solutions where all the components making up the whole have been preselected, and include support for their solutions under a contract which often limits your freedom, for example by limiting you to specific versions of a specific operating system. In this environment, going with a supported operating system is the only sensible choice, and the big Unix vendors all have long lists of applications that can be run on their systems. The benefits of going with these types of solutions lies in that the hardware is intelligent about the operating system, being able to halt the operating system without terminating it, allowing hardware to be re-configured while the operating system is running, is able to take crash-dumps when the system hangs and presents a uniform device tree of the installed hardware to the operating system. This all provides a high level of stability and supportability, something which comes at a price, but worth it when the service delivery to your clients depends on the systems running without interruptions. That is not to say that Linux and MS Windows servers do not have a place in the data center - the exact same rule holds true: Select the platform for which your application is supported - whether it is a mainframe or a network vendor's appliance with an embedded firewall and authentication service. With Linux-based servers I often notice that the applications prefer an environment where a high degree of customizability is provided, and with Windows servers I often find that the applications have a keep-it-simple nature, at least as far as the server configuration is concerned. Linux fits squarely into the Power-user operating system category. Features like the ability of the user to re-compile the operating system to eke out an extra ounce of performance, directly modify the source code to get a specific result, the ability to change and/or replace any single component with one of your own choosing, and a wonderful following of supporters eager to help out one another truly makes this operating system into a power-user's heaven. I find myself more productive when working on Linux: this is without a doubt because I have it set up exactly the way I like it and I have a huge collection of software tools installed and configured to help me maintain the system. Also something that helps is that Linux is much more transparent in reporting what is going on "under the hood" - You can obtain accurate log files of system events, examine what program processes are doing in fine detail, and you have accurate reports of the state of every single component that the operating system controls. This all makes it easier to track down the "root cause" of problems, and in my experience with Linux, using the computer is less of a hit-and-miss affair - if it works a certain way today, it will work the same way tomorrow and the day after. Things like program crashes are much more easily reproducible, and as a result, easier to resolve. All of this requires a certain level of commitment and even enthusiasm to learn more about the workings of the computer and the operating system though, and that is probably not for everyone. The other options available to "power users" with PCs are basically FreeBSD, Solaris, Open Solaris, and Windows. Windows is sometimes not transparent enough and things do not always work the way Microsoft says it does. Solaris' hardware support is still somewhat limited, making it a difficult to justify choice. Open Solaris is maybe still too much of a newcomer to be judged fairly, tough good work is being done, albeit slowly, by projects such as Nexenta GNU Solaris, an OpenSolaris distribution providing the GNU tools in stead of the Solaris defaults. The End-user category are generally for people who don't care that their operating system limits their ability to troubleshoot faults, learn more about the computer or replace sub-components of the operating system. Many of these are company PCs used to run a selection of applications needed to run their business, the support and development of which companies have invested a lot of time and money into. 20 years ago Microsoft was probably deliberately lax in prosecuting pirates of their operating systems and application software. Most of these "pirates" were kids who were learning how computers worked, and by the time they made it into the workforce, Windows was what they expected on their computers and in the data centers. Today this large following of Windows users equates to a workforce who will pretty much need to be retrained if one suddenly wanted to switch to a different operating system. You could argue that the training is not so hard - many applications are web based these days, sending an email remains pretty much the same, etc, but this only holds true in theory: The moment many users are faced with a new login screen that looks different, they feel lost. Suddenly the user name field becomes case sensitive, and your lost user turns into a frustrated user who hasn't even had a chance to try the applications yet. Worse, every application's menus are laid out differently, and that is only once they learn which application does what. Space characters in file names suddenly cause havoc, and the slash in directory paths is turned around. To the end user, it feels as if nothing works the way they are expecting it to work and it takes months before they once more become productive. I must agree with A Russel Jones who concludes that for people who don't want to have to consider whether their applications will work with the window manager that they have installed, learn how to enter commands in terminal, or why security is a good thing when it seems to just make life harder, a Windows based PC is often the best option. Linux can be carefully configured to simulate this in a controlled environment where an IT department makes all the design decisions, installs all the software, and makes sure that the all of the hardware in use is supported - when you have people with the necessary skills. Setting up such an ideal configuration is something that will take time and effort, and that means there is a price tag attached, and this could be more expensive than just running Windows. In the end business decisions always come back to the numbers: What are the benefits (over the alternative), and how does the long-term costs of the various options compare. The Pro and Server editions of MS Windows have the same basic limitations that the Home edition have: The server edition is no more stable, powerful, transparent, manageable, or secure than the Home edition, and this puts it into the "End-user" category. It also fits in with the "keep-it-simple" tendency of this category, even if it is installed on a data center rack mounted server which never crashes. The power user is probably the best off: They have the most freedom to choose what operating system and applications they want to run, how they want to configure their computers, and can usually make their own choices about what programs to use. This is basically because they are able to support themselves when things stop working. So basically it boils down to the age-old adage: Use the right tool for the job. I suspect that you will find that nine times out of ten the categories I described here will hold true though. Disclaimer: This article necessarily relies heavily on generalization. Please keep that in mind if you decide to comment.