Initial Program Load

Maintaining the Linux device driver code base

2011-03-30T12:23:00.003+02:00

After a (sadly) failed attempt to convert my significant other to Linux, I had a discussion with her about why it failed. Root cause.

Her computer works well with Windows, not at all with Linux. The reason is that her laptop will display no better than 800x600 resolution as there is no good SIS671 graphics driver for Linux (and there is for Windows). Nothing recent, functional, supported, viable or workable.

Why isn't there one for Linux?

Because it doesn't make money. Business is the process of converting time into money. Sales people get customers to buy a product or service. Technical people produce the products or deliver the services. Management and administration functions supported and enables the business to operate as a whole. (Or so the theory goes, but that is another story)

And because programmers also need a place to live. And to feed the kids.

There is cost involved – an investment, and there is a price, the return on the investment. A product, in this case a device driver for the graphics processor, needs to be designed, produced and supported. The technical people and the tools they need to do this do not come cheap.

Device drivers for Linux, however, does not make much, if any money for the companies involved. People do not pay for device drivers, rather they (rightly) expect it to be included in the cost of the hardware.

Even closed source Linux drivers are free – the vendor have to cover the costs through the sales of hardware. But the business model is flawed – The cost to deliver the Linux device driver far exceeds the income generated from hardware sales to Linux users. Thus this expense must be subsidised from sales to Windows users.

Unless the Linux user base grows to reach a critical mass, the point where enough Linux users buy the hardware to be able to justify the cost of the driver development and support, the situation will not change.

The above situation is the same, no, actualy worse for other hardware – Webcams, GPS'es, Cell-phones, USB thumb drives, bluetooth hardware, Wi-Fi and network cards, Fingerprint readers and touch-pad input devices. Every single bit of hardware.

The Linux kernel includes almost all device drivers for the hardware because of this situation. It is the only way the Linux community can use most of the consumer hardware available in the world today – that is, by developing the needed device drivers themselves.

As a result Linux supports much much more hardware than Windows does. Windows depends on the driver disks that ships with the hardware because Microsoft does not provide driver software for every bit of hardware out there!

The more you think about it, the more you realise just how unbalanced the situation really is! Microsoft sells its Windows operating system with only basic device drivers included – for proper functionality, features and performance, you need to load the hardware manufacturer's drivers. The hardware manufacturers provide the device drivers because otherwise they would lose the majority of their market – Windows users.

The Linux community, an entity that makes no money, needs to provide device drivers created through donated effort. I am aware of the exceptions, but that does not change the overall picture. The effort to maintain and update the base of device drivers included in the Linux kernel increases as the number of pieces of hardware to be supported increases. In other words: Every time a new piece of hardware appears in the shops.

To add insult to injury, the Linux community locks themselves in with the GPL license, which means they can not, for example, utilise and share effort by other Unix or BSD distributions because the Linux kernel enforces the use of the restrictive GPL.

Even worse, a Linux device driver works only on a specific release of the kernel. This is because the kernel interfaces for device drivers changes, and as a result the device driver needs to be re-compiled for every update, even minor updates, to the kernel. The amount of extra work this would place on hardware manufacturers to ensure that their device driver works on every kernel version is significant, and much more than what is needed for, for example, Windows or Solaris.

The long and the short of it is that to produce and maintain device drivers for Linux is prohibitively expensive, and the market loss as a result of not supporting Linux users is essentially negligible to most hardware manufacturers' bottomline!

Regarding the market share situation: I have long held the belief that through “allowing” us to copy Windows, Bill Gates got the world to using MS Windows. It is what most people grew up with on our computers at home, and what we as a result expected when we entered the workplace. More than just the majority of the work force, today's computer gamer is tomorrow's IT business decision maker.

But there is some light on the horizon: The Wayland Display Server may just give the Linux graphics stack the performance boost it needs to make it a viable gaming platform, which in turn will gain it the adoption of many gamers, and in the long run more market share on the desktop. Now if only Linus would fix the device driver ABIs and APIs to make it that bit easier for hardware manufacturers to support their device driver software on Linux...

There is a lot of fud on the net about how the "deliberately dynamic ABIs" of the Linux kernel makes Linux drivers better maintained, less buggy, etc. Sigh.

Live Upgrade to install the recommended patch cluster on a ZFS snapshot

2011-02-17T15:49:00.005+02:00

Live Upgrade used to require that you find some free slices (partitions) and then fidget with the -R "alternate Root" options to install the patch cluster to an ABE. With ZFS all of those pains have just ... gone away ...

Nowadays Live Upgrade on ZFS don't even copy the installation, instead it automatically clones a snapshot of the boot environment, saving much time and disk space! Even the patch install script is geared towards patching an Alternate Boot Environment!

The patching process involves six steps:

Apply Pre-requisite patches
Create an Alternate Boot Environment
Apply the patch cluster to this ABE
Activate the ABE
Reboot
Cleanup

Note: The system remains online throughout all except the reboot step.

In preparation you uncompress the downloaded patch cluster file. I create a zfs file system and mounted it on /patches, and extracted the cluster in there. Furthermore, you have to read the cluster README file - it contains a "password" needed to install, and information about pre-requisites and gotches. Read the file. This is your job!

The pre-requisites are essentially just patches to the patch-add tools, conveniently included in the Patch Cluster!

Step 1 - Install the pre-requisites for applying the cluster to the ABE

# cd /patches/10_x86_Recommended
# ./installcluster --apply-prereq

Note - If you get an Error due to insufficient space in /var/run, see my previous blog post here!

Step 2 - Create an Alternate boot environment (ABE)

# lucreate -c s10u9 -n s10u9patched -p rpool

Checking GRUB menu...
Analyzing system configuration.
No name for current boot environment.
Current boot environment is named <s10u9>.
Creating initial configuration for primary boot environment <s10u9>.
The device </dev/dsk/c1t0d0s0> is not a root device for any boot environment; cannot get BE ID.
PBE configuration successful: PBE name <s10u9> PBE Boot Device </dev/dsk/c1t0d0s0>.
Comparing source boot environment <s10u9> file systems with the file
system(s) you specified for the new boot environment. Determining which
file systems should be in the new boot environment.
Updating boot environment description database on all BEs.
Updating system configuration files.
Creating configuration for boot environment <s10u9patched>.
Source boot environment is <s10u9>.
Creating boot environment <s10u9patched>.
Cloning file systems from boot environment <s10u9> to create boot environment <s10u9patched>.
<B>Creating snapshot</B> for <rpool/ROOT/s10_0910> on <rpool/ROOT/s10_0910@s10u9patched>.
<B>Creating clone</B> for <rpool/ROOT/s10_0910@s10u9patched> on <rpool/ROOT/s10u9patched>.
Setting canmount=noauto for </> in zone <global> on <rpool/ROOT/s10u9patched>.
Saving existing file </boot/grub/menu.lst> in top level dataset for BE <s10u9patched> as <mount-point>//boot/grub/menu.lst.prev.
File </boot/grub/menu.lst> propagation successful
Copied GRUB menu from PBE to ABE
No entry for BE <s10u9patched> in GRUB menu
Population of boot environment <s10u9patched> successful.
Creation of boot environment <s10u9patched> successful.

There is now an extra boot environment to which we can apply the Patch Cluster. Note - for what it is worth, if you just needed a test environment to play in, you can now luactivate the alternate boot environment and then make any changes to the active system. If the system breaks, all it takes to undo any and all changes is a reboot.

Step 3 - Apply the patch cluster to the BE named s10u9patched.

# cd /patches/10_x86_Recommended
# ./installcluster -B s10u9patched

I am not showing the long and boring output from the installcluster script as this blog post is already far too long. The patching runs for quite a while, plan for at least two hours. Monitor the process and check the log for warnings. Depending on how long it has been since the last patches were applied, some severe patches may be applied which can affect your ability to login after rebooting. Again: READ the README!

Step 4 - Activate the ABE.

# luactivate s10u9patched
System has findroot enabled GRUB
Generating boot-sign, partition and slice information for PBE <s10u9>
A Live Upgrade Sync operation will be performed on startup of boot environment <s10u9patched>.

Generating boot-sign for ABE <s10u9patched>
Generating partition and slice information for ABE <s10u9patched>
Copied boot menu from top level dataset.
Generating multiboot menu entries for PBE.
Generating multiboot menu entries for ABE.
Disabling splashimage
Re-enabling splashimage
No more bootadm entries. Deletion of bootadm entries is complete.
GRUB menu default setting is unaffected
Done eliding bootadm entries.

**********************************************************************

The target boot environment has been activated. It will be used when you
reboot. NOTE: You MUST NOT USE the reboot, halt, or uadmin commands. You
MUST USE either the init or the shutdown command when you reboot. If you
do not use either init or shutdown, the system will not boot using the
target BE.

**********************************************************************

In case of a failure while booting to the target BE, the following process
needs to be followed to fallback to the currently working boot environment:

1. Boot from the Solaris failsafe or boot in Single User mode from Solaris
Install CD or Network.

2. Mount the Parent boot environment root slice to some directory (like
/mnt). You can use the following commands in sequence to mount the BE:

zpool import rpool
zfs inherit -r mountpoint rpool/ROOT/s10_0910
zfs set mountpoint=<mountpointName> rpool/ROOT/s10_0910
zfs mount rpool/ROOT/s10_0910

3. Run <luactivate> utility with out any arguments from the Parent boot
environment root slice, as shown below:

<mountpointName>/sbin/luactivate

4. luactivate, activates the previous working boot environment and
indicates the result.

5. Exit Single User mode and reboot the machine.

**********************************************************************

Modifying boot archive service
Propagating findroot GRUB for menu conversion.
File </etc/lu/installgrub.findroot> propagation successful
File </etc/lu/stage1.findroot> propagation successful
File </etc/lu/stage2.findroot> propagation successful
File </etc/lu/GRUB_capability> propagation successful
Deleting stale GRUB loader from all BEs.
File </etc/lu/installgrub.latest> deletion successful
File </etc/lu/stage1.latest> deletion successful
File </etc/lu/stage2.latest> deletion successful
Activation of boot environment <s10u9patched> successful.

# lustatus
Boot Environment           Is       Active Active    Can    Copy
Name                       Complete Now    On Reboot Delete Status
-------------------------- -------- ------ --------- ------ ----------
s10u9                      yes      no     no        yes    -
s10u9patched               yes      yes    yes       no     -

Carefully take note of the details on how to recover from a failure. Making a hard-copy of this is not a bad idea! Take note that you have to use either init or shutdown to effect the reboot, as the other commands will circumvent some of the delayed action scripts! Hence ...

Step 5 - Reboot using shutdown or init ...

# init 6

Monitor the boot-up sequence. A few handy commands while you are performing the upgrade, includes:

# lustatus
# bootadm list-menu
# zfs list -t all

You will eventually (after confirming that everything works as expected) want to free up the disk space held by the snapshots. The first command cleans up the redundant Live Upgrade entries as well as the relevant ZFS snapshot storage! The second is to remove the temporary ZFS file system used for the patching.

Step 6 - Cleanup

# ludelete s10u9
# zfs destroy rpool/patches

Again no worries about where the space comes from. ZFS simply manages it! Live Upgrade takes care of your grub boot menu and gives you clear instructions on how to recover it anything goes wrong.

Adding a ZFS zvol for extra swap space

2011-02-17T11:27:00.005+02:00

ZFS sometimes truly takes the think work out of allocating and managing space on your file systems. But only sometimes.

Many operations on Solaris, OpenSolaris and Indiana will cause you to run into swap space issues. For example using the new Solaris 10 VirtualBox appliance, you will get the following message when you try to install the Recommended Patch Cluster:

Insufficient space available in /var/run to complete installation of this patch
set. On supported configurations, /var/run is a tmpfs filesystem resident in
swap. Additional free swap is required to proceed applying further patches. To
increase the available free swap, either add new storage resources to swap
pool, or reboot the system. This script may then be rerun to continue
installation of the patch set.

This is fixed easily enough by adding more swap space, like this:

# zfs create -V 1GB -b $(pagesize) rpool/swap2
# zfs set refreservation=1GB rpool/swap2
# swap -a /dev/zvol/dsk/rpool/swap2
# swap -l
swapfile             dev  swaplo blocks   free
/dev/zvol/dsk/rpool/swap 181,2       8 1048568 1048568
/dev/zvol/dsk/rpool/swap2 181,1       8 2097144 2097144

Setting the reservation is important, particularly if you plan on making the change permanent, eg by adding the new zvol as a swap entry in /etc/vfstab. ZFS does not reserve the space for swapping otherwise, so the swap system may think there is space which isn't actually there if you don't do this.

The -b option sets the volblocksize to improve swap performance by aligning the volume I/O units on disk to the size of the host architecture memory page size (4 KB on x86 systems and 8KB on SPARC, as reported by the pagesize command.)

If this is just temporary, then cleaning up afterwards is just as easy:

# swap -d /dev/zvol/dsk/rpool/swap2
# zfs destroy rpool/swap2

It is also possible to grow the existing swap volume. To do so, set a new size and refreservation for the existing volume like this:

# swap -d /dev/zvol/dsk/rpool/swap
# zfs set volsize=2g rpool/swap
# zfs set refreservation=2g rpool/swap
# swap -a /dev/zvol/dsk/rpool/swap

And finally, it is possible to do the above without unmounting/remounting the swap device, by using the following "trick":

# zfs set volsize=2g rpool/swap
# zfs set refreservation=2g rpool/swap
# swap -l | awk '/rpool.swap/ {print $3+$4}'|read OFFSET
# env NOINUSE_CHECK=1 swap -a /dev/zvol/dsk/rpool/swap $OFFSET

The above will calculate the offset in the swap device and add a new "device" to the list of swap devices. This will automatically use the added space in the zvol. The Offset will be shown as the "swaplo" value in swap -l output. Multiple swap devices on the same physical media is not ideal, but on the next reboot (or by deleting and re-adding the swap device) the system will recognise the full size of the volume.

No worries about where the space comes from. ZFS just allocates it! The flip side of the coin is that once you have quotas, reservations, allocations, indirect allocations such as from snapshots, figuring out where your space has gone can become quite tricky! I'll blog about this some time!

Useless Performance Comparisons

2010-12-06T17:52:00.003+02:00

The point of performance comparisons or benchmark articles has to be purely sensational. By far the most of these appear to have little value other than attracting less educated readers to the relevant websites.

In a recent article Michael Larabel of Phoronix reports on the relative performance of various file systems under Linux, specifically comparing the traditional Linux file systems to the new (not yet quite available) native ZFS module. According to the article ZFS performs slower than the other Linux file systems in most of the tests, but I have a number of issues with both how the testing was done and with how the article was written.

Solaris 11 Express should have been included in the test, and the results for OpenIndiana should be shown for all tests. It is crucial that report include other system metrics such as CPU utilization during the test runs.

I also have some even more serious gripes. In particular the blanket statement that some unspecified “subset” of the tests were performed on both a standard SATA hard drive and the SSD drive, but that the results were “proportionally” the same – does not make sense as some tests are more seek latency sensitive than others, and some file systems hide these latencies better than others.

Another serious gripe is that there is no feature comparison. More complex software has more work to do, and one would expect some trade-offs.

Even worse: two of ZFS’s strengths were eliminated by the way the testing was done. Firstly when ZFS is given a “whole disk” as is recommended in the ZFS best practices (as opposed to being given just a partition) it will safely enable the disk’s write cache. It only does this if it knows that there are no other file systems on the disk, i.e when ZFS is in control of the whole disk. Secondly ZFS manages many disks very efficiently, particularly as far as is concerned allocating space: ZFS performance doesn't come into its own right on a single disk system!

Importantly, and especially so since this is very much a beta version of a port of a mature and stable product, we need to understand which of ZFS's features are present, different and/or missing compared to the mature product. For example some of ZFS’s biggest performance inhibitors under FUSE is that it is limited to a single-threaded ioctl (Ed: Apparently this is fixed in ZFS for Linux 0.6.9, but I am unable to tell whether this is the version Phoronix tested) - and not having access to the disk devices at a low level. The KQ Infotech website lists some missing features, particularly interesting is the missing Linux async I/O support. Furthermore the KQ Infotech FAQ states that Direct IO falls back to buffered read/write functions and that missing kernel APIs are being emmulated though the "Solaris Porting Layer".

A quick search highlights some serious known issues, such as the Linux VFS Cache and ZFS ARC cache copy duplication bug, a bug which heavily impacts on performance.

More information about missing features can be found on the LLNL issue tracker page.

If nothing else, the article should mention the fact that there are known severe performance issues and feature incompleteness with the Linux native ZFS module! The way in which Linux allocates and manages virtual address space is inefficient (don't take my word for it, see this and this), requiring expensive workarounds.

Besides all of this my real, main gripe is about this kind of article in general. The common practice of testing everything with “default installation settings” implies that nothing else needs to be done - however when you want the absolute best possible performance out of something, you need to tune it for the specific workload and conditions. In the case of the article in question, the statement reads “All file-systems were tested with their default mount options”, and no other information is given, such as whether the disk was partitioned, whether the different subject file systems where mounted at the same time, what disk the system was booted from and whether the operating system was running with part of the disk hosting the tested file system mounted as its root. We don’t even know whether the author read the ZFS Best Practices Guide.

It can be argued that the average person will not tune the system, or in this case the file system, for one specific workload because their workstation should be an all-round performer, but you should still comply with the best practices recommendations from the vendors, especially if performance is one of your main criteria.

I don’t know whether using defaults is ever acceptable in this kind of article. My issue stems from how these articles are written in a way that suggests that performance is the only or at least the most important factor in choosing an operating system, or file system, of graphics card, or CPU or whatever the subject is. If that were true then at least the system should be tuned to make the most of each of the subject candidates, whether these are hardware or software parts being tested and compared to one another. This tuning is often done by disabling features, configuring the relevant options, and usually to get it right you would need to have someone who is a performance expert on that piece of software or hardware to optimize it for each test.

Specific hardware (or software) often favor one or the other of the entrants. An optimized, feature poor system will outperform a complex, feature rich system on limited hardware. Making the best use of the available hardware might mean different implementation choices when optimizing for performance rather than for functionality or reliability. ZFS in particular comes into its own right, both in terms of features and performance, when it has underneath it a lot of hardware – RAM, disk controllers, and as many dedicated disk drives as possible. The other file systems have likely reached their performance limit on the limited hardware on which the testing was done. Linux is particularly aimed at the non-NUMA, single-core, Single hard drive, single user environment. Solaris, and ZFS, was developed in a company where single user workstations were an almost non-existing target, the real target of course being the large servers of the big corporates.

As documented in the ZFS Evil Tuning Guide, many tuning options exist. One could turn off ZFS check-sum calculations, limit the ARC cache sizes, set the SSD disk as a cache or log device for the SATA disk, and set the pool to cache only meta data, to mention a few. Looking at the hardware available in the Phoronix article, the choices would depend on the specific test – in another test one might stripe between the SATA disk and the SSD disk, in another you might choose to mirror across the two.

The other file system candidates might have different recommendations in terms of how to optimize for performance.

I realize that the functionality would be affected by such tuning, but the article doesn’t look at functionality, or even usability for that matter. ZFS provides good reliability and data integrity, but only in its default configuration, with data check-summing turned on. The data protection levels and usable space in each test might be different, but that again is a function of which features are used and not the subject of the article, not even mentioned anywhere in the article.

As a point in case for the argument about functionality, once needs to consider all that ZFS is doing in addition to being a POSIX compliant file system. It replaces the volume manager. It adds data integrity checking through check-summing. It manages space allocation, including space for file systems, meta-data, snapshots, and ZVOLs (virtual devices created out of a ZFS pool) automatically. Usage can be controlled by means of a complete set of reservation and quota options. Changing settings, such as turning on encryption, the number of copies of data to be kept, whether to do check-summing, etc is dynamic. There is much more as Google will tell.

And just to add insult to injury, the article goes and pits XFS against ZFS, ignoring the many severe reliability issues present with XFS, such as the often reported data corruption under heavy load and severe file system corruption when losing power.

I would really like to see a performance competition one day. The details of how the testing will be done will be given out in advance to allow the teams to research it. Each team is then given access to the same budget from which to buy and build their own system to enter into the competition. Their performance experts then set up and build the systems, and install the software and tune it for the tests on the specific hardware they have available. One team might buy a system with more CPUs while another might buy a system with more disks and SCSI controllers, but the test is fair (barring my observation about how feature poor systems will always perform better on a low-budget system) because the teams each solves the same problem with the same budget. The teams submit to the judges their ready systems to run the performance test scripts and publish their configuration details in a how-to guide. To eliminate cheating, an independent group will try to duplicate the team’s test results using the guide.

I think this would make a fun event for a LAN party – any sponsors interested?

Lost Dog!

2009-05-10T19:20:00.005+02:00

Otto, our Dachhund cross got lost yesterday. If someone found such a dog in the area near the Stellenberg High School and did a Google search I hope that they will hit on this page. Otto is a little brown dog with realy big ears. The dog is my son, Francois' best friend, so it would be realy terrible if we never were to find it again. For the record, we live in Amanda Glen, but the dog could easily walk into the Sonstraal Heights or Stellenberg area. If you saw this dog, please call Johan at 021 910 7160 or Reinette at 021 976 3453. Thank you!

ZFS user quotas available in SNV build 114

2009-04-26T17:02:00.001+02:00

I noted, as per Chris Gerhard's Weblog that user and group Quotas on ZFS will be available soon - the fix to bug ID 6501037 is currently slated for inclusion in ON build 114.

Once this becomes available I will have one fewer item on my list of features missing from ZFS.

Currently to limit users' consumption the workaround documented here is to provide each user with a dedicated directory on which another dataset is mounted and a quota is set. This implies that the user can only create or write to files in that specific directory. To track and limit a user's total usage across an entire ZFS pool requires User quotas - ditto for consumption by group.

According to this post by Matty the feature is implemented in a way which enforces the rule "tardily", that is it is a little "late", and also mentions that translated SIDs (eg when the directory is shared via SMB) are supported.

The PSARC/2009/204/ document here provides details of how the quotas is implemented. Two new zfs subcommands, namely zfs userspace and zfs groupspace reports the consumption, and control is by means of a set of new properties on ZFS file system datasets.

This amounts to good news all around. Maybe I should start tracking bug IDs for all of the items on my feature wish-list!

Oracle becomes the second "IBM"

2009-04-23T19:30:00.002+02:00

As promised, my thoughts on the merger with Oracle.

Disclaimer: We know that the deal has not yet been finalized yet, and these are my personal opinions and thoughts on the matter.

I have read many forum posts about the merger, many of them very negative about the whole deal. Whether it is people fearing that Oracle will kill off this or that product, or people who feel that it is good riddance to Sun, these all indicate a serious level of misunderstanding about the IT industry as a whole, about what Open Source software is, and mostly what Sun is about.

On a personal level the merger scares me: Change is always stressful, and we love the culture at Sun. Sun's culture of allowing the Engineers a virtual free hand in designing products is what drives the innovation and new features.

Sun's products really represent a level of innovation and quality that is hard to match. The cost of Sun's servers and storage products are often said to be too high, but when doing a like for like comparison, Sun products at the same price as those of the competition have better performance, features, power consumption, rack density, upgradability, investment protection, manageability and build quality. I know many of Sun's products well because of the way we work in the SSA (Sub Saharan African) region - The engineers here all support all of Sun products, bar none.

A little bit more about that: The engineers in this region must handle calls, ie analyze, determine fault cause, and often implement the solution (Though this may change with a split in the team having been proposed). The products we support include software and hardware, everything from NAS gateways to Cluster software, and includes many products that are virtually unknown outside of their niche markets.

What enables us to support such a wide variety of products is the quality of the products: They work as documented, and the documentation is available. In addition we have access to the engineering teams and interest groups for discussing unusual problems.

What I am trying to get at is that I have personally dealt with a wide variety of Sun's products, and everywhere I look I see supportability through enterprise level maturity. People who bash the products have had a single bad experience, and unfortunately it is human nature to base opinions on your bad experiences, and not notice when things go well.

Sun's product line includes Storagetek's tape libraries and VTL / VSM mainframe products. It includes the Fujitsu based M9000-64. It includes the Constellation blades and switches, and little servers like the T2000 and even many smaller, though older V210s and V240s that are still being used. In January I had to replace an EEPROM chip on an Ultra 5 on a scientific vessel in the Cape Town harbor!

Sun also have a major investment in SPARC processors. No, they are not the fastest number crunchers, in fact the UltraSPARC processors are quite slow when compared to the fastest processors from the competition. But scaling to hundreds of CPUs is mature technology in the SPARC camp. Multi-core CPU technology: Mature. Multi-terabyte RAM in a system: Mature technology. NUMA? Mature technology. 64-bit processors? Those came out, what, 12 years ago. Systems with over 700 GB/sec internal bus bandwidth? We got it. Adding memory or CPUs to a running system. Mature, and available for 10 years already. You can even remove those components and repair or replace them without stopping the OS, though it does require that you configure the server correctly.

IBM in comparison have a slightly wider footprint in the server hardware arena: Their systems include laptops and Mainframes, and essentially everything in between. But their software product set is not as broad as that of Sun, which includes products like the Luster file system, SunRay software, Java, StarOffice and OpenSolaris, etc etc etc.

What happens when Oracle suddenly attaches all of Sun's products to its own portfolio ... ?

In the enterprise market there are realy only two databases: DB2 and Oracle. I am well aware that PostgreSQL and MySQL and YourFavourtiteDB also have a meaningful place in the market, but those are all to a large extent "alternatives to" using Oracle or DB2.

At present Oracle is not a huge competitor to IBM, except for the Database itself. But when Oracle suddenly adds all of Sun's products to its arsenal, it turns into a different beast. Oracle becomes the second "IBM".

I must say that Oracle is unlikely to kill of many products. They may sell a few products, and I think it would be interesting to see what goes. Funding of some products may stop, but that does not need to mean the end of Open Source products.

One of my biggest gripes with Sun's uninformed critics is how they in the same forum post complain that Oracle will kill of their favourite open source application, and right after that complains about how Sun's products aren't open enough.

Listen to me: The mere fact that you worry about OpenOffice or MySQL or whatever dispearing means you admit what a large contribution Sun is making to your world.

I don't know exactly what the merger will mean for the IT industry as a whole. If Oracle changes the Sun culture to disallow the engineers from being innovative, we will see some of the competition in the market disappear over time. If Oracle sells off some products, they may get new life in another stable or they may disappear, we don't know. But every product that does disappear is a sad case and will be mourned.

If you don't think Sun's products are good, it just shows how little you know. I hope that Oracle will give new life and funding to Sun's R&D. If the Sun culture disapears, I will blame it on bad marketting that failed to turn good products into income, which ultimately resulted in Sun's demise ... but before I make any judgement calls prematurely, let's rather wait and see what happens. Oh and yes, I really do love Sun's products, especially Solaris and the servers.

Voting day in the South African National Election, 2009

2009-04-22T17:26:00.002+02:00

People who know me know that I am strongly apolitical. The problems we are facing in South Africa, especially the living conditions affecting most South Africans, will not be solved by any political party. They simply are not motivated or even enabled to fix things in a satisfactory manner. In short, I believe that it makes no difference who becomes our next Government.

However, like in every one of the past five national elections (since I became eligible to vote), I today cast my vote for an opposition party because I also believe in a multi-partite government: No one party should be allowed to rule unchecked and uncontested. In much the same way that competition is necessary in any industry, opposition serves as a check to keep the ruling party honest: An only incumbent gets a default Carte Blanche, even in a democracy. Very few people are disciplined enough to remain altruistic throughout a dictatorship, if ever they were - I certainly don't believe any of the people on our existing political radar have this ability, regardless of their good intentions.

Strangely this touches on the Sun/Oracle merger, one which many fear would kill of a number of products (and thus the competition) which do not fit in with Oracle's current business model. I'll post on that tomorrow after reading some of the Oracle employees' views on the matter.

Neat way to prevent multiple instances of a script

2008-11-07T12:49:00.004+02:00

Sometimes you need to ensure that you can never have more than one instance of a script running at the same time. This is especially important with scripts that modifies files, and if the script runs for longer than a fraction of a second it becomes more critical. Also if you have many people administrating a system, it becomes more important to ensure they don't step on each other's toes.

Basically the problem is known as the multiple writers problem, and it is solved by something called "semaphores". "Semaphores" is a feature implemented in the kernel with the purpose of providing a way to guarantee that a piece of code be made "mutually exclusive". I don't want to go into this, it is already properly explained on many websites, but have a look at the Wikipedia article if you are interested in the topic.

One technique to get around the problem of multiple instances of a script is to use the existence of a specific file somewhere to flag other instances of the script that there is already a running instance. The "touch" command does not complain about existing files, so you need to to check first whether the file exists already, exit if it does, and create it otherwise. For example

if [ -f /tmp/already_running ] 
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file /tmp/already_running and try again.
   exit 1
else
   touch /tmp/already_running
fi
....

Near the end of this script it is common to delete the file for the next use. This, however, is not the best solution.

If two instances of the script were started at nearly the same instant, what could happen is that instance 1 checks the file, finds it does not exist, but then gets kicked off the CPU so that instance 2 can run. Instance 2 then checks the lock file and also sees it is OK to continue, and then creates the lock. Instance 1 eventually gets CPU time again and, having already previously checked the lock file, believes it is safe to continue running.

This is known as a race condition, and is by definition what Semaphores are meant to prevent. But semaphores are not easily accessible in scripts, or so it might seem.

Now I know you are asking "but what is the chance of such a precice timing of sceduled cpu time to cause this kind of race condition". Yes, the chances are probably low, especially if you are the only person using a specific script. However there is a proper way of checking that we are the only instance running, and it is even simpler to implement thatn the check-file-touch-file method!

The secret lies in the mkdir command.

if ! mkdir /tmp/already_running
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file /tmp/already_running and try again.
   exit 1
fi
....

mkdir will automatically use the kernel built-in semaphores during the actual process of creating the entry in the file system. It will fail if the directory exists, and on succesfull return, the lock will be in place already, so no extra commands are needed to complete the mutex locking process.

The second part of this is to automate the release of the lock when the script exists. Typically you want the lock to be released even if someone kills the script, press Ctrl-C, or if it terminates normally or on an error.

This is done by means of a EXIT trap, and the format of using traps in bourne shell variants is:

trap "do-something-here" EXIT

This trap must be set AFTER obtaining the lock, otherwise a second instance of the script will "inadvertendly" remove the lock obtained by the first instance of the script (because the new instance will basically remove the lock which it was not able to obtain when it exists on not being unable to obtain the lock.

You obviously don't have to have an if-then-fi to print a message to the user - if you are the only person using a script, you can simplify the checking of the lock as follow:

MUTEX_LOCK=/tmp/myscript_already_running
mkdir $MUTEX_LOCK || exit 1
trap "rmdir $MUTEX_LOCK" EXIT

With the above you will simply have an error message from mkdir which you need to interpret as "the script is already running", eg:

mkdir: Failed to make directory "/tmp/myscript_already_running"; File exists

Using this technique, a whole script might look like this:

#!/bin/ksh
# This is myscript v1.0.

#Set up the running environment
MUTEX_LOCK=/tmp/myscript_already_running
...

if ! mkdir $MUTEX_LOCK
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file $MUTEX_LOCK and try again.
   exit 1
fi
trap "rmdir $MUTEX_LOCK" EXIT
...

# Doing work which requires only one instance of the script to be running
...

# THE END

Note there is no "remove lock" statement at the end of the script. This is handled by the trap, which executes on any exit, except of course a kill -9.

Using a kill -9 should in any case only ever be used as a last resort, because it does not allow the program to clean up after itself.

Why X-windows is back-to-front

2008-10-21T22:00:00.003+02:00

This is a very basic introduction to the X-windows protocol, with the purpose of explaining why the server runs on the workstation and the client runs on the server.

Actually the naming is the right way round. Per definition clients initiate connections and servers accept connections. Without trying to be pedantic or philosophical about the difference between a server and a workstation, lets just say that for the purpose of this discussion your desktop machine is the workstation, and the server is some system "service" applications, files, printers, etc from your computer room. When you start an X-windows application on the server, it needs to open a window somewhere. In this case, that somewhere is the screen of your workstation. The X-windows application, really the X-client, will open a TCP/IP connection to port 6000 on your workstation, and then via this session it will send "instructions" to the X-server on how to draw its window.

Clearly something needs to be running on your workstation to accept a connection on port 6000. This piece of software is called the X-server, and such programs are available for most if not all operating systems. In particular it is available for MS Windows and Mac-OS, and Linux and Solaris includes X-server software in the form of Xfree86, Xorg and Xsun.

The X server actually listens on port 6000+N where N is the "Display" or "instance" number. Thus, the first server is Display 0, and listens on TCP/IP port 6000. A second Display would listen on port 6000+1 = 6001, and so on, though having more than one is not particularly common.

There are a few things which seemingly complicates this matter. The first is that in many situations the X-server and the X-client runs on the same system. Essentially this is true for all X applications used on a "local workstation". But the rule still holds - port 6000 listens and accepts connections from the clients running locally. X-clients, which can be anything from Firefox to Gnome-terminal, get a DISPLAY environment variable which tells them to connect to "localhost:0.0"

The second factor is how the application is started. It is possible to walk to the server, enter the command to set the DISPLAY variable (to point to the IP address of your workstation, then start the X-client application, and when you return to your workstation you will find that it is showing the X-application (assuming that the X-server on your desktop is running and accepting connections). However it is much more convenient and more common to log into the server remotely and then set the DISPLAY variable and then start the X-client.

If you use a strategy such as the above often it quickly becomes too much effort and a way to automate the DISPLAY environment variable setting becomes neccesary. Many X-server programs include a few "automation" settings, which often includes connecting to the remote server via rsh, telnet or ssh, logging in, setting the DISPLAY variable, and then starting an X-client program such as a terminal.

In addition SSH has got some special features whereby it can tunnel the X-server setting back to your PC over the encrypted session. This is particularly handy when you have a firewall blocking incomming connections on port 6000. When starting your SSH client up to do this, the client will ask the SSH server to start listening on port 6010, as if it is an X-server for Display nr 10. The SSH server does this (if it is configured to allow this kind of connection) and then starts the shell, usually cleverly setting the DISPLAY value to localhost:10.0. Note that "localhost" refers to the server itself, thus connections to port 6010 will be picked up by the SSH daemon on that server. When such a connection is made, eg by starting an X-client from within the SSH sessions, the SSH daemon on the server accepts the connection and it knows that the specific port is associated with your SSH tunnel, so it will forward the X-windows data back to your workstation via the existing SSH tunnel. The SSH client on your workstation distinguishes the X-windows data from other data along the channel, and makes a connection to the real X-server on the workstation, and forwards the un-encrypted packets received to the local X-server.

So in summary, the X-server runs on the workstation where the display is rendered. The X-client establishes the TCP/IP session to this X-server based on the value of the DIPSLAY setting. In normal situations you will observe the connection as incomming back towards yourself, but usually we associate the "server" of a client-server application with the software running on the remote machine (relatively to where we are sitting), but in this case it is obviously back-to-front.

Note: This brief introduction glosses over the many complex issues and details. Desktop Environments, Display Managers, Composting, Direct Rendering, the Driver model, connections via other than TCP/IP protocols, etc. For a slightly more complete though still very digestible introduction, see the Wikipedia Article on the X-Windowing System. For all the information you can handle head over to the official steering group web site at http://x.org

Making the most of Solaris Man pages

2008-10-18T18:42:00.010+02:00

Solaris man pages (manual pages) are well written, consistent, complete, and generally a great source of information. Here are a few tips to help you get the most out of them. Of course this applies to all Solaris derivatives, including proper Solaris, OpenSolaris and other Open Solaris distributions like Solaris Express, Belenix and Nexenta. It even applies to other Unix flavors like BSD and AIX, and Unix derivatives like Linux.

The simplest (and most obvious) way to use the manual pages are to enter "man command" and then read the entire page.

My first hint however is to change the "PAGER" environment variable. I suggest you set it like this in your .profile file:

PAGER="less -iMsq"; export PAGER

(or for csh and its family members, use setenv PAGER "less -iMsq")

The reasons for this are

a) "less" supports more useful options than does "more". In particular, it supports highlighting, as well as all of the below!

b) The "-i" causes less to ignore case in searches. A definite advantage because you can simply enter /user and it will find the string "user" even when capitalized for use at the start of a sentence.

c) The -M causes "less" to show a more verbose status at the bottom of the screen.

d) -q ... Stop irritating me.

e) -s to "squeeze" blank lines (because otherwise less will "format" pages with blank spaces, as if it is a printed page)

f) You can return to the first line (top) of the page by taping "1" followed by "G"

g) less is more.

The PAGER environment variable is automatically used by the man command, so you can now test it and will find that you can scroll up and down in man pages. To find a word in the man page, tap the forward-slash / followed by the string to search for, and the word wherever it is found will be highlighted in inverse text.

Note: less does have some limitations over more, but these do not apply to viewing man pages. For the curious, the limitation is in the handling of control characters in the input file. In particular, more does a better job of displaying "captured" sessions recorded from a system console or from a shell using the "script" command. Regardless, I solve these by reading capture files using cat -nvet | less -iMq (Note - I like seeing line numbers)

Second Hint: Generate the Manual Pages keyword index database (The so-called windex or apropos information). To do this, you need to run the following command once.

$ catman -w

This will run for a while as it generates and stores the man page keyword index for future use.

Once it completes, the man command's -k option and the "whatis" command will work. This allows you to find what you need much easier. For example to find all man pages for tools, commands and drivers related to Wifi and Wireless networking, you can use

$ man -k wifi wireless

Third hint: Some man pages are not automatically searched or found. In older (proper) Solaris releases, the man command has got a "default list" of locations where to find man pages. In Recent Open Solaris releases, man constructs a list of locations to search. The man manual page states:

  Search Path
     Before searching for a given name, man constructs a list  of
     candidate directories and sections. man searches for name in
     the directories specified by the MANPATH  environment  vari-
     able.

     In the absence of MANPATH, man constructs  its  search  path
     based  upon the PATH environment variable, primarily by sub-
     stituting man for the last component of  the  PATH  element.
     Special  provisions  are added to account for unique charac-
     teristics  of   directories   such   as   /sbin,   /usr/ucb,
     /usr/xpg4/bin, and others. If the file argument contains a /
     character, the dirname portion of the argument  is  used  in
     place of PATH elements to construct the search path.

Note: The (currently apparently undocumented) -p option to man will show the effective man search path.

As implied, you can set in MANPATH a list of directories containing man pages you use regularly. It is also possible to overide the man search path using the man command's -M option. This is particularly true for packages that install under /usr/local, /opt/SUNW...., etc. So to view the man page for "explorer" you would run:

man -M /opt/SUNWexplo/man explorer

(Assuming you have SUNWexplorer installed)

Finally, it helps to understand the structure of a man page. The individual manual pages are divided up into a common set of sections, and knowing them you will often skip straight to a specific (sub)section in the page when looking for specific information.

Some of these subsections are optional, but the names used are consistent. Here is a quick summary of some important sections.

NAME	What this page is about. This is the "whatis" information.
SYNOPSIS	The summary, eg how to use a command.
DESCRIPTION	A more complete description of what this component is.
OPTIONS, OPERANDS and USAGE	These section details how a command is used, eg what command-line options are available for a specific command.
ENVIRONMENT VARIABLES	As the name suggests, this section details Environment variables which affects the behavior of the command.
EXIT STATUS	As the name implies, it explains the possible exit status values. Useful to know that the section exists though.
FILES	A list of files which are relevant to the command or subsystem. In particular here you will find out where a program keeps its configuration files. For a good example, see man dumpadm.
EXAMPLES	Often a great place to skip to when a manual page is complicated.
ATTRIBUTES	An often overlooked section, it explains the status of the item. For commands it tells you which packages the files are in. This section of man pages are fully explained in the "attributes" manual page.
SEE ALSO	One of the most useful parts of man pages, this gives you hints about other pages which are related to the same topic.
NOTES	Various Additional notes.

Other sections, such as SECURITY, ERRORS, SUBCOMMANDS, etc exists in some man pages, but the above are probably the most useful sections.

The entire collection of man pages are divided up into manual sections, such as Section 1 - User Commands, Section 1M - Maintenance commands for sys-admins, Section 3C, the C programming reference information and section 7 - information about device drivers. This is not hugely important, except that you need to understand that some pages occur in more than one section. For example "read" or "signal" By default, man will display the first manual page which matches the file name. If you want to see one of the other pages you need to specify the specific manual section explicitly.

For example

$ man -f signal

signal  signal (3c) - simplified signal management for application processes
signal  signal (3ucb) - simplified software signal facilities
signal  signal.h (3head)    - base signals

We can see that file "signal" has got manual entries in sections 3c (C programming reference), 3ucb (UCB/Posix version), and 3head (the Headers section)

Simply entering ||man signal|| will show the man page from "3C". To see the man page in the "3head" section, enter either

$ man -s 3head signal

$ man signal.3head

I often end up using the second form simply because after viewing a man page I often see that there is another related man page from the SEE ALSO section. I then exit the man page, recall the previous command, and append .section to the command line, which is to say that it is just a matter of convenience.

One last tip: There are online versions of the manual pages at http://docs.sun.com.

You don't need to learn every command and every option by heart. Knowing that you have manual pages, and that you can look up the information you need quickly by looking at related commands from SEE ALSO, and from the keyword index using man -k, and then being able to quickly search through man pages for the right section or keyword absolutely WILL make you a clever, more efficient and over all better sysadmin!

How Solaris disk device names work

2008-07-29T20:28:00.008+02:00

Writing this turned out to be surprisingly difficult as the article kept on growing too long. I tried to be complete but also concise to make this a useful introduction to how Solaris uses disks and on how device naming works.

So to start at the start: Solaris builds a device tree which is persistent across reboots and even across configuration changes. Once a device is found at a specific position on the system bus, and entry is created for the device instance in the device tree, and an instance number is allocated. Predictably, the first instance of a device is zero, (e.g e1000g0) and subsequent instances of device using the same driver gets allocated instance numbers incrementally (e1000g1, e1000g2, etc). The allocated instance number is registered together with the device driver and path to the physical device in the /etc/path_to_inst file.

This specific feature of Solaris is very important in providing stable, predictable behavior across reboots and hardware changes. For disk controllers this is critical as system bootability depends on it!

With Linux, if the first disk in the system is known as /dev/sda, even if it happens to be on the second controller, or have a target number other than zero on that controller. New disk added on the first controller, or on the same controller but with a lower target number, causes the existing disk to move to /dev/sdb, and the new disk then becomes /dev/sda. This used to break systems, causing them to become non-bootable, and was being a general headache. Some methods of dealing with this exists, using unique disk identifiers and device paths based on /dev/disk/by-path, etc.

If a Solaris system is configured initially with all disks attached to the second controlled, the devices will get names starting with c1. Disks added to the first controller later on will have names starting with c0, and the existing disk device names will remain unaffected. If a new controlled is added to the system, it will get a new instance number, e.g c2, and existing disk device names will remain unaffected.

Solaris however composes disk device names (device aliases) of parts which identifies the controller, the target-id, the LUN-id, and finally the slice or partition on the disk.

I will use some examples to explain this. Looking at this device:

$ ls -lL /dev/dsk/c1t*

br--------   1 root     sys       27, 16 Jun  2 16:26 /dev/dsk/c1t0d0p0
br--------   1 root     sys       27, 17 Jun  2 16:26 /dev/dsk/c1t0d0p1
br--------   1 root     sys       27, 18 Jun  2 16:26 /dev/dsk/c1t0d0p2
br--------   1 root     sys       27, 19 Jun  2 16:26 /dev/dsk/c1t0d0p3
br--------   1 root     sys       27, 20 Jun  2 16:26 /dev/dsk/c1t0d0p4
br--------   1 root     sys       27,  0 Jun  2 16:26 /dev/dsk/c1t0d0s0
br--------   1 root     sys       27,  1 Jun  2 16:26 /dev/dsk/c1t0d0s1
br--------   1 root     sys       27, 10 Jun  2 16:26 /dev/dsk/c1t0d0s10
br--------   1 root     sys       27, 11 Jun  2 16:26 /dev/dsk/c1t0d0s11
br--------   1 root     sys       27, 12 Jun  2 16:26 /dev/dsk/c1t0d0s12
br--------   1 root     sys       27, 13 Jun  2 16:26 /dev/dsk/c1t0d0s13
br--------   1 root     sys       27, 14 Jun  2 16:26 /dev/dsk/c1t0d0s14
br--------   1 root     sys       27, 15 Jun  2 16:26 /dev/dsk/c1t0d0s15
br--------   1 root     sys       27,  2 Jun  2 16:26 /dev/dsk/c1t0d0s2
br--------   1 root     sys       27,  3 Jun  2 16:26 /dev/dsk/c1t0d0s3
br--------   1 root     sys       27,  4 Jun  2 16:26 /dev/dsk/c1t0d0s4
br--------   1 root     sys       27,  5 Jun  2 16:26 /dev/dsk/c1t0d0s5
br--------   1 root     sys       27,  6 Jun  2 16:26 /dev/dsk/c1t0d0s6
br--------   1 root     sys       27,  7 Jun  2 16:26 /dev/dsk/c1t0d0s7
br--------   1 root     sys       27,  8 Jun  2 16:26 /dev/dsk/c1t0d0s8
br--------   1 root     sys       27,  9 Jun  2 16:26 /dev/dsk/c1t0d0s9

We notice the following:

1. The entries exist as links under /dev/dsk, pointing to the device node files in the /devices tree. Actually every device has got a second instance under /dev/rdsk. The ones under /dev/dsk are "block" devices, used in a random-access manner, e.g for mounting file systems. The "raw" device links are character devices, used for low-level access functions (such as creating a new file system).

2. The device names all start with c1, indicating controller c1 - so basically all the entries above are on one controller.

3. The next part of the device name is the target-id, indicated by t0. This is determined by the SCSI target-id number set on the device, and not by the order in which disks are discovered. Any new disk added to this controller will have a new unique SCSI target number and so will not affect existing device names.

4. After the target number each disk has got a LUN-id number, in the example d0. This too is determined by the SCSI LUN-id provided by the device. Normal disks on a simple SCSI card all show up as LUN-id 0, but devices like arrays or jbods can present multiple LUNs on a target. (In such devices the target usually indicates the port number on the enclosure)

5. Finally each device identifies a partition or slice on the disk. Devices with names ending with a p# indicates a PC BIOS disk partition (sometimes called an fdisk or primary partition), and names ending with an s# indicates a Solaris slice.

This begs some more explaining. There are five device names ending with p0 through p4. The p0 device, eg c1t0d0p0, indicates the whole disk as seen by the BIOS. The c_t_d_p1 device is the first primary partition, with c_t_d_p2 being the second, etc. These devices represent all four of the allowable primary partitions, and always exists even when the partitions are not used.

In addition there are 16 devices with names ending with s0 though s15. These are Solaris "disk slices", and originate from the way disks are "partitioned" on SPARC systems. Essentially Solaris uses slices much like PCs use partitions - most Solaris disk admin utilities work with disk slices, not with fdisk or BIOS partitions.

The way the "disk" is sliced is stored in the Solaris VTOC, which resides in the first sector of the "disk". In the case of x86 systems, the VTOC exists inside one of the primary partitions, and in fact most disk utilities treats the Solaris partition as the actual disk. Solaris splits up the particular partition into "slices", thus the afore mentioned "disk slices" really refers to slices existing in a partition.

Note that Solaris disk slices are often called disk partitions, so the two can be easily confused - when documentation refers to partitions you need to make sure you understand whether PC BIOS partitions or Solaris Slices are implied. In generally if the documentation applies to SPARC hardware (as well as to x86 hardware), then partitions are Solaris slices (SPARC does not have an equivalent to the PC BIOS partition concept)

Example Disk Layout:

First primary Partition

Second primary Partition

Solaris Type 0xBF or 0x80 Partition
s0	Slice commonly used for root
s1	Slice commonly used for swap
s2	Whole disk (backup or overlap slice)
s3	Custom use slice
s4	Custom use slice
s5	Custom use slice
s6	Custom use slice, commonly /export
s7	Custom use slice
s8	Boot block
s9	Alternates (2 cylinders)
s10	x86 custom use slice
s11	x86 custom use slice
s12	x86 custom use slice
s13	x86 custom use slice
s14	x86 custom use slice
s15	x86 custom use slice

Extended partition
p5	Example: Linux or data partition
p6	Example: Linux or data partition
etc	Example: Linux or data partition

Note that traditionally slice 2 "overlaps" the whole disk, and is commonly referred to as the backup slice, or slightly less commonly, called the overlap slice.

The ability to have slice numbers from 8 to 15 is x86 specific. By default slice 8 covers the area on the disk where the label, vtoc and boot record is stored. Slice 9 covers the area where the "alternates" data is stored - a two-cylinder area used to record information about relocated/errored sectors.

Another example of disk device entries:

$ ls -lL /dev/dsk/c0*

brw-r-----   1 root     sys      102, 16 Jul 14 19:45 /dev/dsk/c0d0p0
brw-r-----   1 root     sys      102, 17 Jul 14 19:45 /dev/dsk/c0d0p1
brw-r-----   1 root     sys      102, 18 Jul 14 19:45 /dev/dsk/c0d0p2
brw-r-----   1 root     sys      102, 19 Jul 14 19:12 /dev/dsk/c0d0p3
brw-r-----   1 root     sys      102, 20 Jul 14 19:45 /dev/dsk/c0d0p4
brw-r-----   1 root     sys      102,  0 Jul 14 19:45 /dev/dsk/c0d0s0
brw-r-----   1 root     sys      102,  1 Jul 14 19:45 /dev/dsk/c0d0s1
...
brw-r-----   1 root     sys      102,  8 Jul 14 19:45 /dev/dsk/c0d0s8
brw-r-----   1 root     sys      102,  9 Jul 14 19:45 /dev/dsk/c0d0s9

The above example is taken form an x86 system. Note the lack of a target number in the device names. This is particular to ATA hard drives on x86 systems. Besides that it works like normal device names I described above.

Below, comparing the block and raw device entries:

$ ls -l /dev/*dsk/c1t0d0p0

lrwxrwxrwx   1 root     root          49 Jun 26 16:22 /dev/dsk/c1t0d0p0 -> ../../devices/pci@0,0/pci-ide@1f,2/ide@1/sd@0,0:q
lrwxrwxrwx   1 root     root          53 Jun  2 16:18 /dev/rdsk/c1t0d0p0 -> ../../devices/pci@0,0/pci-ide@1f,2/ide@1/sd@0,0:q,raw

These look the same, except that the second one points to the raw device node.

For completeness' sake, some utilities used in managing disks:

format

The work-horse, used to perform partitioning (including fdisk partitioning on x86 based systems), analyzing/testing the disk media for defects, tuning advanced SCSI parameters, and generally checking the status and health of disks.

rmformat

Shows information about removable devices, formats media, etc.

prtvtoc

Command-line utility to display information about disk geometry and more importantly, the contents of the VTOC in a human readable format, showing the layout of the Solaris slices on the disk.

fmthard

Write or overwrite a VTOC on a disk. Its input format is compatible with the output produced by prtvtoc, so it is possible to copy the VTOC between two disks by means of a command like this:

prtvtoc /dev/rdsk/c1t0d0s2 | fmthard -s - /dev/rdsk/c1t1d0s2

This is obviously not meaningful if the second disk do not have enough space. If the disks are of different sizes, you can use something like this:

prtvtoc /dev/rdsk/c1t0d0s2 | awk '$1 != 2' | fmthard -s - /dev/rdsk/c1t1d0s2

The above awk command will cause the entry for slice 2 to be committed, and fmthard will then maintain the existing entry or, if none exists, create a default one on the target disk.

Also note, as implied above, Solaris slices can (and often do) overlap. Care needs to be taken to not have file systems on slices which overlap other slices.

iostat -En

Show "Error" information about disks, and often very usefull, the firmware revisions and manufacturer's identifier strings.

format -e

This reveals [enhanced|advanced] functionality, such as the cache option on SCSI disks.

format -Mm

Enable debugging output, particularly makes SCSI probe failures non-silent.

cfgadm and luxadm also deserves honorable mention here. These commands manage disk enclosures, detaching and attaching, etc. but are also used in managing some aspects of disks.

luxadm -e port

Show list of FC HBAs.

luxadm can also for example be used to set the beacon LED on individual disks in FCAL enclosures that support this function. The details are somewhat specific to the relevant enclosure.

cfgadm can be used to probe SAN connected subsystems, eg by doing:

cfgadm -c configure c2::XXXXXXXXXXXX

(where XXXXXXXXXXX is the enclosure port WWN, using controller c2)

Hopefully this gives you an idea about how disk device names, controller names, and partitions and slices all relate to one another.

Reading and Writing ISO images using Solaris

2008-07-16T20:31:00.004+02:00

After my recent post on mounting ISO image files I thought I should write a quick article on the other ways of using these files: Reading a disk in to a file and burning a file to a disk. This is not a complete guide on the topic by a long shot, but if you just want the quick start answer, it is here.

If you have an iso9660 CD (or DVD) image file that you want to burn to a disk, you simply use this command:

# cdrw -i filename.iso

This will write the file named filename.iso to the default cd writer device. If working with DVD media, the session is closed (using Disk at once writing), while for CD media Track-at-once writing is used.

To create an ISO image from a disk, use this command:

# readcd dev=/dev/rdsk/c1t0d0s2 f=filename.iso speed=1 retries=20

readcd needs at least the device and the file to be specified. To discover the device, you can use the command "iostat -En" and look for the Writer device, or you can let readcd scan for a device, using a command like this:

# readcd -scanbus

scsibus1:
 1,0,0 100) 'MATSHITA' 'DVD-RAM UJ-841S ' '1.40' Removable CD-ROM
 1,1,0 101) *
 1,2,0 102) *
 1,3,0 103) *
 1,4,0 104) *
 1,5,0 105) *
 1,6,0 106) *
 1,7,0 107) *

The device 1,0,0 can be used directly, or you can convert it to the Solaris naing convention as I did in the example above.

There are of course other ways of doing it, feel free to comment and tell me about your fevourite method for reading to or burning from ISO-image files.

A short guide to the Solaris Loop-back file systems and mounting ISO images

2008-07-09T12:47:00.002+02:00

The Solaris Loop-back file system is a handy bit of software, allowing you to "mount" directories, files and, in particular, CD or DVD image files in ISO-9660 format.

To make it more user friendly, build 91 of ONV introduces the ability to the mount command to automatically create the loop-back devices for ISO images! The Changelog for NV 91 has got the following note:

Issues Resolved: PSARC case 2008/290 : lofi mount BUG/RFE:6384817Need persistent lofi based mounts and direct mount(1m) support for lofi

In older releases, it was necessary to run two commands to mount an ISO image file. The first to set up a virtual device for the ISO image:

# lofiadm -a /shared/Downloads/image.iso
/dev/lofi/1

And then to mount it somewhere:

# mount -F hsfs -o ro /dev/lofi/1 /mnt

Solaris uses hsfs to indicate the "High Sierra File System" driver used to mount ISO-9660 files. Specify "-o ro" to make it Read-only, though that is the default for hsfs file systems, at least lately (I seem to recall that at one point in the past it was mandatory to specify read-only mounting explicitly.

Looking at what has been happening here, we can see the Loop-back device by running lofiadm without any options:

# lofiadm

Block Device             File                           Options
/dev/lofi/1              /shared/Downloads/image.iso -

And the mounted file system:

# df -k /mnt

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/lofi/1          2915052 2915052       0   100%    /mnt

The new feature of the mount command requires a full path to the ISO file (Just like lofiadm does, at any rate it does for now)

# mount -F hsfs -o ro /shared/Downloads/image2.iso /mnt

To check the status:

# df -k /mnt

Filesystem            kbytes    used   avail capacity  Mounted on
/shared/Downloads/image2.iso
                     7781882 7781882       0   100%    /mnt

And when we run lofiadm we see it automatically created a new device, /dev/lofi/2:

# lofiadm

Block Device             File                           Options
/dev/lofi/1              /shared/Downloads/image.iso -
/dev/lofi/2              /shared/Downloads/image2.iso -

Some of the other uses of the Loop-back file system:

You can mount any directory on any other directory:

# mkdir /mnt1
# mount -F lofs -o ro /usr/spool/print /mnt2

Note the use of lofs as the file system "type". This is a bit like a hard-link to a directory, and it can exist across file systems. These can be read-write or read-only.

You can also mount any individual file onto another file:

# mkdir /tmp/mnt
# echo foobar > /tmp/mnt/X
# mount -F lofs /usr/bin/ls /tmp/mnt/X
# ls -l /tmp/mnt

total 67
-r-xr-xr-x   1 root     bin        33396 Jun 16 05:43 X

# cd /tmp/mnt
# ./X
X
# ./X -l

total 67
-r-xr-xr-x   1 root     bin        33396 Jun 16 05:43 X

The above feature incidentally inspired item nr 10 on my ZFS feature wish list.

This allows for a lot of flexibility. In deed this functionality is central to how file systems and disk space is provisioned in Solaris Zones. If you play around with it you will find plenty of uses for it!

Some days I'm glad that I'm not a network administrator

2008-07-07T10:33:00.003+02:00

ZFS missing features

2008-07-06T21:36:00.002+02:00

What would truly make ZFS be The Last Word in File Systems (PDF)?

Why every feature of course! Here is my wishlist!

Nested vdevs (eg Raid 1+Z)
Hirarchical Storage management (migrate rarely used files to cheaper/slower vdevs)
Traditional Unix Quotas (i.e for when you have multiple users owning files in a the same directories spread out across a file system)
A way to convert a directory on a ZFS file system into a new ZFS file system, and the corresponding reverse function to merge a directory back into its parent (because the admin made some wrong decision)
Backup function supporting partial restores. In fact partial backups should be possible too, eg backing up any directory or file list, not necesarily only at the file system level. And restores which does not require the file system to be unmounted / re-mounted.
Re-layout of pools (to accomodate adding disks to a raidz or converting a non-redundant pool to raidz or removing disks from a pool, etc) (Yes I'm aware of some work in this regard)
Built-in Multi-pathing capabilities (with automatic/intelligent detection of active paths to devices), eg integrated MPxIO functionality. I'm guessing this is not there yet because people may want to use MPxIO for other devices not under ZFS control and this will create situations where there are redundant layers of multipathing logic.
True Global File System functionality (multiple hosts accessing the same LUNs and mounting the same file systems with parallel write. Or even just a sharezfs (like sharenfs, but allowing the client to access ZFS features, eg to set ZFS properties, create datasets, snapshots, etc, similar in functionality to what is possible with granting a zone ownership of a zfs dataset.)
While we're at it: In place conversion from, eg UFS to ZFS.
The ability to snapshot a single file in a ZFS file system (So that you can affect per-file version tracking)
An option on the zpool create command to take a list of disks and automatically set up a layout, intelligently taking into considderation the number of disks and the number of controllers, allowing the user to select from a set of profiles determining optimization for performance, space or redundancy.

So... what would it take to see ZFS as the new default file system on, for example USB thumb drives, memory cards for digital cameras and cell phones, etc? In fact, can't we use ZFS for RAM management too (snapshot system memory)?

The Pupil will surpass the tutor

2008-07-05T20:57:00.000+02:00

Linux is an attempt at making a free clone of Unix. Initially it aimed to be Unix compatible, though I feel that goal has become less and less important as Linux grew in maturity.

Now all of a sudden we have a complete turn-about as the big Unices want to be like Linux! - Linux is attractive for a variety of reasons, including a fast, well refined kernel, lots of readily available and free applications, good support and, because of these, a growing and loyal following. The utilities available with most Linux distributions are based on the core utilities found in the big Unices and have a large collection of new additions, all working together in a more or less coherent way to build a usable platform.

Nowadays many new Unix administrators have at least some Linux experience and with this background can be easily frustrated when looking for Linux specific utilities (Where is top in Solaris). End users would like to see the applications that used on Linux to run on Unix. And the ability to run Linux on cheap and cheerful PC hardware does not detract from Linux's popularity by any means.

So Sun Microsystems, just like IBM with AIX, finds itself looking at Linux to see what this platform is doing right to make it sucesfull. To me this, more than anything else, is proof that Linux finally grew up.

I expect that a leap-frog game will emmerge between Linux and Unix, particularly Solaris, where the two are competing with innovative features to be the platform of choice for both datacenter and desktop applications.

Congratulations to the Linux community on a job well done.

Let ZFS manage even more space more eficiently

2008-07-01T16:47:00.003+02:00

The idea of using ZFS to manage process core dumps begs to be expanded to at least crash dumps. This also enters into the realm of Live Upgrade as it eliminates the need to sync potentially a lot of data on activation of a new BE!

Previously I created a ZFS file system in the root pool, and mounted it on /var/cores.

The same purpose would be even better served with a generic ZFS file system which can be mounted on any currently active Live-Upgrade boot environment. The discussion here suggests the use of a ZFS file system rpool/var_shared, mounted under /var/shared. Directories such as /var/crash and /var/cores can then be moved into this shared file system.

So:

/ $ pfexec ksh -o vi
/ $ zfs create rpool/var_shared
/ $ zfs set mountpoint=/var/shared rpool/var_shared
/ $ mkdir -m 1777 /var/shared/cores
/ $ mkdir /var/shared/crash
/ $ mv /var/crash/`hostname` /var/shared/crash

View my handiwork:

/ $ ls -l /var/shared

total 6
drwxrwxrwt   2 root     root           2 Jun 27 17:11 cores
drwx------   3 root     root           3 Jun 27 17:11 crash

/ $ zfs list -r rpool

NAME                       USED  AVAIL  REFER  MOUNTPOINT
rpool                     13.3G  6.89G    44K  /rpool
rpool/ROOT                10.3G  6.89G    18K  legacy
rpool/ROOT/snv_91         5.95G  6.89G  5.94G  /.alt.tmp.b-b0.mnt/
rpool/ROOT/snv_91@snv_92  5.36M      -  5.94G  -
rpool/ROOT/snv_92         4.33G  6.89G  5.95G  /
rpool/dump                1.50G  6.89G  1.50G  -
rpool/export              6.83M  6.89G    19K  /export
rpool/export/home         6.81M  6.89G  6.81M  /export/home
rpool/swap                1.50G  8.38G  10.3M  -
rpool/export/cores          20K  2.00G    20K  /var/cores
rpool/var_shared            22K  3.00G    22K  /var/shared

Just to review the current settings for saving crash dumps:

/ $ dumpadm

      Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/crash/solwarg
  Savecore enabled: yes

Set it to use the new path I made above:

/ $ dumpadm -s /var/shared/crash/`hostname`

      Dump content: kernel pages
       Dump device: /dev/zvol/dsk/rpool/dump (dedicated)
Savecore directory: /var/shared/crash/solwarg
  Savecore enabled: yes

Similarly I update the process core dump settings:

/ $ coreadm -g /var/shared/cores/core.%z.%f.%u.%t
/ $ coreadm

     global core file pattern: /var/shared/cores/core.%z.%f.%u.%t
     global core file content: default
       init core file pattern: core
       init core file content: default
            global core dumps: disabled
       per-process core dumps: enabled
      global setid core dumps: enabled
 per-process setid core dumps: disabled
     global core dump logging: enabled

And finally, some cleaning up:

/ $ zfs destroy rpool/export/cores
/ $ cd /var
/var $ rmdir crash
/var $ ln -s shared/crash
/var $ rmdir cores

As previously, the above soft link is just in case somewhere there is a naughty script or tool with a hard coded path to /var/crash/`hostname`. I don't expect to find something like that in oficially released Sun software, but I do some times use programs not officially released or supported by Sun.

This makes me wonder what else can I make it do! I'm looking forward to my next Live Upgrade to see how well it preserves my configuration before I attempt to move any of the spool directories from /var to /var/shared!

Use Live upgrade already

2008-06-30T08:25:00.001+02:00

If you are still not using live upgrade, you need to make learning it a priority. It will save you hours and make your manager happy because it costs nothing and gives you a simple, good and fast method for regressing your changes. You just need a few (about 10) GB of free disk space, be it in your root ZFS pool, on an un-used disk, or even any slice on any disk in the system.

The Live Upgrade concept is simple: make a copy of your "boot environment", run the upgrade or patching against this copy (called the alternative boot environment), and finally activate it.

Creation of the new boot environment is done by running a few simple commands which copies and updates the files in the new boot environment, an operation that can (and does) take a considerable amount of time, but runs in the back ground while the system is up and running, with all services online and active.

The Live Upgrade commands comes from three packages that you should install from the target OS's install media - for example if you want to upgrade from Solaris 9 to Solaris 10, you install SUNWlucfg, SUNWluu and SUNWlur from the Solaris 10 media (or run the liveupgrade20 install script in Tools/Installers directory)

Then once this is completed, another command (luactivate) is run to confirm that the new boot environment must be activated on the next reboot. On SPARC systems, this process modifies the boot-device in the OBP, while on i386 systems it updates Grub with a new "default".

Then all that is left is to do the actual reboot. During the reboot some special files and directories will be synchronized one last time - this is because between the time the system was copied over to the clone, and the time when the reboot runs, various things could possibly change: People still log in and change their passwords, recieve and send mail, spool jobs to the printers, etc. The administrator could even create new login accounts! To deal with this, Live Upgrade will synchronize a pre-determined list of files and directories during the first boot of the new boot environment.

The list of files copied is available here, and can be customized by editing the /etc/lu/synclist file.

The Live Upgrade system has got the intelligence built in to allow the new boot environment to find the files in the old boot environment during the boot up process, so this is completely automatic.

Recent Solaris Express installations prepare for the use of live upgrade by automatically setting up a slice and mounting is as "/second_root", but you need to unmount it and remove it from /etc/vfstab before live upgrade will allow you to use it. If you don't have a free slice, make one (backup /export, unmount it, and create two smaller slices in its place, one for live upgrade and one to restore /export to). This will be cheaper than performing upgrades the traditional way.

Using a dedicated ZFS file system to manage process core dumps

2008-06-19T15:33:00.004+02:00

ZFS just bristles with potential. Quotas, Reservations, turning compression or atime updates on or off without unmounting. The list goes on.

So now that we have ZFS root (Since Nevada build SNV_90, and even earlier when using OpenSolaris or other distributions) lets start to make use of these features.

First thing is, on my computer I don't care about access time updates on files or directories, so I disable it.

/ $ pfexec zfs set atime=off rpool

That is not particularly spectacular in itself, but since it is there I use it. The idea is of course to save a few disk updates and the corresponding IOs.

Next: core dumps. One of my pet hates. Many processes dumps core in your home dir, these get overwritten or forgotten, and then there are any number of core files lying around all over the file systems, all off these just wasting space since I don't really intent do try to analyze any of them.

Solaris has got a great feature by which core dumps can be all directed to go to a single directory and, on top of that, to have more meaningful file names.

So the idea is to create a directory, say /var/cores and then store the core files in there for later review. But knowing myself these files will just continue to waste space until I one day decide to actually try and troubleshoot a specific issue.

To me this sounds like a perfect job for ZFS.

First I check that there is not already something called /var/cores:

/ $ ls /var/cores
/var/cores: No such file or directory

Great. Now I create it.

/ $ pfexec zfs create rpool/export/cores
/ $ pfexec zfs set mountpoint=/var/cores rpool/export/cores

And set a limit on how much space it can ever consume:

/ $ pfexec zfs set quota=2g rpool/export/cores

Note: This can easily be changed at any time, simply by setting a new quota.

Which creates the below picture.

/ $ df -h

Filesystem             size   used  avail capacity  Mounted on

rpool/ROOT/snv_91       20G   5.9G   7.0G    46%    /

/devices                 0K     0K     0K     0%    /devices

/dev                     0K     0K     0K     0%    /dev

ctfs                     0K     0K     0K     0%    /system/contract

proc                     0K     0K     0K     0%    /proc

mnttab                   0K     0K     0K     0%    /etc/mnttab

swap                   2.3G   416K   2.3G     1%    /etc/svc/volatile

objfs                    0K     0K     0K     0%    /system/object

sharefs                  0K     0K     0K     0%    /etc/dfs/sharetab
/usr/lib/libc/libc_hwcap1.so.1

                        13G   5.9G   7.0G    46%    /lib/libc.so.1

fd                       0K     0K     0K     0%    /dev/fd

swap                   2.3G   7.2M   2.3G     1%    /tmp

swap                   2.3G    64K   2.3G     1%    /var/run

rpool/export            20G    19K   7.0G     1%    /export

rpool/export/home       20G   6.8M   7.0G     1%    /export/home

rpool                   20G    44K   7.0G     1%    /rpool

rpool/export/cores     2.0G    18K   2.0G     1%    /var/cores

SHARED                  61G    24K    31G     1%    /shared

 ... snip ...

And checking the settings on the /var/cores ZFS file system

/ $ zfs get all rpool/export/cores

NAME                PROPERTY         VALUE                  SOURCE

rpool/export/cores  type             filesystem             -

rpool/export/cores  creation         Thu Jun 19 14:18 2008  -

rpool/export/cores  used             18K                    -

rpool/export/cores  available        2.00G                  -

rpool/export/cores  referenced       18K                    -

rpool/export/cores  compressratio    1.00x                  -

rpool/export/cores  mounted          yes                    -

rpool/export/cores  quota            2G                     local

rpool/export/cores  reservation      none                   default

rpool/export/cores  recordsize       128K                   default

rpool/export/cores  mountpoint       /var/cores             local

rpool/export/cores  sharenfs         off                    default

rpool/export/cores  checksum         on                     default

rpool/export/cores  compression      off                    default

rpool/export/cores  atime            off                    inherited from rpool

rpool/export/cores  devices          on                     default

rpool/export/cores  exec             on                     default

rpool/export/cores  setuid           on                     default

rpool/export/cores  readonly         off                    default

rpool/export/cores  zoned            off                    default

rpool/export/cores  snapdir          hidden                 default

rpool/export/cores  aclmode          groupmask              default

rpool/export/cores  aclinherit       restricted             default

rpool/export/cores  canmount         on                     default

rpool/export/cores  shareiscsi       off                    default

rpool/export/cores  xattr            on                     default

rpool/export/cores  copies           1                      default

rpool/export/cores  version          3                      -

rpool/export/cores  utf8only         off                    -

rpool/export/cores  normalization    none                   -

rpool/export/cores  casesensitivity  sensitive              -

rpool/export/cores  vscan            off                    default

rpool/export/cores  nbmand           off                    default

rpool/export/cores  sharesmb         off                    default

rpool/export/cores  refquota         none                   default

rpool/export/cores  refreservation   none                   default

Note that Access-time updates on this file system is off - the setting has been inherited from the pool. The only "local" settings are the mountpoint and the quota which corresponds to the items that I've specified manually.

Now just to make new core files actually use this directory. At present, the default settings from coreadm looks like this:

/ $ coreadm

     global core file pattern: 

     global core file content: default

       init core file pattern: core

       init core file content: default

            global core dumps: disabled

       per-process core dumps: enabled

      global setid core dumps: disabled

 per-process setid core dumps: disabled

     global core dump logging: disabled

Looking at the coreadm man page, there is a fair amount of flexibility in what can be done. I want core files to have a name identifying the zone in which the process was running, the process executable file, and the user. I also don't want core dumps to overwrite when the same process keeps on faulting, so I will add a time stamp to the core file name.

/ $ pfexec coreadm -g /var/core/core.%z.%f.%u.%t

And then I would like to enable logging of an event any time when a core file is generated, and also to store core files for Set-UID processes:

/ $ pfexec coreadm -e global-setid -e log

And finally, just to review the core-dump settings, these now look like this:

/ $ coreadm

     global core file pattern: /var/core/core.%z.%f.%u.%t

     global core file content: default

       init core file pattern: core

       init core file content: default

            global core dumps: disabled

       per-process core dumps: enabled

      global setid core dumps: enabled

 per-process setid core dumps: disabled

     global core dump logging: enabled

Now if that is not useful, I don't know what is! You will soon start to appreciate just how much space is wasted and just how truly rigid and inflexible other file systems are once you run your machine with a ZFS root!

Update: More on how to make x86 Solaris with Grub boot verbosely

2008-06-14T11:54:00.002+02:00

Since I posted a while ago on how to make Solaris boot verbosely, I found a better way. Or rather, I learned a bit more about this.

In stead of just adding -v to the kernel line, add "-v -m verbose"

The "-m verbose" portion passes the verbose option to SMF, giving you verbose information about startup of services.

The "-v" causes the messages which normally goes to the system log to also be emitted on the console.

My grub entry for verbose booting now looks like this:

# Solaris SNV91 Verbose Boot
title Solaris SNV_91 Verbose Boot
findroot (BE_SNV_91,1,a)
kernel$ /platform/i86pc/kernel/$ISADIR/unix -v -m verbose -B $ZFS-BOOTFS
module$ /platform/i86pc/$ISADIR/boot_archive
# End Solaris SNV_91 Verbose

Of course you don't need the entry to be in grub - if your system is not booting, use the edit feature to add these options to the kernel line in the grub item you want to boot from.

Also just a note on the spash image. Removing it is entirely optional - not removing it will not hide any bootup messages (as I previously through)

Not a day goes by that I don't learn something new about Solaris.

Sharing a ZFS pool between Linux and Solaris

2008-06-04T11:46:00.009+02:00

If you are multi-booting between Linux and Solaris (and others like FreeBSD, OpenBSD and Mac OS X, I expect) you will sooner or later encounter the problem of how to share disk space between the operating systems. FAT32 is not satisfactory due to its lack of POSIX features, in particular file ownership and access modes, not to mention its sub par performance. ext2/3 is not an option because you only get read-only support for it in Solaris, and similarly UFS enjoys only read-only support in Linux. The whole situation is rather depressing.

Enter ZFS.

This all started because I discovered that I can have a ZFS root file system without having to install OpenSolaris. The trick as some of you may know, is to select "Solaris Express" from the first menu on booting the install disk, and then select one of the two "Interactive Text" options from the next menu. This puts you back into 1984 in terms of installers, but you get the option of using ZFS for root!

Note: It might be possible to do this with the default installer, but on my computer the installer just would not run (I got some daft error about fonts and mouse themes). With a ZFS root, the Swap and Dump automatically goes onto dedicated vdevs, and you save a lot in terms of pre-allocated space.

I have of course used ZFS on my laptop previously as a test, but the benefits were limited by the fact that I still had "slices" for the OS and a small ZFS pool on a spare slice.

I'm not sure which build of Nevada first introduced the ZFS root option in the installer, but it is available in build 90 at least.

My choice of Linux distribution is Ubuntu 8.04. The steps to setting up a ZFS pool shared across operating systems are as follow:

1. Select a Partitioning scheme with minimal space allocated to each of Ubuntu and Nevada.
I decided to put Ubuntu in an Extended partition with a 10 GB Logical Partition for the OS, /var and /home, and a 1 GB Logical partition for Swap.
For Solaris I allocated a 24 GB primary partition to become the ZFS root pool, which includes Swap, Dump, OS and Live-upgrade space.
The balance of the 100 GB disk will be shared between Ubuntu and Solaris using ZFS.

Note: Linux and Solaris has got some different views on how disk partitioning works.
Due to historical reasons, in particular due to compatibility with Solaris on SPARC hardware, Solaris slices live in a single primary partition with an identifier of 0x82 (SOLARIS) or 0xbf (SOLARIS2) somewhat like how logical fdisk partitions live inside an "extended partition".

2. Install Ubuntu first, creating only the partitions for it. Remember to not have any external drives connected as it can screw up the order in which drives are detected and as a result bugger up the Grub menu list.

During the installation you create an Admin user. This will eventually in the future become a "backup" admin user.

3. Reboot and load patches/updates, and backup the Grub /boot/grub/menu.lst file to an external media such as a USB thumb drive for easy access. The Ubuntu Grub does not understand ZFS, so you need to use Nevada's Grub to manage the multi-booting.

4. Also set Ubuntu to use the hardware clock as local time in stead of UTC. (This is what Solaris uses) To do this change UTC=yes to UTC=no in /etc/default/rcS, then reboot.

5. Install Nevada. Use either of the Interactive Text installer options, but for simplicity's sake specify the system as non-networked.

6. Reboot and create a user for every-day use, and add this user to the "Primary Administrator" role using usermod -P "Primary Administrator" <userid>

7. Add the Ubuntu Grub entries you saved in step 3 to the end of the Nevada grub menu.lst file. This will be stored in /boot/grub/menu.lst (The default pool name is rpool)

8. Reboot back into Ubuntu, then follow the Linux ZFS-FUSE instalation instructions to get ZFS-FUSE installed. I used the trunk to get the latest ZFS updates from Opensolaris.org included. Also see this Ubuntu Wiki page, and Ralf Hildebrand's blog for more info.

For reference, this is the procedure I used

apt-get install mercurial build-essential scons libfuse-dev libaio-dev devscripts build-essential zlib1g-dev
cd ~
hg clone http://www.wizy.org/mercurial/zfs-fuse/trunk
cd trunk/src
scons
sudo scons install

9. Create an fdisk partition for the shared ZFS pool using the remaining disk space. I used a primary partition and set the identifier to W95 FAT32, though this is probably unimportant.

10. While still running running Ubuntu, create a ZFS pool on this new fdisk partition using a command like this:

sudo /usr/local/sbin/zfs-fuse
sudo /usr/local/sbin/zpool create -m /shared SHARED

I like to give my ZFS pools names in all-capitals, purely because it makes the ZFS pool devices stand out better in the output from df and mount.

WARNING: I found that if I created the ZFS pool under Solaris, it refused to import into Ubuntu, but if I created it under Linux it imports/exports just fine in both directions. Both pools are created as version 10 pools, so the reason for this is not obvious. If you do decide to experiment with creating the pool under Solaris, when you want to realy get rid of the pool you will discover you need to dd zeros over the pool before creating it again, otherwise the condition remains unchanged despite destroying and re-creating the pool. If you do experiment with this please do share your results!

11. Export the ZFS pool using

/usr/local/sbin/zpool export SHARED

12. Reboot into Nevada and import the pool using

/usr/local/sbin/zpool import SHARED

Note: If you forget to export before you shut down, you will need to add -f to force the import after booting into the other OS.

At this point I just sat there and stared in wonder at how well it actually works. There is beauty in finally seeing this working!

13. Create some init.d / rc scripts to automate the import/export on shutdown/startup.

14. Now you can start customizing both operating environments. You may want to setup Automatic network configuration by enabling the SMF for NWAM in Solaris, eg by doing:

pfexec svcadm disable physical:default
pfexec svcadm enable physical:nwam

I'm looking forward to testing Live Upgrade on my setup with ZFS root, and to getting a shared home directory to work well for both Solaris and Ubuntu. I have created a login ID with the same gid/uid and a home directory under the shared ZFS pool, but after a few changes it got broken under Ubuntu, probably due to subtle differences in how Gnome/Desktop config items are stored and/or expected.

Despite my initial sceptism about FUSE, it is actually quite functional. All-in-all I love being able to share a file system, well, many files systems, between the two operating environments!

How to get Solaris with Grub on x86 to boot verbosely

2008-05-28T00:01:00.000+02:00

The fact is, great as OpenSolaris is, things do sometimes go wrong, and often booting with verbose output will help you identify why, or at least where in the boot process, things are breaking. How to enable verbose booting is a really simple question, yet one to which I found the answer to be quite elusive for months. I still don't have an official answer, but this is how I managed to do it:

First, find your grub boot menu list file. If you are using ZFS boot, it is in /rpool/boot/grub and if you are using UFS it will be in /boot/grub

Make a backup of the file, then open it up in your favorite editor, and comment out the splash image line near the start.

Now copy the section for your Solaris image. On my system it looks like this:

title LiveUpgradeSNV90_2008-05-27

findroot (BE_LiveUpgradeSNV90_2008-05-27,0,e)

kernel$ /platform/i86pc/kernel/$ISADIR/unix

module$ /platform/i86pc/$ISADIR/boot_archive

You need to change two things: 1) Give it a new title, and 2) Add -v to the kernel line.

Note: -v is in fact a boot-arg, supplied to the unix program. For detailed information have a look at the man page for boot(1m)

After saving your changes, use bootadm to check for mistakes, eg

# bootadm list-menu

The location for the active GRUB menu is: /boot/grub/menu.lst

default 3

timeout 10

0 LiveUpgradeSNV89_2008-05-21

1 LiveUpgradeSNV89_2008-05-21 Solaris xVM

2 LiveUpgradeSNV89_2008-05-21 failsafe

3 LiveUpgradeSNV90_2008-05-27

4 LiveUpgradeSNV90_2008-05-27 Solaris xVM

5 LiveUpgradeSNV90_2008-05-27 failsafe

6 Verbose_Boot_LiveUpgradeSNV90_2008-05-27

Entry nr 6 is the new Entry I added by making a copy of entry nr 3 in the list.

Now all that remains is to test it by rebooting the box. Q.E.D.

How to start a Trivial-FTP server on Solaris 10

2008-03-10T14:47:00.001+02:00

Managing network services under Solaris 10 is a bit different to what it used to be with older Solaris releases. As an example I will show how to set up a Trivial FTP server.

The tftp service software ships with Solaris, but is not configured by default. In my work I need a tftp server quite often, for example it is used to update flash firmware on the system controllers on some of Sun's server systems.

With Solaris 10 you need to define a “SMF service manifest” ... something which can be as complex or as simple as you want it to be. You will find a commented entry for a tftp service in the /etc/inetd.conf file, but simply un-commenting and sig-HUPing inetd it is not enough to start the service. You need to manage the service through the SMF tools.

Solaris 10 ships with a very handy tool to convert inetd.conf entries to SMF service manifests, in the form of the inetconv command.

Make a copy of the /etc/inetd.conf file, eg in /tmp and edit it to un-comment the tftp service entry. You can do some basic customizing, eg to the startup options for the service program, eg if you want to use a non-default directory for the service, change the definition on the line to reflect this. If there are any other service entries in this file you should remove them before continuing, leaving only the entry for the service that you are busy configuring. In the below example I set /export to be the tftp base directory:

tftp dgram udp6 wait root /usr/sbin/in.tftpd in.tftpd -s /export

Then run inetconv, specifying the modified file you created, like this

Bash-3.2# inetconv -i /tmp/myservice_inetd.conf

The new service will be created as follow:

tftp -> /var/svc/manifest/network/tftp-udp6.xml

Importing tftp-udp6.xml ...Done

You can now check the service and start or stop it with the normal svc commands, like this:

bash-3.2$ svcs tftp/udp6

STATE STIME FMRI

online 10:30:05 svc:/network/tftp/udp6:default

Automating the system identification for a Solaris zone to speed up zone deployment

2008-03-09T21:17:00.002+02:00

Recapping, I demonstrated how to create a basic Solaris zone from scratch. Then I showed how to use ZFS snapshots to add the ability to “reset” a zone to a clean state, and how to speed up the definition step by exporting a zone configuration file and then using this as a template for defining zones.

This can save a considerable amount of time with complex zones. The other two steps of creating a zone, namely installing it (populating it with files) and setting it up by completing the system identification during the first boot can also be improved one, the first by using the zoneadm “clone” feature, and the second by using a pre-defined sysidcfg file (and maybe a few other tweaks) injected into the zone file system before it is booted the first time.

This blog entry talks about the second of these.

The sysidcfg file is simply a text file with lines specifying the values for the various options. This file is placed in the zone's /etc directory in its root file system, before it is booted. Then during boot-up, the file is read and any specified values prompted, while any missing items will be prompted for as per normal.

The items that can be set are as follow:

Item	Variable Name	Description of Values
Security Policy	security_policy	Kerberos or NONE. If set to “kerberos”, additional properties can be set. If not specified, a value will be prompted.
Name Service	name_service	NIS, NIS+, LDAP, DNS, NONE. Some additional properties are available when using NIS, NIS+, LDAP or DNS. If not specified, you will be prompted for the appropriate value(s).
NFSv4 Domain Name	nfs4_domain	Specify either the keyword “dynamic”, or provide the value to be used for the NFS4 domain name as a Fully Qualified domain name. If not specified, you will be prompted for the appropriate value(s).
Region and Time zone	timezone	Ether give the time zone from /usr/share/lib/zoneinfo/* or else specify a GMT-offset value. If not specified, you will be prompted for this information.
Terminal Type	terminal	Specify the TERM type, eg vt100. If not specified, you will be prompted for this value.
Locale	system_locale	Specify a locale, eg C, such as found from /usr/lib/locale. If not specified, you will be prompted for this value.
Root Password	root_password	The Encrypted root password. To get this, the easiest is to make a dummy user, set its password to what you want, and then copy the encrypted value from the /etc/shadow file. Other options include writing a little perl script or C program to produce the encrypted version of a password. If not specified, you will be prompted during the first boot.
Network Settings	network_interface	Except for the hostname, these are normally obtained from the zone definition. It can be specified here to override those values, but will not be prompted if not specified.

Note: It is entirely possible to use sys-unconfig in a zone. Doing so will have a similar effect to running sys-unconfig on a global zone or normal Solaris system: The zone will halt and on the next boot you will be presented with prompts for the system identification items. Be Aware that sys-unconfig also removes the zone's root key, and a new one will be generated on the first boot after the system was un-configured.

Something else to note is that a zone's “hostname” and “zone name” does not have to be the same. If you do keep it the same, there will be less opportunity for confusion. While the other network settings for a zone is obtained automatically from the zone's definition, the hostname will still be prompted. To eliminate this prompt, include a network settings section in the zone's sysidcfg file.

Some items available in the sysidcfg file for a normal system can not be set during a zone's system identification as it relies on configuring the kernel and a zone does not have its own dedicated kernel. These include items like power management and the Date and Time, including a Time-server.

An example of a basic sysidcfg file might look like this:

bash-3.2# cat sysidcfg

nfs4_domain=dynamic

security_policy=NONE

timezone=Africa/Johannesburg

terminal=vt100

system_locale=C

name_service=NONE

network_interface=PRIMARY {hostname=ziggy.mydomain}

In the above example the keyword PRIMARY is used to automatically select the only interface configured on this zone. This effectively allows for setting the zone name in the sysidcfg template with minimal fuss. It is of course also possible to use the interface name.

If any of the options are omitted from the file, those items will be prompted for in the usual manner. I did not specify the root login password, so that will be the only item which will be prompted for during the boot up process.

To test this, do the following:

Define the zone (using zonecfg)
Install the zone (using zoneadm -z zonename install
Copy the sysidcfg file to the zone's etc file, eg
cp sysidcfg.template /export/zones/zonename/root/etc/sysidcfg
Boot the zone and connect to its console, eg
zoneadm -z zonename boot; zlogin -C zonename

And voila! Now you can automate the zone definition and the zone's system identification. In the next part I'll show how to speed up the Installation step.

First steps to cloning a Solaris Zone

2008-02-23T21:40:00.003+02:00

Today I want to just mention a few concepts that I've been deliberately neglecting - some of the ideas I mentioned in my 15-minute-to-your-first-zone guide and in my post about Automating Zone creation, and that I will still be posting about in the next few posts are based on this.

Firstly Sun has cleverly integrated Zone management with the ZFS file system.

1. If the parent directory of a Zone's root is on a ZFS file system, then zoneadm will create a new ZFS file system for the rootpath of the zone.

2. Stopping and starting the zone will mount and unmount the zone's root file system

3. Cloning a zone by means of the zoneadm utility will automatically use a ZFS snapshot to create a ZFS clone which will be mounted on the rootpath of the new zone.

This is even more interesting because prior to release 11/06 of Solaris 10, running a zone with its rootpath on a ZFS file system was an unsupported configuration.

The second thing is that resources which are only needed while a zone is running, is created and destroyed dynamically when the zone is started or halted. In particular this applies to network interfaces and loopback file systems. When you start up a zone, you will notice new entries were created for its interfaces, and new file systems were mounted. The file systems which are so managed are normally hidden from df in the global zone, but shows up when you run df with the new -Z switch.

The next concept is that of how zlogin works. You can think of zlogin as a kind of "su" command, but in stead of running a commands under a different userid, it runs a command in a different zone. The default command which it runs is a shell. You can also compare is to using ssh or rexec to run a specified command somewhere else, though there is no networking involved.

The concept that every process has got a UID and GID which controls its access to files and system calls in extended in Solaris by a new field storing the process' Zone-ID. This, together with Process Permission flags and a chroot is essentially what zones are, but more on that later.

Using zlogin to run a command or create a new shell in a zone will create a wtmpx login record having zone:global as the origin of the session.

zlogin with the -C option depends on the zone's console device, and creates a wtmpx entry in the zone with the console recorded as the origin of the login session.

A few things which I consider to be good Zone management habbits:

1. Keep an entry for each zone's IP address in the global zone's hosts file, and maintain the /etc/inet/netmasks file with entries for all the subnets you will be using.

2. If you put each zone in its own file system then they can not all "fill up" at the same time. With ZFS file systems, this requires that you set quotas and/or reservations on the zones' root file systems.

Finally a very simple yet important and eventually very powerful concept, namely exporting and importing of zone configurations.

I want to demonstate this using as an example the configuration from an existing zone. First we export it and store it in a text file, like this

globalzone # zonecfg -z myfirstzone export > /tmp/zone_config.txt

Have a look at the file ...

globalzone # cat /tmp/zone_config.txt

create -b

set zonepath=/export/zones/myfirstzone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.131

set physical=e1000g0

end

Now just make a few small changes to the file - specifically we update the zonepath and the IP address

globalzone # sed '

/zonepath/ s/myfirstzone/firstclone/;

/address=/ s/100.131/100.132/

' /tmp/zone_config.txt > /tmp/clone_config.txt

Of course you could use your favourite text editor to do that, but using sed is just so sexy.

globalzone # cat /tmp/clone_config.txt

create -b

set zonepath=/export/home/zones/firstclone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.132

set physical=e1000g0

end

We will feed this config file into zonecfg. Of course you could just manually set each of those entries, but zone configurations can easily get complex - you may have multiple network interfaces, many file systems, and several other non-default settings like resource controls, something I'll get to in due course.

Right now what I have on my system (the "before" picture)

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

Creating the zone config based on this:

globalzone # zonecfg -z firstclone -f /tmp/zone_config.txt

Then the "after" picture showing the new zone configured...

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

- firstclone configured /export/zones/firstclone native shared

All that remains is to populate this new "cloned" zone with user-land bits...

globalzone # timex zoneadm -z firstclone install

A ZFS file system has been created for this zone.

Preparing to install zone <firstclone>.

Creating list of files to copy from the global zone.

Copying <188162> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <1307> packages on the zone.

Initialized <1307> packages on zone.

Zone <firstclone> is initialized.

Installation of <1> packages was skipped.

The file </export/zones/firstclone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

real 28:13.55

user 4:46.13

sys 7:18.22

And we're done!

Over 28 minutes – that is still much to slow. In the next post I will use this concept and add to it to take “cloning” to the next level - It should not take more than a few seconds.

Automating Solaris Zone creation

2008-02-21T15:57:00.003+02:00

Zones can be treated as cheap, disposable application containers. Automated Zone creation is not necessarily there to allow you to rapidly deploy 1000s of Zones (though it could certainly be used for this purpose given sufficient planning), but allows you to create and use, then delete and easily re-create zones freshly and with a consistent configuration.

You will find that most, if not all, of your zones will use the same naming-services configuration, be in the same time-zone, attach to the same network interface (just with different IP addresses), etc. Many of the System Identification and system configuration settings will be identical or very similar between the Zones.

You might even find that with each new zone you create the same set of user-ids and have them all get their home directories from a central home-directory server. Basically repeat work. Computers are, in fact, good at repeatedly doing the same task over and over, without getting bored.

If all you want to achieve is to have a clean state to which you can restore a zone easily, then a fine plan would be to use file system snapshots, something like this:

1. Preparation / Setup

1.1. Create a file system structure in which to store the Zone. Since we've got ZFS for free with Solaris there's really no reason not to use it.

1.2. Set up the Zone in this file system, and complete the configuration up to the point where you want to be able to revert back to.

1.3. Shut down the Zone and take a snapshot.

2. Using this Zone:

2.1 Make any instance specific "custom" configuration changes (add some disk space, user-ids, tweak some settings)

2.2 Start the zone and let the users loose in it.

3. Reverting to the clean status

3.1 Bring the zone down (purely to make sure that no processes have files open in the file system containing the zone)

3.2 Recover the file system back to the Snap-shot state.

3.3 Go back to nr 2 above.

Before I show an example of doing this using ZFS, suffer me to mention the other techniques involved in automating Solaris Zone creation (Each of which I will cover in a separate blog post in detail)

Firstly copying the Zone configuration. This involves creating a zone config and exporting it to a file to be used as a template in the future. Then each time you want to create a zone based on this template, you just make a few small changes such as the zone-name and IP address, then import this modified copy of the template into a new zone, after which you continue with the normal zone installation.

Using a sysidcfg file and a few other tricks to speed up the zone configuration is quite similar to using a sysidcfg file to pre-configure a system from a jumpstart, and can by used to automate settings such as the timezone, locale, terminal type, networking, and name-services, amongst others.

Cloning Zones to speed up the install process. The Zone management framework from Sun gives us the ability to "clone" a master "template" zone. This involves creating one (or more) template zones which you then leave fully installed and configured, but don't actually ever start up or use, other than to tweak their configurations. This saves time during the actual install and subsequent configuration steps.

With that out of the way, on to the example of how to make a simple disposable Zone. As always the fixed-width text represents what you should see on the screen. I highlight the bits you enter.

globalzone# zpool create SPACE c0d0s4

globalzone# zfs create SPACE/zones

globalzone# zfs set mountpoint=/export/zones SPACE/zones

globalzone# zfs create SPACE/zones/disposable

globalzone# chmod 0700 /export/zones/disposable

globalzone# zfs set atime=off SPACE/zones/disposable

Disabling of “atime” above is a personal preference thing. Now we set up a simple zone. Yours can be as complicated or as simple as you want it to be.

globalzone# zonecfg -z disposable

zonecfg:disposable> set zonepath=/export/zones/disposable

zonecfg:disposable> add net

zonecfg:disposable:net> set physical=e1000g0

zonecfg:disposable:net> set address=192.168.24.133

zonecfg:disposable:net> end

zonecfg:disposable> verify

zonecfg:disposable> commit

zonecfg:disposable> exit

globalzone# zoneadm -z disposable install

cannot create ZFS dataset SPACE/zones/disposable: dataset already exists

Preparing to install zone .

Creating list of files to copy from the global zone.

Copying <9386> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <1307> packages on the zone.

Initialized <1307> packages on zone.

Zone is initialized.

Installation of <1> packages was skipped.

Installation of these packages generated warnings:

The file contains a log of the zone installation.

For the eagle-eyed amongst you, the WebStackTooling failure is due to the fact that this is a sparse zone and I'm running beta software (Nevada Build 80). In a sparse zone the /usr file system is read-only and The WebStackTooling is trying to create or change some files. I'm just ignoring this error for now as it does not bother me.

So far, so good. Lets save a backup of what we've got so far.

globalzone# zfs snapshot SPACE/zones/disposable@freshly_installed

Now we perform the first boot and system identification. Below is an abbreviated copy-paste showing the flow of the process.

globalzone# zoneadm -z disposable boot; zlogin -C disposable

[Connected to zone 'disposable' console]

Configuring Services ... 150/150

Reading ZFS config: done.

>>> Select a Language

>>> Select a Locale

>>> What type of terminal are you using?

Creating new rsa public/private host key pair

Creating new dsa public/private host key pair

Configuring network interface addresses: e1000g0.

>>> Host name for e1000g0:1 disposable

>>> Configure Security Policy:

>>> Name Service

>>> NFSv4 Domain Name:

>>> Region and Time zone: Africa/Johannesburg

>>> Root Password

System identification is completed.

rebooting system due to change(s) in /etc/default/init

[NOTICE: Zone rebooting]

SunOS Release 5.11 Version snv_80 64-bit

Use is subject to license terms.

Hostname: disposable

Reading ZFS config: done.

disposable console login: root

Password:

Feb 20 21:33:05 disposable login: ROOT LOGIN /dev/console

Sun Microsystems Inc. SunOS 5.11 snv_80 January 2008

You may want to make a few more changes now that the Zone is running. Some ideas may be to set up User-IDs, enable/disable some services, and set up some NFS and/or autmounter file systems.

# mkdir /export/home

# useradd -c "Joe Blogs" -d /export/home/joeblogs -m joeblogs

# passwd joeblogs

Assuming you've done all you want, this is the point where we have a cleanly built zone, running, and essentially the point that we would like to be able to return to after we did whatever make-and-break or sandbox testing. The Zone should be halted before we take the snapshot, even if only to close all open files.

# halt

Feb 20 21:33:12 disposable halt: initiated by root on /dev/console

Feb 20 21:33:12 disposable syslogd: going down on signal 15

[NOTICE: Zone halted]

[Connection to zone 'disposable' console closed]

Now just take another ZFS snapshot:

globalzone# zfs snapshot SPACE/zones/disposable@system_identified

=================

Now the Zone is ready for you to let your users loose in it. Allow them to have full root access, go crazy, run "rm -r /", etc.

globalzone# zoneadm -z disposable boot; zlogin -C disposable

zoneadm: zone 'disposable': WARNING: e1000g0:1: no matching subnet found in netmasks(4) for 192.168.24.133; using default of 255.255.255.0.

[Connected to zone 'disposable' console]

Hostname: disposable

Reading ZFS config: done.

disposable console login: root

Password:

Feb 20 21:40:11 disposable login: ROOT LOGIN /dev/console

Last login: Wed Feb 20 21:33:05 on console

Sun Microsystems Inc. SunOS 5.11 snv_80 January 2008

Now perform some "work" - Create a few directories, modify some files, etc. I chose to run sys-unconfig.

# sys-unconfig

WARNING

This program will unconfigure your system. It will cause it

to revert to a "blank" system - it will not have a name or know

about other systems or networks.

This program will also halt the system.

Do you want to continue (y/n) ? y

sys-unconfig started Wed Feb 20 21:40:30 2008

sys-unconfig completed Wed Feb 20 21:40:30 2008

Halting system...

svc.startd: The system is coming down. Please wait.

svc.startd: 59 system services are now being stopped.

svc.startd: The system is down.

[NOTICE: Zone halted]

Then, back in the global zone, examine the available ZFS snapshots:

globalzone# zfs list

NAME USED AVAIL REFER MOUNTPOINT

SPACE 684M 14.1G 18K /SPACE

SPACE/zones 684M 14.1G 19K /export/zones

SPACE/zones/disposable 684M 14.1G 624M /export/zones/disposable

SPACE/zones/disposable@freshly_installed 790K - 523M -

SPACE/zones/disposable@system_identified 59.2M - 611M -

These four commands can go nicely into a little "revert" script.

globalzone# zfs clone SPACE/zones/disposable@system_identified \

SPACE/zones/reverted_temp

globalzone# zfs promote SPACE/zones/reverted_temp

globalzone# zfs destroy SPACE/zones/disposable

globalzone# zfs rename SPACE/zones/reverted_temp SPACE/zones/disposable

That took just a few seconds, and we are ready to start using the zone again...

global# zoneadm -z disposable boot; zlogin -C disposable

[Connected to zone 'disposable' console]

SunOS Release 5.11 Version snv_80 64-bit

Use is subject to license terms.

Hostname: disposable

Reading ZFS config: done.

disposable console login:

As expected, you will find that all changes are reverted. Besides the normal application test environment, one other area where I think this would be quite handy is in a class-room situation, where you can allow the students full root access in the zone, and at the end of the day quickly recover the system to a sane state for the next day's class.

All in all that was Q-E-D. This principle, as well as the information from my previous blog posting will form the basis of the next few posts.

15 minutes to your first Solaris zone

2008-02-16T20:14:00.011+02:00

So you got OpenSolaris on your local machine and you would like to try out Zones. This tutorial will give you a 15 minute by-example guide to setting up your first zone, and after that explain a few things hands-on, showing you where what is.

There are a few things to understand about Zones before we start:

First thing is the concept of the Global Zone - the "base" operating system instance is the Global zone and all other "local zones" run under the control of the global zone.

Secondly you get three types of local Zones, being Full root zones, Sparse root zones, and Branded Zones. These differ in what gets copied into the zone's root file system, as well as in how they start up. Branded Zones, which starts up using custom scripts and provides special hooks into the operating system, can emulate a different kernel, but that is a topic for another day.

A Full root zone gets a complete copy of all the Solaris installed package files, which needs about 5 GB of storage. A Sparse zone only gets its own /etc and /dev, which saves on space but it uses a read-only loop-back mounted copy of other directories, such as /opt and /usr. Copying all the files into a Full root zone takes some 10 minutes longer than setting up a sparse zone and uses 5 GB more, but because /usr and /opt is writable it results in a more flexible (read more useful) example environment.

Normally before you set up a zone you would need to do thorough planning, taking care of a things like disk space/file systems, CPU and memory resource allocations, maybe even special boot options. But for a quick start lets just assume defaults for most things.

You do need to work out a few details before you start. 1. Select a name for the zone. This can also be the "hostname" for the virtual OS instance in the zone. 2. Select an IP address and identify on which interface it will live. 3. Identify a spot on your file system where the Zone root will be installed.

Lets call the zone "myfirstzone", let it use IP address 192.168.24.131 on interface e1000g0, and give it a directory /export/zones/myfirstzone

mkdir -m 0700 /export/zones/myfirstzone

Yes, you guessed right, later we can add more zones under /export/zones. Be aware, if you don't restrict the permissions on the directory, the zone installation later on will fail with a security error.

The commands for working with zones are:

zonecfg - Change a zone's configuration / setup. This will create and/or modify the xml base configuration of a zone. zoneadm - Manipulate running zones, eg rebooting and showing their status. zlogin - Connect to a zone (Create a shell session in a zone or connect to the zone's console)

Without further ado, creating the zone using zonecfg zonecfg -z myfirstzone zonecfg:myfirstzone> create zonecfg:myfirstzone> set zonepath=/export/zones/myfirstzone zonecfg:myfirstzone> add net zonecfg:myfirstzone:net> set address=192.168.24.131 zonecfg:myfirstzone:net> set physical=e1000g0 zonecfg:myfirstzone:net> end zonecfg:myfirstzone> commit zonecfg:myfirstzone> exit bash-3.2#

The basic Zone has now been set up. We can view it using this command zonecfg -z myfirstzone export create -b set zonepath=/export/zones/myfirstzone set autoboot=false set ip-type=shared add net set address=192.168.24.131 set physical=e1000g0 end

Now to populate it with Solaris packages/files, very simply use the zoneadm install command. This process takes a good 15 minutes or more, and you should see output like this:

bash-3.2# zoneadm -z myfirstzone install Preparing to install zone . Creating list of files to copy from the global zone. Copying <187209> files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <1282> packages on the zone. Initialized <1282> packages on zone. Zone is initialized. Installation of <1> packages was skipped. The file contains a log of the zone installation.

The install process is highly disk IO intensive. If you're building a zone on the same disk from which the system is running, the resulting disk contention can cause it to run quite a bit longer, up to an hour on systems with slow disks.

Use this time to scan through the man pages for the zone commands. There is also an introductory man page, i.e zones(5).

Also while this install is running use the time to look at the output from these commands: zonecfg -z myfirstzone info

zoneadm list -vc

Once the installation process finishes, you can boot the zone and log in. During the first login, you need to login on the zone console and provide the system identification information prompted.

zoneadm -z myfirstzone boot; zlogin -C myfirstzone [Connected to zone 'myfirstzone' console] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone ... ... ... System identification is completed. rebooting system due to change(s) in /etc/default/init [NOTICE: Zone rebooting] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone Reading ZFS config: done. zoneone console login:

Now log in as root with the password you just set, and run a few commands in the zone; - Do a ps -ef in the zone, and another one in the Global zone, and compare the output. - Also compare the output from a few other commands, particularly df -h, uptime, zonename, hostname, ifconfig, and uptime. On the ifconfig -a command output, note the zone identifier for each interface, as well as the multiple loopback identifiers.

For now I will leave you to marvel at the differences and similarities, but I will point out one process running in the global zone, namely zoneadmd. It manages the zone environment, ensures that its file systems and network interfaces are mounted and plumbed, etc. There will be an instance of it for every running non-Global zone.

Disconnecting from the zone Console requires that you enter the "disconnect" escape sequence, by default ~. (A tilde followed by a full stop) If you're already working remotely using something like SSH, use ~~. - The extra tilde character will inform SSH that it should not itself act on the disconnect command, but rather pass it on down the server. It is also possible to change your escape sequences, but I'll reserve that for a small blog article on another day. Note that disconnecting the console is NOT the same as logging out of the zones - processes like prstat will continue to run on the zone's console and you can reconnect later by again by running "zlogin -C myfirstzone"

Now that the zone is running, lets do a few interesting things with it.

First, in the Global Zone, have another look at what you see from zoneadm list -vc. Other interesting command are "prstat -Z", as well as "df -Z"

Next try out zlogin sans the -C option which would put you on the console. Use exit, logout, or ^D to disconnect.

Create a new user in the zone zlogin myfirstzone mkdir /export/home useradd -d /export/home/joe -m joe passwd joe

Use SSH to login to the zone from another machine, using the zone's IP address. Confirm where you logged in to using hostname, and zonename.

Now lets start an FTP server in the zone: zlogin myfirstzone svcadm enable ftpd

zlogin as used above runs the specified command directly in the zone. You can check the results using by ftp-ing to the zone, though you need to use a user like joe created above, which can login via the network.

Finally, lets add some disk space to our existing zone. This could have been done during the configuration process, but that would have defeated the purpose of showing how to add something to a zone.

I don't have an extra disk device, so I'll use a loop-back (lofi) device. cd /export/zones mkfile 1g fakedev_for_myfirstzone lofiadm -a `pwd`/fakedev_for_myfirstzone newfs /dev/lofi/1 /dev/rlofi/1: Unable to find Media type. Proceeding with system determined parameters. newfs: construct a new file system /dev/rlofi/1: (y/n)? y /dev/rlofi/1: 2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors 1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632, 2083232, 2092832

There are at least 3 ways in which this file system can be added into the domain. I will mount it in the global zone, then loop it into "myfirstzone" (This allws you to mount it into multiple zones should you so wish.)

mkdir mountpoint_for_myfirstzone_space1 mount /dev/lofi/1 `pwd`/mountpoint_for_myfirstzone_space1 zonecfg -z myfirstzone zonecfg:myfirstzone> add fs zonecfg:myfirstzone:fs> set type=lofs zonecfg:myfirstzone:fs> set special=/export/zones/mountpoint_for_myfirstzone_space1 zonecfg:myfirstzone:fs> set dir=/space1 zonecfg:myfirstzone:fs> end zonecfg:myfirstzone> verify zonecfg:myfirstzone> confirm zonecfg:myfirstzone> exit

Make the change take effect zoneadm -z myfirstzone reboot

Log back into the zone, and look at the output of df again. Compare this with the output from df in the global-zone ...

I'll post a follow-up article explaining some of the more advanced zone configuration topics, but hopefully you'll be impressed with how easy this all is. If you want to know why zones work, thing chroot jail combined with ppriv's (man privileges)

Is [insert_OS_here] ready for the desktop

2007-10-27T20:25:00.000+02:00

What a daft question. Yet I've heard it repeated so many times that I almost get tempted to answer it.

Who's desktop, that is my counter question. Of course it utterly depends on what you want to use your desktop for, and more than that, HOW you want to employ your desktop. For some people no computer will be able to replace what they do on a drafting board, with an air brush, by means of filters and lenses, or using their lathe or whatever else. Those are probably the real artists.

The rest of us content ourselves with playing at it using whatever software we can find that best meets our needs. I loudly proclaim that don't have a single MS Windows file anywhere on my computers, but I secretly reserve the right to open a remote desktop onto my wife's PC from time to time and there run my favorite home floor planning program. (I have yet to find a floor planning/home design program which runs on anything other than MS Windows). By which I mean that each operating system has got some gaps and some areas where it excels.

I run Nevada on my work laptop, because it is the only operating system that is ready for the desktop. Qualifier: When at a customer's site I often need to be able to run core dumps, crash dumps, and explorers though analysis tools, jumpstart servers, and perform a few other miscellaneous tasks which are much harder to do on any other operating system.

On the other hand Linux is the only operating system that I can qualify as desktop ready when it comes to my normal desktop PC. On this particular system I try out new programs, fiddle with kernel options, play with hardware bits, do some (amateurish) video and photo editing, etc. I'm sure I could do most of that on [Open]Solaris or even Windows, but I'm not that masochistic. Linux just makes it easy for the enthusiast to get under the bonnet of the OS and scratch around. After a discussion with a colleague a few days ago I must say that this is not always a benefit (Linux by its very nature makes it difficult to support in a mission critical environment)

But the point I am trying to make is that you need to decide what you want to do, and how you want to go about doing it. Then next you need to consider whether there are any trade offs. Can you live with your Linux system not supporting your webcam, can you live with the limited selection of software applications available under Solaris, or can you accept Windows' inherent security and stability issues.

So next time you're tempted to ask whether your operating system is "ready for the desktop", please do qualify the question with what you use your "desktop" for.

Tutorial: Caveat with redirecting from su or sudo

2007-08-21T11:57:00.001+02:00

One of the reasons why Linux is more secure than its competition is because it clearly distinguishes between running programs as an administrator and running programs as an "ordinary" user.

You should always run your programs with the least amount of privileges needed for them to work. Running a web browser as root for example is just looking for trouble. So you've embraced the idea and you are logging in as an ordinary, unprivileged user most of the time for normal work, using su or sudo for the occasional change you need to make to a printer configuration or to mount a new disk, etc.

This is a highly recommended practice. But there are a few situations where the use of su or sudo to raise privileges does not do what may at first seem logical.

Example: You want to append a line to your /etc/fstab file using the echo command

$ NEW_LINE="/dev/scd0 /media/cdrom0 udf,iso9660 user,noauto 0 0" $ sudo echo "$NEWLINE" >> /etc/fstab
bash: /etc/fstab: Permission denied

If you were to try the above you will notice that sudo doesn't even prompt for your password.

The problem occurs because of the permissions with which the redirection is being done.

What happens in the above case is that the shell, which is running as your ordinary, unprivileged user, tries to open the output stream attached to the file, /etc/fstab before it tries to execute the command "sudo". This fails, so sudo never gets executed. The same would happen with su (though this particular example would require some quoting trickery to make it work).

There are two ways to work around this problem. The first is to put the echo command, with its redirection, into a little one-line script, and then use su or sudo to execute the script. This is cumbersome as you need to first create the scriptlet.

The second workaround is to use a command which will open the file in append mode. Linux supplies us with the handy "tee" command, which can be used like this:

$ tee -a /etc/vfstab

This command will save a copy of all of its output in the file /etc/vfstab. the -a option means it will be appended to the existing file. To obtain the desired effect, we need to run the command with root privileges, and we obviously need to give it some input, like this:

$ echo "$NEW_LINE" | sudo tee -a /etc/fstab

What happens here is:

Firstly, echo is executed with the priviledges of the normal user. There is no need, buit if you do add "sudo" to the echo command it would not cause any problems besides wasting some CPU-time.
Secondly, the shell then re-directs the output from the echo to the sudo command.
The sudo command runs "tee" as root, and attaches tee to its own standard input and output streams. Therefore tee receives the standard input redirected from the echo command.
The tee command, running as root, opens the output file. This is important - in the case of the redirection the output file was being opened by the parent shell, not by the command which has got raised privileges as in this case!
Once all of this preparation work is in place, the output goes through and all is well and done.

So to summarise, redirecting the output from su or sudo to a restricted file or directory does not work because the redirection itself is done by the parent shell and therefore with the current user's privileges.

Have faith in Linux

2007-06-19T11:35:00.000+02:00

Linux does not have an inherent right to exist. Now that may sound like blasphemy to many Linux aficionados, but let me explain.

I've just read one too many posts in Linux support forums where a new user coming from a Windows background have a problem with something that worked or were easy to do under Windows, and seems impossible or hard to do under Linux. Often the person is taking a slightly accusatory tone, maybe because they have been fooled into thinking everything will be plain-sailing. The unfortunate all-to-often response that I see is that "Linux is not trying to compete with Windows".

A statement like that is a wimpish cop-out, and is wrong on so many levels it isn't even funny. People who read this response gets the message that Linux cannot compete with Windows. Before you write this again, go read one of the many "Why Linux is Better than Windows" pages, and think about that statement again.

Of course Linux is competing with Windows: A computer runs only one operating system at a time, either Windows or Linux or some other operating system (barring virtualization of course).

Importantly, without users using, and therefore computers running Linux, there is no Linux. Developers need motivation to write drivers and programs, and with no users, much of this motivation disappears, and Linux disappears into the annals of time as a futile though interesting exercise in creating an alternative to Windows.

So without Users, there is No Linux. Linux' right to exist resides in its user base, and every user who goes back to Windows because of insensitive answers is one less reason to develop Linux. You may consider each sensitive, well-thought-out, positive, encouraging answer to a frustrated users' plea for help to be an investment into the user base which will ensure that tomorrow you still HAVE a Linux platform.

Maybe we should take a page out of the fanatic "fanboy" Linux followers' book and become a bit more aggressive in our approach to promoting and backing Linux as a better-than-Windows platform.

People will always compare their experience of Linux with what they are used to getting from Windows. Unfortunately human nature have us focus on the negatives, and this means people overlook the great features of Linux and open source software because of a few small troubles.

I find there are two things that need to be said when new Linux users are frustrated with the platform.

One, remind the person that Linux development is behind on some areas due to restrictions, such as DRM and copyrights existing in the Windows software, which slows down or outright prevents development of software for Linux. At the same time remind the user that in many more areas Linux is far ahead of its Windows competition.

And second: encourage the person to continue looking for a solution - New programs and packages appear every day, so a newly hatched solution might exist but still be unknown to most people.

So go and arm yourself with some real information on why people should switch completely to Linux so that you don't come across as a fanboy, and stop wimping out when Linux does fall short of its competition.

Tutorial: Redirection and Pipes

2007-06-09T20:29:00.000+02:00

Today I realized I should not gloss over two seemingly simple items, the first being pipes and redirection, and the other being substitution. This article will explain the first of these for the benefit of those new to Linux and Unix how to link together simple commands to build powerful tools.

Before I start, a quick introduction to the Unix command-line syntax is appropriate.

The simplest Unix command is just a command word, which is the full path name of the file to be executed, starting with a slash representing the root. For example, if you enter /usr/bin/ls, the shell will run the ls program found in the /usr/bin directory.
Similarly if you enter a command without specifying the directory, the shell will search all the directories specified in the PATH variable, in the order they are listed, and will execute the first program it finds with the name specified.
Slightly more complex commands take options and/or arguments. These are extra "words" which modifies the behavior of the program. For example the ps program lists processes associated with your terminal session, but if you add the "-e" option, it will list all processes running on the system.
Simple options which turn on a different behavior in the command are called switches, and these can be stacked. For example the -f switch, which causes ps to include additional details about each listed process and the -e switch mentioned above, can be stacked, like this:

$ ps -ef

... to produce a listing of "every process" with "full defaults" A note on the formatting in this article: When a line is preceded with a "$", it means that it is a command to be entered at the shell prompt, though the $ itself must not be entered. The usual shell prompt is a $ for normal users, or a # for root, though your system may well show additional details in the prompt string. Lines in blue text and that does not start with a "$" is the output generated by the command. There is more to commands, but this is enough to be able to understand the rest of this tutorial, so without further delay on to the real subject: Redirection and Pipes. Redirection allows you to "store" the output from a command directly into a file on disk, or to read lines from a file on disk.

$ ps -ef >tmp/processlist.txt

If you run the command above you get ... nothing, and you get it fast.. That's right. The ps -ef command on its own produces the expected output, but the ">tmp/processlist.txt" portion causes this output to go into the file /tmp/processlist.txt in stead of appearing in your terminal. You can examine the newly created file with the ls command (Which "lists" information about files and/or directories)

$ ls -l /tmp/processlist.txt 
-rw-r--r-- 1 user1 user1 23100 Jun 4  10:06 /tmp/processlist.txt

And you can actually view its contents with any one of the "cat", "less" or "more" commands, like this:

$ cat /tmp/processlist.txt

Now that you have this file on the disk you can use it over and over. Enter the above cat command again, and it shows the file contents again. Much more interesting tough, is to filter out specific lines from the file. The head command prints the first 10 lines from the specified file.

$ head /tmp/processlist.txt 
$ tail -4 /tmp/processlist.txt

And you guessed it, tail prints the last 10 lines. Specifying a switch with head or tail, like the "-4" above lets you to control the number of lines being displayed if you want something other than the default 10. Much more interesting than "head" or "tail" is grep, which I used extensively in a previous tutorial.

The "grep" command filters lines out based on a special rule for searching, called a "regular expression". These can be quite complex, but does not always need to be, e.g:

$ grep root /tmp/processlist.txt

The above command reads the specified file and prints only the lines that contain the string "root". You can also get grep to do the inverse, like this:

$ grep -v root /tmp/processlist.txt

Note the "-v" switch which modified grep's behavior to print lines not matching the expression. In the above examples, "grep", "head", "tail" and "cat" were expressly told what file to open and search through. The /tmp/processlist.txt filename specified is "an argument" of each command. Combining grep searches with redirection allows us to create more stored files for further parsing.

$ grep root /tmp/processlist.txt >tmp/root_processes.txt 
$ grep -v root /tmp/processlist.txt >tmp/nonroot_processes.txt

This then creates two files, with complementary information - the first with all the lines containing the string root in the file /tmp/root_processes.txt, and the other file with all the lines that does not contain this string. In the redirection examples above the "greater than sign" (>) was used like an arrow pointing from the command ... to the file. I mentioned above that redirection can also read from a file, in which case you just need to switch the direction of the "arrow" For example

$ wc -l < /tmp/nonroot_processes.txt

Will read the file /tmp/nonroot_processes.txt and print the number of lines found. (The wc command is the so-called "word count command", and the -l switch is used to modify its behavior to count lines in stead. The "grep", "head", "tail" and "cat", etc commands could also have been used in this fashion. See for example:

$ cat

which is almost identical to

$ cat /tmp/root_processes.txt

In fact, the difference is of academic importance only, and has got to do with how the file is opened and who owns the file handle. The output is identical. However in the case of wc, the output is not the same. GASP! How can wc report something different depending on when the file was opened after I just said where the file is opened is of academic importance only?The answer is that wc reports the same information in that the number of lines will be correct in both cases, eg

$ wc -l

But in the first case wc does not print the file name, while in the second it does. This is because the redirected input is read via a special file handle called "standard input", whereas with files being opened explicitly by the program, the program is automatically aware of the file name and its location on disk. And with that little behavioral artifact, I have arrived at the concept centrally to all redirection and pipes (which I will get to in a minute). Unix and Linux commands write their output to "standard output", and reads input from "standard input". When you perform redirection, you literally redirect this output to go somewhere else or the reading to come from somewhere else. If you run two commands, both redirecting their output to the same file, in succession, it will cause the output from the first command to be lost. The redirection will overwrite the file. In order to append more lines to an existing file, use the double-redirection arrows (>>), like this:

$ cat /tmp/nonroot_processes.txt >/tmp/parts
$ cat /tmp/root_processes.txt >>/tmp/parts

This is especially useful in logging of events, i.e. when you do not want information about old events to be lost when new events are recorded. Next up: Piping. It is possible to take the output from one command and redirect it into another command running at the same time, without having to first store it in a file. This is achieved by means of the "|" pipe symbol. Example

$ ps -ef | more

The "more" command is made for situations where you have more lines being displayed on the screen than what can fit in the terminal, causing the lines to scroll past before you can read them. When a screen-full is reached, "more" pauses the scrolling of the output, and allows you to hit ENTER for one more line, SPACE for another screen-full, or "q" to "quit" immediately. (Note; less is the successor to more, and allows you to scroll backwards, up or down by half a screen-full, and most importantly to me, when you use it to search for words, it highlights them on the screen) The above method of "pausing" terminal output is used every day by Unix and Linux administrators world wide, especially with large files. Another example:

$ egrep -i "warn|err" /var/log/messages | less

The egrep command is an extended grep. In particular, it makes it easy to specify multiple search strings. The above command will show lines with either the string "warn" or "err", and the -i switch makes the search case insensitive, therefore you will also get ERR, Warn, etc.

The pipe to less is used in case there is a lot of output. Running the above command on your Linux machine every week or so will tell you a lot about its health as basically every veguely standard piece of software will log its messages to this file. It is possible to make longer pipe chains. For example

$ ps -ef | grep root | less

Run the ps command with switches -e and f; pipe its output into grep and filter out lines containing the string root; Pipe this output into less to make it manageable on the screen. A pipe really just redirects the output from one command into the input of another command without having to first store it on disk. You can usually only have one input and two output redirectors per command (There are some exceptions, but that is a somewhat advanced topic). That is right - two output redirectors. Earlier you saw that with redirection, the command had no output. But this command still produces output:

$ ls -l /etc/file799.xyz > /tmp/fileinfo.txt
/etc/file799.txt: No such file or directory

Basically the output is a message warning you that the ls command encountered an error situation. This error message is not printed on the standard output - In Unix and Linux, error messages are printed on a special, dedicated file, called "standard error". The below example uses all three redirections:

$ runreport </tmp/report_options >tmp/main_report.txt 2>tmp/report_errors.txt

Here the command "runreport" is some imaginary program. It will be reading the lines in the /tmp/report_options file, which supposedly controls how it reports. The report output will be stored in a file in /tmp/main_report.txt. If any errors are encountered, these will be recorded in the file /tmp/report_errors.txt You will see a 2>for redirecting standard output. The >redirection of standard output has in fact got an implied 1, for file handle number 1, which is "standard output". The three standard file descriptors are:

0 - Standard Input or STDIN
1 - Standard Output or STDOUT
2 - Standard Error or STDERR

Note that in the above examples, the file to which the standard output is being directed does not capture the error messages. It is possible to create a single file with both the normal output and any error messages interleaved, as it is often handy to see where and when a job or process produced error messages. To do this, there is a way to redirect the standard error into the standard output. (Read that again). Example

$ runreport >tmp/report_options >tmp/main_report.txt 2>&1

The "&1" at the end of the above line means standard output, and this will receive "2> which is standard error. This is getting quite lengthy, but a final note on syntax. You can generally insert spaces in anywhere except inside a file name or command name, which has to be a single word. So the following commands are equivalent:

$ ls -l /tmp >/tmp/fileinfo 
$ ls -l /tmp>tmp/fileinfo

In fact, you can put redirection anywhere on the command line, so the below is a 3^rd valid and equivalent variation:

$ >tmp/fileinfo ls -l /tmp

However you can not break up the "2>&1" operator, or the double-redirect ">>" used for appending ouptput to existing files.

That is redirection and pipes in a nutshell. Next up is command line substitution.

Tutorial: pipes and redirection

2007-06-04T09:38:00.000+02:00

Temporarily taken off-line as the Wysiwig editor is causing havoc with the redirection tags and the resulting article shown is inaccurate!!! Should be back up in a few minutes at most...

When will MS-Windows start using the Linux Kernel

2007-05-28T13:28:00.000+02:00

Many moons ago (In April 2005) I posted a little prediction that Microsoft will start selling a product based on the Linux Kernel. This has not happened yet, but I hear other people say the same thing quite often lately. I'll quote my original slashdot journal entry here for completeness' sake:

Hmmmm
... Microsoft is making money because their Windows operating systems are popular. While it is not the subject of this Journal entry, I do want to briefly touch on why I think this is so: 1) Microsoft "allowed" us to copy and play with Windows and, as a result grow used to and become familiar with it when we were young! 2) As a result, people expect to use Windows in the workplace, be it corporate or otherwise. 3) In similar vein, Microsoft encourages game development with their free DirectX driver. This gains new followers and the cycle continues. But: I do not think Microsoft will forever be able to continue this road. In particular, I think the strength of Linux's underlying kernel is hurting Microsoft. But (on the above but) nothing stops Microsoft from building their own Linux derivative product (except maybe pride). Imagine running Linux with the full true MSFC built in, and the full MS Windows APIs available to programmers. Essentially a "windows wrapper" around the Linux kernel, sold by Microsoft. Every game, productivity application, back-office program and specialist application runs on this powerful operating system unmodified, and so does Linux and X-Windows applications (due to the built-in X-server and MS-style windowing manager). Microsoft will be able to boast online kernel upgrades, device-driver upgrades, kernel parameter tuning, and have all the Open source application in the world running on their OS instantaneously... Also all the benefits of the open source community's support carries over to this new version of MS Windows So, I see a day, not too far away, when this will be reality. In fact, sign me up as the first user of MS-Linux! P.S. Before I get any flames - I am no expert on how an MSFC call is different from an API call, or how many layers of emulation would be required to make this Windows-on-linux-kernel product a reality, but this is probably in any case a moving target.

It would only make sense. We have GNU/Linux, GNU/OpenSolaris, and eventually why not MS/Linux. Microsoft can save a packet in OS development cost, and all the Linux-vs-MS-Windows wars can stop. Back then I boldly said "sign me up as the first user", but lately the freedom of Open source software became much more important to me. I do not foresee that Microsoft will give away their MS/Linux product for free, in fact I am sure it will be a closed-source proprietary affair. But then again, Sun surprised us, so why not Microsoft? Just like Sun had, I expect Microsoft to have a long and laborious legal battle ahead of them with licenses and patents of other included products - that is if they were to ever open up their platform, so don't hold your breath.

Tutorial: awk sees text files as rows and columns

2007-05-27T14:55:00.001+02:00

Building on the regular expressions Tutorial I posted yesterday, this introduction to awk will, if you have never used "awk" before, revolutionize your shell scripting experience. As I hinted in the Title, the awk command "sees" its input as a collection of rows and columns. So the simplest awk command would be one which prints column 1 of a text file. Before I show an example I must mention two things about awk.

Awk uses the $ sign extensively in its internal language, so it is common practice to use single-quotes ('....') to escape awk "statements to protext them from the shell.
Awk's default "field seperator" is a collection of blanks, i.e. spaces and tabs.

Keeping this in mind, if we look at the output from a command such as ls -l:

root@linwarg:/etc# ls -l
total 1996
drwxr-xr-x  8 root   root      4096 2007-04-16 10:03 acpi
-rw-r--r--  1 root   root      2077 2006-09-09 21:08 adduser.conf
-rw-r--r--  1 root   root        46 2007-05-26 15:24 adjtime
-rw-r--r--  1 root   root        50 2006-09-09 21:24 aliases
drwxr-xr-x  2 root   root      8192 2007-05-26 16:01 alternatives
-rw-r--r--  1 root   root       395 2007-03-05 08:38 anacrontab
drwxr-xr-x  7 root   root      4096 2007-04-16 10:05 apm
...

We see that the file size is always listed in the 5th column. So lets print just the sizes:

ls -l | awk '{print ($5)}'

This highlights some more aspects of the awk scripting language.

The awk language surrounds statements in curly-braces.
The "$5" reminds of the shell's default variable referring to the 5th command-line argument, but here, being between a pair of single-quotes, it is interpreted by awk, not by the shell. As is immediately evident, the above command prints only the 5th column of every row of input.

The awk language has got two special "address" prefixes that can be given to commands, being the keywords "BEGIN" or "END". Prefixing on of these to a statement will cause the statement to be executed only once, either immediately before reading and parsing the first line of input, or in the case of END once after reading and parsing the last line of input. Now consider this more complex example:

ls -l | awk '{TOTAL=TOTAL+$5} END {print (TOTAL)}'

The awk script command consists of two separate statements. The second statement is prefixed by the END keyword. What this script does is, for each line of input encountered, add the value of the 5th field (implicitly converted to a numeric value) to the variable TOTAL. After all lines of input has been read, print the value of the TOTAL variable (implicitly converted to a text string). Note that awk initiates variables to zero/null values on first reference. The above example prints only a single line, showing the value of the TOTAL obtained. In fact, all awk commands takes addresses - The above examples both uses a default (blank) address that matches all input lines. Addresses can be specified as simple regular expressions, or as complex conditions. A typical use would be to add up only the 5th column for files with some commonality in the name, eg to add up all the files with names like "*txt", you can use:

ls -l | awk '/txt$/ {TOTAL=TOTAL+$5} END {print ("Total for txt files:", TOTAL)}'

The above line only executes the TOTAL=TOTAL+$5 statement when the expression /txt$/ is matched by the current input line. The expression is the "address", indicating when to execute the statement. Statements can be complex, for example to print the list of files being added up, with the total a the end, we can add an extra step into the command like this:

ls -l | awk '/txt$/ {print;TOTAL+=$5} END {print ("Total for txt files:", TOTAL)}'

Note - I used the short-form of the add instruction to keep the command from becoming too long. The print command use with no arguments as in the above example, simply emits the complete input line. awk has also got a printf statement which can be used with great results, for example to simply re-format the input from the ls -l command:

ls -l | awk '{printf ("%-30s %20d %s $%3d %8s %8s %s\n", $8, $5, $1, $2, $3, $4, $6, $7)}'

There are (probably many) better ways to do this, but this example shows some of the mathematical capability of awk:

echo 5 3 | awk '{printf ("%5.3f\n", $1/$2)}'

awk has got a few built-in variables which are constantly updated automatically. The NF variable , for example, reveals the number of fields on the current line. The NR variable contains the current input line (or record) number. Interestingly, the NF variable can be used in conjunction with a $, eg "$NF" to refer to the last field (or word) on a line, even when every line has got a different number of fields, eg:

ls -l | awk '{print ($NF)}'

This can be further advanced, for example to get the second-to-last field, one can use something like:

ls -l | awk '{print ($(NF-1))}'

Note the extra set of braces in the above example. The awk language justifies that awk scripts sometimes be placed in their own separate files, allowing the awk statements to be formatted with indentation, etc. This is especially useful when having many complex statements, and eliminates the problems sometimes experienced with the shell expanding/interpreting special characters in the awk script. I will in the future do a follow-up article on awk as this is only just barely scratching at the surface of its capabilities.

Tutorial: grep and regular expressions

2007-05-27T14:24:00.001+02:00

This little article will basically be just enough to get you working with regular expressions. Also I hope that is may serve as a nice introduction to the regular expressions family of man pages to those curious to learn more.

When parsing the messages files or output from strace (Linux) or truss (Solaris) it is often useful to use slightly more complex REs than when finding an entry in the hosts file. Even when checking for the existence of an entry in for example the hosts file from a script, you may want to avoid false matches.

The grep command is a handy, simple command which we can use to test REs. The same REs typically also work with commands like sed and awk and even to a lesser extent in the shell.

To test an RE (expression) we can simply use:

grep "expression" filename

Back to basics: The simplest RE is a character that stands for itself, and by joining together a bunch of these we get a series of characters that matches a specific string. The command "grep xyz /etc/hosts" will find (and print) all lines that contains the sequence "xyz" anywhere.

To match only entries Starting or ending with a string, you can use one of these formats:

grep ^word filename
grep word$ filename

The characters ^ and $ are special regular expressions, but only when used as respectively, the first and last item in the RE. The ^word expression matches lines STARTING with word, while word$ matches lines which END with word.

An easy way to find lines which contain both of two strings is by means of two grep commands, like this:

grep "abcmno" filename | grep "mnoxyz"

This would however match lines that contain "abcmnoxyz", as well as the fact that it will not care about the order in which the two strings appear on the line. To force a check for both entries, with a specific expression being before the other, you can use some wild-cards. Example:

grep abcmno.*mnoxyz filename

Note the dot-star ... The dot says "any character", and the * says "zero-or-more-of-them"

The dot wild-cart on its own (I.e without the * as in above) will match any one character.

grep abcmno.mnoxyz filename

This is similar to using a "?" in shell file name expansion.

The next aspect of regular expressions is similar to the "dot" whildcard, but in stead of matching any character, it matches a character from a list of possibilities. Example:

grep abc[mno]xyz

This matches any of the following

abcmxyz
abcnxyz
abcoxyz

The [...] is used to indicate a list of possibilities. The list can be quite large, and can contain some simple sequences, like [a-z] (for a through z) or [k-p] (for k through p). Another common possiblitiy is [0-9] for any "numeric" character.

A typical use of series using this method is to make one character in a search case-insensitive, for example to find both "foo" and "Foo":

grep "[Ff]oo" filename

We can combine wildcards to search for lines which contain a single number and nothing else:

grep "^[0-9][0-9]*$" filename

The first [0-9] will match a numeric in the first position on the line Note the preceding ^.
The second [0-9]* will match zero-or-more additional numerics, that is, besides the numeral matched by the first [0-9] occurance.
The trailing $ means that after the last numeric, there must be and end-of-line.
Conversely, there cannot be anything other than a numeric character anywhere on the line.

Another use of the [...] is to exclude the list. For example to match anything which is NOT a numeric, you can use:

grep "[^0-9]$" filename

This example will find lines NOT ending in a numeric. The ^ as the first character inside the [...] set indicates that the "compliment" of the [...] sequence is to be used.

It is worth noting that in shell file-name expansion, an exclamation mark "!" can sometimes be used to indicate that the search search string must be inversed. For example listing all files starting with a single dot, but excluding the double-dot (parent-directory) can be obtained like this:

ls .[!.]*

The above can be read as list files with at least two characters in the name, the first must be a dot, the second must be anything other than a dot. Note: Some shells support use of either ^ or ! equally in this regard.

The silence is too loud

2007-05-15T13:26:00.000+02:00

A lot is happening in my life, right now, but I promise that I am working on a few articles, so hang in there to all three of my regular readers. Which reminds me... What exactly is real life? Why can't my online life be a valid part of my real life? I am of the opinion that phrases like "having a life" and "get a life" are irrelevant, because I need to interact with people and websites, just as much as I need to eat and sleep. Happiness is something that has got a lot of bearing on this. I don't watch much TV, couldn't care less about jock sports, and would rather "spend my life" in front of the computer than by going to parties, sport matches and ... whatever else passes for "having a life" these days. Oh, sure I can appreciate the occasional movie. However I just normally prefer to be in control of my entertainment. I like to steer my own path through the web as I read up on whatever topic I am busy investigating, and find watching TV and spectator sports too boring due to the unidirectional, uninvolved nature of it. It feels like I'm being spammed with other people's lives! Many people have written pieces on their school-days experiences with so-called sports jocks, and while I have had my share of these encounters, I will not waste space by just echoing these sentiments. However one thing that I do want to say is that many people don't realize, or have no tolerance for, is that there are different ways in which one can live a happy, fulfilling, and satisfactory life. Mine happens to use a tool called "The Computer" quite often. It is free of much of the elements of the world that I dislike, and those elements which it does contain are usually things I can ignore or avoid. At any rate, I promise not to let the blog deteriorate into yet another dust-collector.

USB gadgets abound

2007-05-03T15:43:00.000+02:00

Before I start, a quick introduction to the Unix command-line syntax is appropriate.

The simplest Unix command is just a command word, which is the full path name of the file to be executed, starting with a slash representing the root. For example, if you enter /usr/bin/ls, the shell will run the ls program found in the /usr/bin directory.
Similarly if you enter a command without specifying the directory, the shell will search all the directories specified in the PATH variable, in the order they are listed, and will execute the first program it finds with the name specified.
Slightly more complex commands take options and/or arguments. These are extra "words" which modifies the behavior of the program. For example the ps program lists processes associated with your terminal session, but if you add the "-e" option, it will list all processes running on the system.
Simple options which turn on a different behavior in the command are called switches, and these can be stacked. For example the -f switch, which causes ps to include additional details about each listed process and the -e switch mentioned above, can be combined, like this:

$ ps -ef

& to produce a listing of "every process" with "full defaults" A note on the formatting in this article: When a line is preceded with a "$", it means that it is a command to be entered at the shell prompt, though the $ itself must not be entered. The usual shell prompt is a $ for normal users, or a # for root, though your system may well show additional details in the prompt string. Lines in blue text and that does not start with a "$" is the output generated by the command. There is more to commands, but this is enough to be able to understand the rest of this tutorial, so without further delay on to the real subject: Redirection and Pipes. Redirection allows you to "store" the output from a command directly into a file on disk, or to read lines from a file on disk.

$ ps -ef >tmp/processlist.txt

$ ls -l /tmp/processlist.txt 
-rw-r--r-- 1 user1 user1 23100 Jun 4  10:06 /tmp/processlist.txt

And you can actually view its contents with any one of the "cat", "less" or "more" commands, like this:

$ cat /tmp/processlist.txt

$ head /tmp/processlist.txt 
$ tail -4 /tmp/processlist.txt

The "grep" command filters lines out based on a special rule for searching, called a "regular expression". These can be quite complex, but does not always need to be, e.g:

$ grep root /tmp/processlist.txt

The above command reads the specified file and prints only the lines that contain the string "root". You can also get grep to do the inverse, like this:

$ grep -v root /tmp/processlist.txt

Note the "-v" switch which modified grep's behavior to print lines not matching the expression. In the above examples, "grep", "head", "tail" and "cat" were expressly told what file to open and search through. The /tmp/processlist.txt filename specified is "an argument" of each command. Combining grep searches with redirection allows us to create more stored files for further parsing.

$ grep root /tmp/processlist.txt >tmp/root_processes.txt 
$ grep -v root /tmp/processlist.txt >tmp/nonroot_processes.txt

$ wc -l < /tmp/nonroot_processes.txt

$ cat

which is almost identical to

$ cat /tmp/root_processes.txt

$ wc -l

$ cat /tmp/nonroot_processes.txt >/tmp/parts
$ cat /tmp/root_processes.txt >>/tmp/parts

$ ps -ef | more

The "more" command is made for situations where you have more lines being displayed on the screen than what can fit in the terminal, causing the lines to scroll past before you can read them. When a screen-full is reached, "more" pauses the scrolling of the output, and allows you to hit ENTER for one more line, SPACE for another screen-full, or "q" to "quit" immediately. (Note; less is the successor to more, and allows you to scroll backwards, up or down by half a screen-full, and most importantly to me, when you use it to search for words, it highlights them on the screen) The above method of "pausing" terminal output is used every day by Unix and Linux administrators world wide, especially with large files. Another example:

$ egrep -i "warn|err" /var/log/messages | less

The pipe to less is used in case there is a lot of output. Running the above command on your Linux machine every week or so will tell you a lot about its health as basically every standard Linux and Unix component will log its messages to this file. It is possible to make longer pipe chains. For example

$ ps -ef | grep root | less

$ ls -l /etc/file799.xyz > /tmp/fileinfo.txt
/etc/file799.txt: No such file or directory

$ runreport </tmp/report_options >tmp/main_report.txt 2>tmp/report_errors.txt

0 - Standard Input or STDIN
1 - Standard Output or STDOUT
2 - Standard Error
or STDERR

$ runreport >tmp/report_options >tmp/main_report.txt 2>&1

$ ls -l /tmp >/tmp/fileinfo 
$ ls -l /tmp>tmp/fileinfo

In fact, you can put redirection anywhere on the command line, so the below is a 3^rd valid and equivalent variation:

$ >tmp/fileinfo ls -l /tmp

However you can not break up the "2>&1" operator, or the double-redirect ">>" used for appending ouptput to existing files. That is redirection and pipes in a nutshell. Next up is command line substitution.

A review of the Nokia 9300 lite

2007-04-17T15:46:00.000+02:00

A friend asked me for a summary of the good and bad points on my cell phone, the Nokia 9300 Lite. This soon turned into a little review of the phone, so I decided to share my findings on this phone here. I am by no means a Nokia fan. I think they are generally over priced and under featured, comes with too few accessories, and have a few too many Nokia-specific quirks which are designed to lock you in. When it became time for me to select a new phone, I got this one due to, at least in part, circumstances outside of my control. Despite this, the phone does have some positive features! The good:

Sending text and multimedia messages works well enough. It uses your contacts list, including "groups" in the list, automatically, and indicates how many characters you have typed and when/if the message becomes a concatenated message "as-you-type". There is no dictionary or predictive text input however.

SMS storage is good. I currently have some 2000 messages in SENT items and Archive. You can search the SMSes, but this is slow (go watch a few episodes of [insert favourite series here]). You can define many folders and sub-folders for organizing messages.

The calendar is one of its best features, It syncs perfectly with Outlook and Lotus Notes. I will write more on it at the end.

I like the keyboard. This is the feature I bought the phone for. MXit users should all get this phone so they can start making proper use of the cheap messaging MXit gives you by typing sentences.

I like being able to see websites in a format that sort-of resembles what they look like on a computer (display is 640 pixels wide)

Downloaded apps are good. I got PuTTY, Unzip, and Sudoku :-) I've seen an FTP server. 99% of the software is not free, though, and "free download" is a pain.

Built-in backup/restore (to/from memory-card)

Recharge is quick. About 2 hours.

Comes free with many contract options nowadays

The spreadsheet is totally capable, but I've never used the Word Processing application. The display "zoom +/-" feature, (tied to a meta-control like key) works in many applications. I some applications it even switches to a kiosk-like full-screen mode.

Copy-n-paste/Cut-n-paste, which is appropriately bound to CTRL-C, CTRL-V and CTRL-X. This works basically everywhere.

Text-field navigation by means of CTRL-arrow-key to jump words, and the also CTRL-SHIFT-arrow-keys to incrementally select text a word at a time. These features together helps to make for a great user interface. (Note: These features does not work in some Java applications.)

The Bad

I only get about two days on a charge, (three if I'm lucky) and I don't use it overly much. Talk-time is OK, I get the impression that it is using the phone in folder open mode (screen lit up) that uses most battery.

Display is only 240 pixels high (under correction) but if they added 20% to the display height it would have made a helluva difference.

The default picture viewer sucks. I still need to find a replacement app. I have never tried to view a movie, but the phone includes some version of Real Player (too lazy to check version now)

The processor is too slow. The phone becomes laggy easily (searches, Unzip operations, etc)

The phone has got too little RAM. Opening a large photo causes "System Error: Out of Memory" This causes the phone to become extremely slow.

Some missing features: Camera, Vibrating alert, 3G, Wifi and a dictionary or at least predictive text input. You want to browse the web but the lacking 3G means you only get GPRS speed. This would have been fine if it didn't have such a great interface for browsing the web, which makes you WANT to use the phone to browse the web....

Only one user-programmable key. I currently have it tied to the voice/audio recorder app, and I wish for a second one to tie to the File manager application. For me, the Word Processor shortcut key is a total loss/waste.

The phone is bulky and uncomfortably in a pants or shirt-pocket. This problem can however be mostly overcome by using a good quality "holster"

The menu in the phone (external) display is really stripped down from what you may be used to on other Nokia phones. You can still make calls from the directory, send SMSes, add new entries with basic fields into contacts, and find most "Settings" options, but you cannot compose or view multimedia messages, and you can select a profile, but you cannot change profile settings. You can add a number to an entry in the address book but you cannot access most contact fields. This is really more of a "getting used to it" issue than a bad point, but there it is.

Also often some options in applications aren't immediately obvious. For example you don't know that you can cancel just about any dialog by pressing ESC, because it isn't indicated anywhere. The "Menu" button works like the Mac OS menu - some options are global, and other sub-menus are context (current application) specific (Once you get used to looking for functions and options on the main menu this becomes a positive point!).

Finally, some Java applications assume that the hot keys are below the display, and draws the function indicators in the bottom left and right hand corners of the screen, but the buttons are situated along the right-hand side of the screen.

The Ugly: I like making audio recordings for later reference. But I cannot seem to find a way to convert these files to a format that is playable on a PC, and I absolutely hate anything that uses legacy formats or standards as a means of locking you in to a specific vendor's products. Worse, if you set the Log to record everything and wrap after 999 days (the max), the log rapidly becomes big. When this happens, accepting of incoming phone calls becomes laggy, only if the caller number is hidden (private number). After a few days, the lag becomes longer than the time that the phone rings before answering the call. To be a bit clearer, what happens is the following: The phone receives a call and starts to ring. You press the "Accept" button, and the ringing tone stops, but the display continues to show a flashing indication of the new incoming call. The phone lags in this state, while the person on the other end continues to hear that you have not yet picked up the call. After some lag, the connection is eventually made. This lag increases as the phone's log gets longer, until it exceeds the time that the network will let the call ring before diverting to your voice mail. This only happens for calls from private-numbers, and I am hoping that a software upgrade will fix or at least make it better. Note: The log records all communication events, eg, Data calls, Phone calls made, received and missed, GPRS connections, SMS and MMS messages sent and received, etc. The Calendar deserves a paragraph on its own - It has got ample options to satisfy the most demanding user, including a Description and a Notes field on each entry, setting of the Alarm time before each event can be adjusted in minutes, hours or days. Repeating events can be Weekly, daily, monthly-by-day (3rd Wednesday), by date (27th), yearly (3rd Monday in Feb) etc. Repeating events can be indefinite or up to an end-date. Events can have a status (crossed out, public synchronization, etc). You can color-code events, and events can be either "all-day" or "timed". You also have various views, e.g. Day, Month, Week, Yearly schedule, etc. The Contacts area is another strong-point. You have the ability to set up profiles, e.g. for friends, family, business contacts, etc. These "profiles" allow you to basically "hide" certain unused fields, eg You may not care about the birthday field for colleagues. Another feature that I like is that you can add duplicates of fields, and it will then name them sequentially by default (Email Address 2), or you can rename them. You can turn searching on specific fields on/off, but not all fields are indexed. If you turn on searching on non-indexed fields, you get warned that searches will be slower. In my experience the impact of turning on searching on these non-indexed fields are not worth the performance overhead, which is quite severe. Searching is incremental and matches on any start-of-word, in any field for which searching is turned on. So if you type "Mic" it will find "Sun Microsystems" (company Field) as well as Michiel (First name field) ... The organizer interface could have been better integrated with the phone, e.g. if you search for a person in the contacts you cannot make a call immediately. Fortunately if you then switch to the telephone application, the last person from contacts will be highlighted automatically. Not all aspects of the phone is customizable either. For example you can select a color scheme to use on the main part of the interface, but the "notes" background will always be a light tan/brown color. I use the phone to log in to my Desktop PC, using the free PuTTY client. The keyboard does not have a pipe (vertical bar) character, but pressing Shift-Control-P will send the pipe character. Similarly pressing Shift-Control-Q will send a back-quote. The phone works well enough for this purpose. I need to just also mention the PC application suite which is included with the phone. It syncs well enough with outlook and with Lotus Notes, but beyond that it is pretty basic, and barely worth mentioning. I'll call it minimally functional. For me personally the single biggest detracting factor is that the phone is under-powered in terms of memory and processor speed. Sometimes after going to the "Desk", the phone initially lags before you can move the highlight to select another item using the arrow keys. Then when Entering to open that item, the phone once again lags. For a comparison test I timed the opening of some Java applications on this phone and compared it with a Sony Ericsson K800. Both the MXiT and the Opera Mini client were about 10 times faster to open on the K800. Maybe I am just impatient, but I thinks that the slow performance and laggy interface makes for a generally bad experience on an otherwise very usable phone. Do you have any thoughts, opinions or ideas on this phone? If so, feel free to comment!

A file server for home (Part 2)

2007-03-30T16:44:00.000+02:00

In this second part of "A file server for home" I will introduce a few factors which I am currently considering in making the decisions about what my file server will eventually look like. Monetary constraints: That money would be a factor is so obvious it is almost overlooked. It is the single most important contributor in any business decision, and while one might be tempted to think that power users have a large budget for computer spend, there exist a certain culture among many hackers, particularly in the "over-clocking" community, in which one try to make the most of the least through being clever in stead of by throwing money at the problem. While I am not into “over-clocking”, I can still respect the art from the sideline. I am looking for a cost-effective way to achieve resilience and stability, and the principle remains: I will try to build the best file server possible while spending as little as possible through (hopefully) being clever with configuration decisions. With this in mind I am going to try to do two things: I will attempt to reuse what I have, and when I cannot help it, I will buy components with an upgrade path and maximal life expectancy. Physical Constraints: One of the implied ideas of a file server is that it is going to be used to house a number of disk drives in the chassis. Keeping in mind that I want to reuse some of my existing hard drives and want future expansion capability, it is clear that the chassis must not be too limited in terms of the number of drives that it supports. I had a look at the cases that I have lying around in my garage, particularly a lovely, if somewhat older, full tower case. Unfortunately none of them are ATX form factor compatible, and I will have to use a motherboard which supports SATA hard drives, which implies that the motherboard will be fairly new and therefore will be an ATX form factor board, which effectively eliminates the use of any of the unused cases that I already have: Hence the need for a new chassis. I started searching and reading reviews on web sites, and one thing that frustrated me was the limited space for installing hard drives in many cases - Most top out at 8, sometimes no more than six drives are supported, and that is counting the bays to be used for CD/DVD and floppy-disk drives. Only the very expensive cases support ten or more disk drives. Additionally, I hoped to find a case in which the disk drives are side-accessible so that I would be able to service drives without having to resort to removing the graphics card, etc. As luck would have it I found someone who wanted to sell a case because it was too large and too heavy to carry around to LAN parties. The case in question turned out to be the Thermaltake VA8000. With a native capability to house 14 drives and the ability to be extended further by using special drive bay modules, it was an almost perfect fit to my requirements. On checking the condition of the case I found that there was some scratches on the side panel, the keys are missing and the front doors would no longer latch into the closed position. These are all really minor issues, and while cleaning the case I discovered that the little magnets meant to keep the doors from swinging open had just fallen out and were stuck to metal parts elsewhere in the chassis, so that was easily sorted with a bit of crazy-glue. What is more is that the case's monstrous size makes it easy to access the drives even with a bunch of cards installed, despite the fact that the drives are not side accessible, so this chassis is going to form the perfect basis for my file server. In fact this case deserves a full review on its own, so watch this space for an upcoming post on this topic. I still need to finalize some of the other components for this server, but currently the items that I am considering are the following:

Motherboard: I have a Gigabyte GA-K8NXP-SLI motherboard which I am currently using as my desktop "workstation". The SLI part is way overkill for a server, but this motherboard sports no less than eight SATA ports and two IDE controllers, for a maximum of 12 drives.
CPU and Memory: Seeing that I will need to replace the motherboard in my desktop and the new motherboard is unlikely to use the same socket type, it means that my existing processor and memory will be freed up, and will serve just fine in the server. Therefore the server will inherit my existing AMD Athlon 64 3000+ processor and 2 GB of DDR400 memory.
Graphics card: This motherboard does not come with an on-board graphics card, so I will have to provide one. My existing card is a Gigabyte GV-3D1 dual 6600GT, which would be totally overkill for any server, but might not be compatible with whatever new motherboard I eventually get for my desktop. I will test the GV-3D1 on my new motherboard and if it works it will stay in my desktop and I will just use an old PCI graphics card in the server. Otherwise I will use it in the file server purely to prevent it from being wasted, and then I would need to start evaluating what to buy for my desktop, though that is an entirely different discussion in itself.
DVD-Writer: The drive I have right now is a very capable LG Lightscribe, but is slower on some media formats, particularly the DVD "minus" variety of disks. I will buy an extra DVD-writer, and then decide which one of the two goes into my desktop, and put the other one in the server. This is not something that needs to be decided now, but it will depend on what my usage pattern turns out to be: Will I be writing more disks on my desktop or will I need it more often in the server to make backups...?
Tape Drive: I am hoping to find a tape drive to ease the backup process, ideally either an LTO 3 or LTO 4 drive. This is something that I will add to the file server when I find a suitable device.
Hard Drives: The server will start off by inheriting the six disk drives currently in my desktop system, these being: an 80 GB IDE drive, a 320 GB IDE drive and four 160 GB SATA drives. More drives will be added over time as and when required.
Power Supply: Hard drives do not actually use huge amounts of power. It is typical for a drive to use around the 10W mark, while during peak usage or start up they drives can use up to about 30W each. Hard disk spin up is staggered, and if I assume that out of about 10 potential drives I am unlikely to often have more than one drive operating at peak performance at any one point in time, with maybe two more drives operating at around the 20W mark and the rest mostly idle at 10W, I will realize around (7*10) + (2*20) + 30 = 140 W for the disk drives. The graphics card could potentially peak at anything from 20W for an old PCI card to around 100W if I were to use the GV-3D1 card (I don't expect to ever see the graphics card in the server peak but I will use peak values here just to be safe). The CPU, RAM and Motherboard chip set adds another 150W during peak usage, bringing the total to around 400W. This is peak usage, but a 500W power supply gives me some head-room for movement and in any case you do not want to operate at the limit of a power supply's capacity as it is a likely source of instability. A good power supply is something that I will need to purchase.

The Gigabyte motherboard has got one caveat that I need to mention: Four of the SATA ports are provided by a Silicon Image SI3114 controller, which does something non-standard to the disks connected to it. What I have found is that with drives that I have previously used on the SI3114 the partition table is not recognized when you try to use them on a different SATA controller. This also happens if I try to use one of these drives in an external USB enclosure like the NexStar III SATA enclosure that I have. On the other hand, disks which were previously partitioned while connected on the nVidia chipset's built-in SATA controller is recognized on other motherboards and in my external enclosure without any problems. The reason why I mention this is that if the motherboard ever should fail, I will be in trouble because these controllers are not present on to many motherboards, and so, unless I can get another motherboard with the same type of controller, I would have a hard time reading the data from the disks that were connected to the SI3114 controller. I must admit that this is not something that I have been able to verify yet, but there is this risk. Note: All of the motherboard SATA ports are set to operate in JBOD mode, which allows me to select how I want to set up the RAID in software, and also allows me to change the RAID layout later. I also still have to make a final decision on what operating system to use. I would be a poor Solaris fan if I did not at least consider Solaris as an option, but in my experience it is just more suited to a server environment where the applications you want to use are unlikely to change much and all well supported under Solaris. OpenSolaris offers me the same capability to set up zones and ZFS file systems, but many applications are easier to get to run under OpenSolaris. Seeing that I will want to use the server for several other purposes besides simple file sharing and because I have not yet identified the actual software that I will use, I will eliminate Solaris at this point just because it will be too restrictive in future. If file serving was the only purpose, then it would have been perfect. I will also eliminate Windows because it is not free, too difficult to configure, doesn’t have enough free software, and is just generally a pain to work with. If I were to use Linux I would have trouble getting ZFS to work - the Linux ZFS port is not nearly stable enough for my liking. This means I will probably have to use OpenSolaris. The decision to use OpenSolaris is not yet cast in stone, but it looks like the right choice because I will be able to set up Zones/Containers and I will be able to use ZFS features like snapshots and raid-z. The four 160 GB disks that I have will nicely make up one raid-z "stripe". For the rest of the disks I will need to find an optimal way to configure them, but I still need to investigate the optimal way to set up the disk. I am also considering is to get the free Foundation Suite Basic disk management software as I know it very well and it too provides simple and online management of disk space. In part three of this series I will go in depth in how I selected an operating system for this file server.

Choosing between Solaris, Linux and MS-Windows

2007-03-19T13:02:00.000+02:00

Today I will take this very contentious, highly flame-war-provoking issue head on. Many people are likely to disagree with me, but that is OK - opinions are a big part of what makes the Internet tick. Naturally you need to start off by considering your specific requirements and the resources that you have available: Many RISC based computers support only a very specific Unix operating system, while many consumer devices such as web-cams and WiFi adapters are difficult if not impossible to get to work with anything other than MS Windows. It will make sense to take stock of your existing skills to support that environment when choosing an operating system for a server application if multiple options exist. To start off with I would like to group operating systems into three categories the way I see it. These are intentionally vague, and operating systems can migrate between the categories. The categories are:

The End-User operating system: The question "Does it do the job?" comes to mind. I will include MS Windows and MacOS into this category. For reasons I will get into soon, this includes the Server, Pro, and Home editions of MS Windows.
The Power-user operating system: "I'll hack it until it works the way I want it to work". This category without any doubt includes Linux.
The server operating system: This is for specifically supported hardware running specifically supported applications, and includes all the Uni'ces, in particular Solaris because it is available for free and on non-RISC hardware, and probably FreeBSD as well.

I'll discuss each of these categories a bit more in depth, taking them in the opposite order. In the enterprise data center we usually find many large, multi-processor servers designed and ratified to run specific operating systems, which in turn are ratified and supported for use with specific applications and databases. Vendors in this environment sell solutions where all the components making up the whole have been preselected, and include support for their solutions under a contract which often limits your freedom, for example by limiting you to specific versions of a specific operating system. In this environment, going with a supported operating system is the only sensible choice, and the big Unix vendors all have long lists of applications that can be run on their systems. The benefits of going with these types of solutions lies in that the hardware is intelligent about the operating system, being able to halt the operating system without terminating it, allowing hardware to be re-configured while the operating system is running, is able to take crash-dumps when the system hangs and presents a uniform device tree of the installed hardware to the operating system. This all provides a high level of stability and supportability, something which comes at a price, but worth it when the service delivery to your clients depends on the systems running without interruptions. That is not to say that Linux and MS Windows servers do not have a place in the data center - the exact same rule holds true: Select the platform for which your application is supported - whether it is a mainframe or a network vendor's appliance with an embedded firewall and authentication service. With Linux-based servers I often notice that the applications prefer an environment where a high degree of customizability is provided, and with Windows servers I often find that the applications have a keep-it-simple nature, at least as far as the server configuration is concerned. Linux fits squarely into the Power-user operating system category. Features like the ability of the user to re-compile the operating system to eke out an extra ounce of performance, directly modify the source code to get a specific result, the ability to change and/or replace any single component with one of your own choosing, and a wonderful following of supporters eager to help out one another truly makes this operating system into a power-user's heaven. I find myself more productive when working on Linux: this is without a doubt because I have it set up exactly the way I like it and I have a huge collection of software tools installed and configured to help me maintain the system. Also something that helps is that Linux is much more transparent in reporting what is going on "under the hood" - You can obtain accurate log files of system events, examine what program processes are doing in fine detail, and you have accurate reports of the state of every single component that the operating system controls. This all makes it easier to track down the "root cause" of problems, and in my experience with Linux, using the computer is less of a hit-and-miss affair - if it works a certain way today, it will work the same way tomorrow and the day after. Things like program crashes are much more easily reproducible, and as a result, easier to resolve. All of this requires a certain level of commitment and even enthusiasm to learn more about the workings of the computer and the operating system though, and that is probably not for everyone. The other options available to "power users" with PCs are basically FreeBSD, Solaris, Open Solaris, and Windows. Windows is sometimes not transparent enough and things do not always work the way Microsoft says it does. Solaris' hardware support is still somewhat limited, making it a difficult to justify choice. Open Solaris is maybe still too much of a newcomer to be judged fairly, tough good work is being done, albeit slowly, by projects such as Nexenta GNU Solaris, an OpenSolaris distribution providing the GNU tools in stead of the Solaris defaults. The End-user category are generally for people who don't care that their operating system limits their ability to troubleshoot faults, learn more about the computer or replace sub-components of the operating system. Many of these are company PCs used to run a selection of applications needed to run their business, the support and development of which companies have invested a lot of time and money into. 20 years ago Microsoft was probably deliberately lax in prosecuting pirates of their operating systems and application software. Most of these "pirates" were kids who were learning how computers worked, and by the time they made it into the workforce, Windows was what they expected on their computers and in the data centers. Today this large following of Windows users equates to a workforce who will pretty much need to be retrained if one suddenly wanted to switch to a different operating system. You could argue that the training is not so hard - many applications are web based these days, sending an email remains pretty much the same, etc, but this only holds true in theory: The moment many users are faced with a new login screen that looks different, they feel lost. Suddenly the user name field becomes case sensitive, and your lost user turns into a frustrated user who hasn't even had a chance to try the applications yet. Worse, every application's menus are laid out differently, and that is only once they learn which application does what. Space characters in file names suddenly cause havoc, and the slash in directory paths is turned around. To the end user, it feels as if nothing works the way they are expecting it to work and it takes months before they once more become productive. I must agree with A Russel Jones who concludes that for people who don't want to have to consider whether their applications will work with the window manager that they have installed, learn how to enter commands in terminal, or why security is a good thing when it seems to just make life harder, a Windows based PC is often the best option. Linux can be carefully configured to simulate this in a controlled environment where an IT department makes all the design decisions, installs all the software, and makes sure that the all of the hardware in use is supported - when you have people with the necessary skills. Setting up such an ideal configuration is something that will take time and effort, and that means there is a price tag attached, and this could be more expensive than just running Windows. In the end business decisions always come back to the numbers: What are the benefits (over the alternative), and how does the long-term costs of the various options compare. The Pro and Server editions of MS Windows have the same basic limitations that the Home edition have: The server edition is no more stable, powerful, transparent, manageable, or secure than the Home edition, and this puts it into the "End-user" category. It also fits in with the "keep-it-simple" tendency of this category, even if it is installed on a data center rack mounted server which never crashes. The power user is probably the best off: They have the most freedom to choose what operating system and applications they want to run, how they want to configure their computers, and can usually make their own choices about what programs to use. This is basically because they are able to support themselves when things stop working. So basically it boils down to the age-old adage: Use the right tool for the job. I suspect that you will find that nine times out of ten the categories I described here will hold true though. Disclaimer: This article necessarily relies heavily on generalization. Please keep that in mind if you decide to comment.

A file server for home (Part 1)

2007-03-13T11:01:00.000+02:00

In this first part of "A file server for home" I will cover my situation and how I decided that I need a file server. In a follow-on "part 2" I will get into the actual implementation. I am the type of person who fixes their computer when it is not broken - For me it is all about learning about the computer: how it works, how it can work better, and what else I can make it do. Re-installing the operating system, upgrading, testing any interesting sounding programs that I chance upon on the internet, it is all part of the game. All of this means that my data is at constant risk. I could make a mistake when I re-partition a disk, or a misbehaving piece of software could cause havoc with my data. To make matters worse, all of my data is at risk of failure of an actual disk drive! And I often enough get my computer into a state where it is unable to boot for a few days - this instance being a good example. Now you must understand that the content of my hard drives represent many years' worth of collecting shareware, downloading documents, Linux install disk images, recording logs of my online activities, stored email archives, my digital photos and much more. Replacing this would be all but impossible! The only answer is to take the data out of my computer and save it externally. The first question you may ask is why don't I just make backups? Oh I do.... It is just that I fall behind in keeping my DVD-R based backups up to date, and in any case, CDR and DVD-based backups have been known to be fallible. Each disk that I do make contains a little bit of everything, some photos, some documents, some shareware and what not else, which makes organizing the collection of backup disks another nightmare. What is more, to take a single full backup will take a stack of about 160 disks. So basically taking backups to DVD is just simply not a viable option. A very simple alternative option would be to just get a Home NAS device, but I have discarded the idea as too expensive, too inflexible, and too limiting. There is an alternative which will better meet my needs: A dedicated file server. Actually the word "dedicated" needs to be qualified, as you will see soon enough. Building a file server PC provides me with the following benefits:

I can build a DVD or tape drive into the file server to take backups directly without going via the network for the times when I need to dump some files onto a removable media. This can not be done with NAS devices.
I can add many disk drives to a PC - my current motherboard supports 12 drives without the use of S-ATA port multipliers or even USB-based drives, and additional S-ATA controllers and/or port multipliers can be added for even more disk drives. Very few NAS devices actually support more than two drives or even allow you to extend a raid array.
Existing NAS devices share the data via SMB only. Being a Unix and Linux user I want to have directories shared via NFS and possibly other protocols in the future.
I could install and run programs on a file server, for example to make the machine act as a streaming media server. Granted, some NAS devices already includes this functionality, but you're limited to whatever functions it comes with - you don't have the freedom to replace the software with something of your own.
A PC file server can potentially do other non-file server things, such as being a print server, Scan server, Web server, Mail server, Folding-at-home station, and of particular interest to me, a web proxy and DNS-caching server.
A PC file server can have multiple network ports for better connectivity; it could even have a WiFi port.
NAS devices do not commonly provide RAID-5 disk protection. A PC would support any level of RAID and hot-spare drives to boot. RAID-5 in software is often criticized as being "CPU-heavy", but in an idle file server I'd just be happy to give that processor something to do! Mirroring is too expensive for my "home-user" size budget.
Many NAS devices don't support power-save or sleep modes for the drives. I believe that it is important to turn the drives off to extend the life-time of the drives, particularly in an always-on server.

Essentially one of the advantages of a dedicated file server is that it does not need to be dedicated, it can do other things besides! One must of course also consider what you lose when you go for a computer acting as a file server rather than a nicely build NAS appliance:

To some the physical appearance of an appliance is appealing, but in my case the server will sit tucked away in the corner under a desk, so I do not consider this important.
NAS devices usually have some form of online disk replacement and array re-building, but I can afford the downtime to replace drives and while doing maintenance.
A NAS device is simple and probably fairly stable - you just press the button and voila ... it is running, while with a server you have some instability and, due to the added complexity, more chance for things to go wrong or stop working. Being the hacker that I am, this is possibly a good thing rather than a bad thing, but don't tell my wife.
And finally, some may consider the portability of a NAS device an advantage.

There is one other important consideration, at least to me. With a PC based file server I would be able to log in on the server and do file maintenance directly on the server. To re-organize the directory structure and layout with a NAS device would require me to copy the files back-and-forth between the device and a PC. In particular, moving files from one disk or partition to another will often require the files to be passed via the network twice. This would be slow and in practice it would probably cause me to not actually get around to sorting my data (which is currently in a very sad state of disorganization). With a server I would be able to log on into the server and then perform this organization directly on, and internally to, the server. This is important because I am more likely to actually do the data organization when it doesn't have to pass over the network the whole time, and as a result my data will be better organized and therefore ultimately more useful! I am now in the planning stage for my file server. It will probably have around 1TB of usable disk space to start off with, made up mostly from my existing hard drives. But I will talk about the design and implementation in part 2 of this little series, so keep watching this space! (In other words, add it to your RSS feed aggregator)

Today is the first day of the rest of my life

2007-03-12T21:41:00.000+02:00

The big question: What to put into my first blog entry. Of course, this came only after the other big question: choosing a home for my personal blog, a decision the gravity of which I never before really considered! Now, having gone through the seemingly simple process of selecting a blogging website, setting up a URL and picking a name for my blog, I realize just how serious a decision it is. And then to make my blog into something real of course I have to now write this first entry. Gee! Who will read my blog? Prospective employers? Friends? Colleagues? Criticizers? I guess I will not be able to please everybody. And what tone to take? Make it formal, casual, serious, slapstick? I am sure that, as with much else in life, this blog will grow and take on a shape naturally, based on whatever inspires me every day. I have of course long thought that I should take the plunge and set up a blog, but every time I thought about it I stopped when it came to selecting a site - one might say I've been having commitment issues. Deciding on whether to brave it by going on my own, setting up my own website for the extra freedom, against the simpler option of picking a specialist hosting service made it no easier. But in the end I decided to give it to the experienced pros, and now here I am. I also only had vague ideas of what to call the blog, and after much consideration came up with the title "Initial Program Load". As the name suggest this site will be mostly about computers. This is only natural as computers are my great passion. But without a doubt other things will appear, including, but not limited to, some mad ravings about politics, events, and philosophy. How this all turns out remains to be seen. Now let the show begin.