Sunday, May 17, 2009

Be who you are

I'm writing this as I sit in the airplane, feeling guilty about the carbon footprint I'm leaving (though I understand that flying is one of the more efficient means of transport) thinking back about my recent rather intense experience in Dar Es Salaam where I went to assist with investigating some really frustrating problems one of our customers is having with systems we provided that just should not prove this difficult to diagnose and resolve. Maybe more on that in a next issue.

Pole Sana.

It mean I feel with you, a Swahili expression of empathy. Like saying "I am very sorry for you". Yes, I've been collecting a few Swahili phrases. Kiswahili, the language, is spoken in a large part of Africa, but my benevolent teachers don't always agree on the meaning of phrases.

For example one tells me Jambo is a greeting used to greet kids or young children, and can be taken exception to if used to greet an older person. Another tells me that it is completely exchangeable with saying "Mambo" - "Hello". Fortunately both agrees that saying "Badai" is "good bye".

Some of my other phrases includes "Asanti" - Thank you, and "Karibu" - You're welcome, "Habari Asubuhi" - A greeting form for use in the morning, and "Habari Layo" - which can be used any time of the day to say "How are you". To say you're well, you would use "Nzuri Sana" (I am good, Very).

Some of my spellings may be wrong - I am really just trying to capture the pronunciation for myself to remember it for my next visit a Swahili speaking country.

Tanzania is a very positive country and the people here have an amazing attitude towards life. I never blogged about my experience in Kenia as I was more left to my own devices there, but still had a similarly positive experience of the people's attitude towards life is. In Tanzania however I met a number of stunning people, both from the Expat community and of the local people, and so experienced more of the local hospitality. There is an inexplicable and incredibly positive energy throughout Dar Es Salaam - its almost like magic, hocus pocus, but I have a theory that people unbeknownst to themselves sense these positive energy sources and tend to conglomerate around these, choosing to live where they “feel good” and thus forming towns and cities around the world. The whole atmosphere there makes you feel like to can achieve anything, and it is very addictive!

Sadly I seem to have become dull to this feeling in my home town of Cape Town. I remember feeling the same thing there in the first few weeks after moving there about 12 years ago.

Tanzania taught me a few things about myself, or maybe I should say re-taught me. Dar Es Salaam literally means House of Peace, and it really is a place where you will find peace for your soul. As this was a business trip I had to work and so did not get to see anything besides a little bit of the city, but at least I did get a few toes in the mud when I walked across a dirt road one morning to buy airtime for my cellphone.

I haven't had a toe in the mud in years. These past few years touching mud has been a thing which became almost inconceivable - dirt is something to be avoided at all times. But as a child I loved playing in mud, I used to dig mud pits with a little gardening spade big enough for me to sit in. I've just become preoccupied with living the clean, calculated, precise little life that I've cut out for myself, driving to the office, doing customer support work, and going back home, trying to find time to spend with my two-year-old son and still trying to maintain some sense of my own identity.

D'ar, as Dar Es Salaam is called by the Ex pats, did something to me - It forced me to re-evaluate who I am. I need to get my toes in mud a bit more often, sleep in places where mice runs over you at night, have a few cold showers and sometimes not know where I'm going to be going the next day. Sure I still need the grunt in between to pay the bills, but on weekends when you are off you need to discover some small beaches, pitch a tent when the found beach tells you to, and if you're not drinking from a stream, drink red wine while trying to identify the stars and make up names for your loved one's toes.

Toes seems to feature quite prominently in Tanzania – at least it did for me. The local people generally wear sandals, nothing else is practical in the heat and humidity that permeates everything even this late in Autumn. Toes can be quite a private thing for us “Europeans” - maybe the Swahili people feel different about that. At least the Praying mantises doesn't seem to mind walking on toes or hands, jumping onto you and walking all over you in their search of mosquitoes, of which there are more than enough in this humid country. And then of course, like I said, I've had this mud on my toes. Horrifying. Wondrous!

But the thing that really stood out for me is people's attitude and aptitude. Those are concepts which in the past few years have become more and more important to me, and forms a basis on which I tend to judge people. In short it is not what you do, but it is how you go about it, both the matter of your skill as well as your dedication and commitment. There is something of that in the expression “Do whatever you do do well”.

I've quickly learned that what I have seen in my home country of South Africa does not apply here, and it shames me. Tanzania is not a wealthy country, but the people's pride is more than justified. They go about what they do with a gusto and earnestness which I can take a lesson from.

Tanzania is a country with which one can fall in love with if you are willing to leave your comfort zone (and air conditioned four-by-four) for a bit. I have found peace in the house of peace.

Sunday, May 10, 2009

Lost Dog!

Otto, our Dachhund cross got lost yesterday. If someone found such a dog in the area near the Stellenberg High School and did a Google search I hope that they will hit on this page. Otto is a little brown dog with realy big ears. The dog is my son, Francois' best friend, so it would be realy terrible if we never were to find it again. For the record, we live in Amanda Glen, but the dog could easily walk into the Sonstraal Heights or Stellenberg area. If you saw this dog, please call Johan at 021 910 7160 or Reinette at 021 976 3453. Thank you!

Sunday, April 26, 2009

ZFS user quotas available in SNV build 114

I noted, as per Chris Gerhard's Weblog that user and group Quotas on ZFS will be available soon - the fix to bug ID 6501037 is currently slated for inclusion in ON build 114.

Once this becomes available I will have one fewer item on my list of features missing from ZFS.

Currently to limit users' consumption the workaround documented here is to provide each user with a dedicated directory on which another dataset is mounted and a quota is set. This implies that the user can only create or write to files in that specific directory. To track and limit a user's total usage across an entire ZFS pool requires User quotas - ditto for consumption by group.

According to this post by Matty the feature is implemented in a way which enforces the rule "tardily", that is it is a little "late", and also mentions that translated SIDs (eg when the directory is shared via SMB) are supported.

The PSARC/2009/204/ document here provides details of how the quotas is implemented. Two new zfs subcommands, namely zfs userspace and zfs groupspace reports the consumption, and control is by means of a set of new properties on ZFS file system datasets.

This amounts to good news all around. Maybe I should start tracking bug IDs for all of the items on my feature wish-list!

Thursday, April 23, 2009

Oracle becomes the second "IBM"

As promised, my thoughts on the merger with Oracle.

Disclaimer: We know that the deal has not yet been finalized yet, and these are my personal opinions and thoughts on the matter.

I have read many forum posts about the merger, many of them very negative about the whole deal. Whether it is people fearing that Oracle will kill off this or that product, or people who feel that it is good riddance to Sun, these all indicate a serious level of misunderstanding about the IT industry as a whole, about what Open Source software is, and mostly what Sun is about.

On a personal level the merger scares me: Change is always stressful, and we love the culture at Sun. Sun's culture of allowing the Engineers a virtual free hand in designing products is what drives the innovation and new features.

Sun's products really represent a level of innovation and quality that is hard to match. The cost of Sun's servers and storage products are often said to be too high, but when doing a like for like comparison, Sun products at the same price as those of the competition have better performance, features, power consumption, rack density, upgradability, investment protection, manageability and build quality. I know many of Sun's products well because of the way we work in the SSA (Sub Saharan African) region - The engineers here all support all of Sun products, bar none.

A little bit more about that: The engineers in this region must handle calls, ie analyze, determine fault cause, and often implement the solution (Though this may change with a split in the team having been proposed). The products we support include software and hardware, everything from NAS gateways to Cluster software, and includes many products that are virtually unknown outside of their niche markets.

What enables us to support such a wide variety of products is the quality of the products: They work as documented, and the documentation is available. In addition we have access to the engineering teams and interest groups for discussing unusual problems.

What I am trying to get at is that I have personally dealt with a wide variety of Sun's products, and everywhere I look I see supportability through enterprise level maturity. People who bash the products have had a single bad experience, and unfortunately it is human nature to base opinions on your bad experiences, and not notice when things go well.

Sun's product line includes Storagetek's tape libraries and VTL / VSM mainframe products. It includes the Fujitsu based M9000-64. It includes the Constellation blades and switches, and little servers like the T2000 and even many smaller, though older V210s and V240s that are still being used. In January I had to replace an EEPROM chip on an Ultra 5 on a scientific vessel in the Cape Town harbor!

Sun also have a major investment in SPARC processors. No, they are not the fastest number crunchers, in fact the UltraSPARC processors are quite slow when compared to the fastest processors from the competition. But scaling to hundreds of CPUs is mature technology in the SPARC camp. Multi-core CPU technology: Mature. Multi-terabyte RAM in a system: Mature technology. NUMA? Mature technology. 64-bit processors? Those came out, what, 12 years ago. Systems with over 700 GB/sec internal bus bandwidth? We got it. Adding memory or CPUs to a running system. Mature, and available for 10 years already. You can even remove those components and repair or replace them without stopping the OS, though it does require that you configure the server correctly.

IBM in comparison have a slightly wider footprint in the server hardware arena: Their systems include laptops and Mainframes, and essentially everything in between. But their software product set is not as broad as that of Sun, which includes products like the Luster file system, SunRay software, Java, StarOffice and OpenSolaris, etc etc etc.

What happens when Oracle suddenly attaches all of Sun's products to its own portfolio ... ?

In the enterprise market there are realy only two databases: DB2 and Oracle. I am well aware that PostgreSQL and MySQL and YourFavourtiteDB also have a meaningful place in the market, but those are all to a large extent "alternatives to" using Oracle or DB2.

At present Oracle is not a huge competitor to IBM, except for the Database itself. But when Oracle suddenly adds all of Sun's products to its arsenal, it turns into a different beast. Oracle becomes the second "IBM".

I must say that Oracle is unlikely to kill of many products. They may sell a few products, and I think it would be interesting to see what goes. Funding of some products may stop, but that does not need to mean the end of Open Source products.

One of my biggest gripes with Sun's uninformed critics is how they in the same forum post complain that Oracle will kill of their favourite open source application, and right after that complains about how Sun's products aren't open enough.

Listen to me: The mere fact that you worry about OpenOffice or MySQL or whatever dispearing means you admit what a large contribution Sun is making to your world.

I don't know exactly what the merger will mean for the IT industry as a whole. If Oracle changes the Sun culture to disallow the engineers from being innovative, we will see some of the competition in the market disappear over time. If Oracle sells off some products, they may get new life in another stable or they may disappear, we don't know. But every product that does disappear is a sad case and will be mourned.

If you don't think Sun's products are good, it just shows how little you know. I hope that Oracle will give new life and funding to Sun's R&D. If the Sun culture disapears, I will blame it on bad marketting that failed to turn good products into income, which ultimately resulted in Sun's demise ... but before I make any judgement calls prematurely, let's rather wait and see what happens. Oh and yes, I really do love Sun's products, especially Solaris and the servers.

Wednesday, April 22, 2009

Voting day in the South African National Election, 2009

People who know me know that I am strongly apolitical. The problems we are facing in South Africa, especially the living conditions affecting most South Africans, will not be solved by any political party. They simply are not motivated or even enabled to fix things in a satisfactory manner. In short, I believe that it makes no difference who becomes our next Government.

However, like in every one of the past five national elections (since I became eligible to vote), I today cast my vote for an opposition party because I also believe in a multi-partite government: No one party should be allowed to rule unchecked and uncontested. In much the same way that competition is necessary in any industry, opposition serves as a check to keep the ruling party honest: An only incumbent gets a default Carte Blanche, even in a democracy. Very few people are disciplined enough to remain altruistic throughout a dictatorship, if ever they were - I certainly don't believe any of the people on our existing political radar have this ability, regardless of their good intentions.

Strangely this touches on the Sun/Oracle merger, one which many fear would kill of a number of products (and thus the competition) which do not fit in with Oracle's current business model. I'll post on that tomorrow after reading some of the Oracle employees' views on the matter.

Friday, November 7, 2008

Neat way to prevent multiple instances of a script

Sometimes you need to ensure that you can never have more than one instance of a script running at the same time. This is especially important with scripts that modifies files, and if the script runs for longer than a fraction of a second it becomes more critical. Also if you have many people administrating a system, it becomes more important to ensure they don't step on each other's toes.

Basically the problem is known as the multiple writers problem, and it is solved by something called "semaphores". "Semaphores" is a feature implemented in the kernel with the purpose of providing a way to guarantee that a piece of code be made "mutually exclusive". I don't want to go into this, it is already properly explained on many websites, but have a look at the Wikipedia article if you are interested in the topic.

One technique to get around the problem of multiple instances of a script is to use the existence of a specific file somewhere to flag other instances of the script that there is already a running instance. The "touch" command does not complain about existing files, so you need to to check first whether the file exists already, exit if it does, and create it otherwise. For example

if [ -f /tmp/already_running ] 
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file /tmp/already_running and try again.
   exit 1
else
   touch /tmp/already_running
fi
....

Near the end of this script it is common to delete the file for the next use. This, however, is not the best solution.

If two instances of the script were started at nearly the same instant, what could happen is that instance 1 checks the file, finds it does not exist, but then gets kicked off the CPU so that instance 2 can run. Instance 2 then checks the lock file and also sees it is OK to continue, and then creates the lock. Instance 1 eventually gets CPU time again and, having already previously checked the lock file, believes it is safe to continue running.

This is known as a race condition, and is by definition what Semaphores are meant to prevent. But semaphores are not easily accessible in scripts, or so it might seem.

Now I know you are asking "but what is the chance of such a precice timing of sceduled cpu time to cause this kind of race condition". Yes, the chances are probably low, especially if you are the only person using a specific script. However there is a proper way of checking that we are the only instance running, and it is even simpler to implement thatn the check-file-touch-file method!

The secret lies in the mkdir command.

if ! mkdir /tmp/already_running
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file /tmp/already_running and try again.
   exit 1
fi
....

mkdir will automatically use the kernel built-in semaphores during the actual process of creating the entry in the file system. It will fail if the directory exists, and on succesfull return, the lock will be in place already, so no extra commands are needed to complete the mutex locking process.

The second part of this is to automate the release of the lock when the script exists. Typically you want the lock to be released even if someone kills the script, press Ctrl-C, or if it terminates normally or on an error.

This is done by means of a EXIT trap, and the format of using traps in bourne shell variants is:

trap "do-something-here" EXIT

This trap must be set AFTER obtaining the lock, otherwise a second instance of the script will "inadvertendly" remove the lock obtained by the first instance of the script (because the new instance will basically remove the lock which it was not able to obtain when it exists on not being unable to obtain the lock.

You obviously don't have to have an if-then-fi to print a message to the user - if you are the only person using a script, you can simplify the checking of the lock as follow:

MUTEX_LOCK=/tmp/myscript_already_running
mkdir $MUTEX_LOCK || exit 1
trap "rmdir $MUTEX_LOCK" EXIT

With the above you will simply have an error message from mkdir which you need to interpret as "the script is already running", eg:

mkdir: Failed to make directory "/tmp/myscript_already_running"; File exists

Using this technique, a whole script might look like this:

#!/bin/ksh
# This is myscript v1.0.

#Set up the running environment
MUTEX_LOCK=/tmp/myscript_already_running
...

if ! mkdir $MUTEX_LOCK
then
   echo Can not continue - the lock file already exists!
   echo If you are sure that no other instance of this script
   echo is running, delete the file $MUTEX_LOCK and try again.
   exit 1
fi
trap "rmdir $MUTEX_LOCK" EXIT
...

# Doing work which requires only one instance of the script to be running
...

# THE END

Note there is no "remove lock" statement at the end of the script. This is handled by the trap, which executes on any exit, except of course a kill -9.

Using a kill -9 should in any case only ever be used as a last resort, because it does not allow the program to clean up after itself.

Tuesday, October 21, 2008

Why X-windows is back-to-front

This is a very basic introduction to the X-windows protocol, with the purpose of explaining why the server runs on the workstation and the client runs on the server.

Actually the naming is the right way round. Per definition clients initiate connections and servers accept connections. Without trying to be pedantic or philosophical about the difference between a server and a workstation, lets just say that for the purpose of this discussion your desktop machine is the workstation, and the server is some system "service" applications, files, printers, etc from your computer room. When you start an X-windows application on the server, it needs to open a window somewhere. In this case, that somewhere is the screen of your workstation. The X-windows application, really the X-client, will open a TCP/IP connection to port 6000 on your workstation, and then via this session it will send "instructions" to the X-server on how to draw its window.

Clearly something needs to be running on your workstation to accept a connection on port 6000. This piece of software is called the X-server, and such programs are available for most if not all operating systems. In particular it is available for MS Windows and Mac-OS, and Linux and Solaris includes X-server software in the form of Xfree86, Xorg and Xsun.

The X server actually listens on port 6000+N where N is the "Display" or "instance" number. Thus, the first server is Display 0, and listens on TCP/IP port 6000. A second Display would listen on port 6000+1 = 6001, and so on, though having more than one is not particularly common.

There are a few things which seemingly complicates this matter. The first is that in many situations the X-server and the X-client runs on the same system. Essentially this is true for all X applications used on a "local workstation". But the rule still holds - port 6000 listens and accepts connections from the clients running locally. X-clients, which can be anything from Firefox to Gnome-terminal, get a DISPLAY environment variable which tells them to connect to "localhost:0.0"

The second factor is how the application is started. It is possible to walk to the server, enter the command to set the DISPLAY variable (to point to the IP address of your workstation, then start the X-client application, and when you return to your workstation you will find that it is showing the X-application (assuming that the X-server on your desktop is running and accepting connections). However it is much more convenient and more common to log into the server remotely and then set the DISPLAY variable and then start the X-client.

If you use a strategy such as the above often it quickly becomes too much effort and a way to automate the DISPLAY environment variable setting becomes neccesary. Many X-server programs include a few "automation" settings, which often includes connecting to the remote server via rsh, telnet or ssh, logging in, setting the DISPLAY variable, and then starting an X-client program such as a terminal.

In addition SSH has got some special features whereby it can tunnel the X-server setting back to your PC over the encrypted session. This is particularly handy when you have a firewall blocking incomming connections on port 6000. When starting your SSH client up to do this, the client will ask the SSH server to start listening on port 6010, as if it is an X-server for Display nr 10. The SSH server does this (if it is configured to allow this kind of connection) and then starts the shell, usually cleverly setting the DISPLAY value to localhost:10.0. Note that "localhost" refers to the server itself, thus connections to port 6010 will be picked up by the SSH daemon on that server. When such a connection is made, eg by starting an X-client from within the SSH sessions, the SSH daemon on the server accepts the connection and it knows that the specific port is associated with your SSH tunnel, so it will forward the X-windows data back to your workstation via the existing SSH tunnel. The SSH client on your workstation distinguishes the X-windows data from other data along the channel, and makes a connection to the real X-server on the workstation, and forwards the un-encrypted packets received to the local X-server.

So in summary, the X-server runs on the workstation where the display is rendered. The X-client establishes the TCP/IP session to this X-server based on the value of the DIPSLAY setting. In normal situations you will observe the connection as incomming back towards yourself, but usually we associate the "server" of a client-server application with the software running on the remote machine (relatively to where we are sitting), but in this case it is obviously back-to-front.

Note: This brief introduction glosses over the many complex issues and details. Desktop Environments, Display Managers, Composting, Direct Rendering, the Driver model, connections via other than TCP/IP protocols, etc. For a slightly more complete though still very digestible introduction, see the Wikipedia Article on the X-Windowing System. For all the information you can handle head over to the official steering group web site at http://x.org