Saturday, February 23, 2008

First steps to cloning a Solaris Zone

Today I want to just mention a few concepts that I've been deliberately neglecting - some of the ideas I mentioned in my 15-minute-to-your-first-zone guide and in my post about Automating Zone creation, and that I will still be posting about in the next few posts are based on this.

Firstly Sun has cleverly integrated Zone management with the ZFS file system.

1. If the parent directory of a Zone's root is on a ZFS file system, then zoneadm will create a new ZFS file system for the rootpath of the zone.

2. Stopping and starting the zone will mount and unmount the zone's root file system

3. Cloning a zone by means of the zoneadm utility will automatically use a ZFS snapshot to create a ZFS clone which will be mounted on the rootpath of the new zone.

This is even more interesting because prior to release 11/06 of Solaris 10, running a zone with its rootpath on a ZFS file system was an unsupported configuration.

The second thing is that resources which are only needed while a zone is running, is created and destroyed dynamically when the zone is started or halted. In particular this applies to network interfaces and loopback file systems. When you start up a zone, you will notice new entries were created for its interfaces, and new file systems were mounted. The file systems which are so managed are normally hidden from df in the global zone, but shows up when you run df with the new -Z switch.

The next concept is that of how zlogin works. You can think of zlogin as a kind of "su" command, but in stead of running a commands under a different userid, it runs a command in a different zone. The default command which it runs is a shell. You can also compare is to using ssh or rexec to run a specified command somewhere else, though there is no networking involved.


The concept that every process has got a UID and GID which controls its access to files and system calls in extended in Solaris by a new field storing the process' Zone-ID. This, together with Process Permission flags and a chroot is essentially what zones are, but more on that later.


Using zlogin to run a command or create a new shell in a zone will create a wtmpx login record having zone:global as the origin of the session.

zlogin with the -C option depends on the zone's console device, and creates a wtmpx entry in the zone with the console recorded as the origin of the login session.

A few things which I consider to be good Zone management habbits:

1. Keep an entry for each zone's IP address in the global zone's hosts file, and maintain the /etc/inet/netmasks file with entries for all the subnets you will be using.

2. If you put each zone in its own file system then they can not all "fill up" at the same time. With ZFS file systems, this requires that you set quotas and/or reservations on the zones' root file systems.

Finally a very simple yet important and eventually very powerful concept, namely exporting and importing of zone configurations.

I want to demonstate this using as an example the configuration from an existing zone. First we export it and store it in a text file, like this

globalzone # zonecfg -z myfirstzone export > /tmp/zone_config.txt

Have a look at the file ...

globalzone # cat /tmp/zone_config.txt

create -b

set zonepath=/export/zones/myfirstzone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.131

set physical=e1000g0

end

Now just make a few small changes to the file - specifically we update the zonepath and the IP address


globalzone # sed '

/zonepath/ s/myfirstzone/firstclone/;

/address=/ s/100.131/100.132/

' /tmp/zone_config.txt > /tmp/clone_config.txt

Of course you could use your favourite text editor to do that, but using sed is just so sexy.

globalzone # cat /tmp/clone_config.txt

create -b

set zonepath=/export/home/zones/firstclone

set autoboot=false

set ip-type=shared

add net

set address=192.168.100.132

set physical=e1000g0

end

We will feed this config file into zonecfg. Of course you could just manually set each of those entries, but zone configurations can easily get complex - you may have multiple network interfaces, many file systems, and several other non-default settings like resource controls, something I'll get to in due course.

Right now what I have on my system (the "before" picture)

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

Creating the zone config based on this:

globalzone # zonecfg -z firstclone -f /tmp/zone_config.txt

Then the "after" picture showing the new zone configured...

globalzone # zoneadm list -vc

ID NAME STATUS PATH BRAND IP

0 global running / native shared

- myfirstzone installed /export/zones/myfirstzone native shared

- disposable installed /export/zones/disposable native shared

- firstclone configured /export/zones/firstclone native shared

All that remains is to populate this new "cloned" zone with user-land bits...

globalzone # timex zoneadm -z firstclone install

A ZFS file system has been created for this zone.

Preparing to install zone <firstclone>.

Creating list of files to copy from the global zone.

Copying <188162> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <1307> packages on the zone.

Initialized <1307> packages on zone.

Zone <firstclone> is initialized.

Installation of <1> packages was skipped.

The file </export/zones/firstclone/root/var/sadm/system/logs/install_log> contains a log of the zone installation.

real 28:13.55

user 4:46.13

sys 7:18.22

And we're done!


Over 28 minutes – that is still much to slow. In the next post I will use this concept and add to it to take “cloning” to the next level - It should not take more than a few seconds.

Thursday, February 21, 2008

Automating Solaris Zone creation

Zones can be treated as cheap, disposable application containers. Automated Zone creation is not necessarily there to allow you to rapidly deploy 1000s of Zones (though it could certainly be used for this purpose given sufficient planning), but allows you to create and use, then delete and easily re-create zones freshly and with a consistent configuration.

You will find that most, if not all, of your zones will use the same naming-services configuration, be in the same time-zone, attach to the same network interface (just with different IP addresses), etc. Many of the System Identification and system configuration settings will be identical or very similar between the Zones.

You might even find that with each new zone you create the same set of user-ids and have them all get their home directories from a central home-directory server. Basically repeat work. Computers are, in fact, good at repeatedly doing the same task over and over, without getting bored.

If all you want to achieve is to have a clean state to which you can restore a zone easily, then a fine plan would be to use file system snapshots, something like this:

1. Preparation / Setup

1.1. Create a file system structure in which to store the Zone. Since we've got ZFS for free with Solaris there's really no reason not to use it.

1.2. Set up the Zone in this file system, and complete the configuration up to the point where you want to be able to revert back to.

1.3. Shut down the Zone and take a snapshot.

2. Using this Zone:

2.1 Make any instance specific "custom" configuration changes (add some disk space, user-ids, tweak some settings)

2.2 Start the zone and let the users loose in it.

3. Reverting to the clean status

3.1 Bring the zone down (purely to make sure that no processes have files open in the file system containing the zone)

3.2 Recover the file system back to the Snap-shot state.

3.3 Go back to nr 2 above.

Before I show an example of doing this using ZFS, suffer me to mention the other techniques involved in automating Solaris Zone creation (Each of which I will cover in a separate blog post in detail)

Firstly copying the Zone configuration. This involves creating a zone config and exporting it to a file to be used as a template in the future. Then each time you want to create a zone based on this template, you just make a few small changes such as the zone-name and IP address, then import this modified copy of the template into a new zone, after which you continue with the normal zone installation.

Using a sysidcfg file and a few other tricks to speed up the zone configuration is quite similar to using a sysidcfg file to pre-configure a system from a jumpstart, and can by used to automate settings such as the timezone, locale, terminal type, networking, and name-services, amongst others.

Cloning Zones to speed up the install process. The Zone management framework from Sun gives us the ability to "clone" a master "template" zone. This involves creating one (or more) template zones which you then leave fully installed and configured, but don't actually ever start up or use, other than to tweak their configurations. This saves time during the actual install and subsequent configuration steps.

With that out of the way, on to the example of how to make a simple disposable Zone. As always the fixed-width text represents what you should see on the screen. I highlight the bits you enter.

globalzone# zpool create SPACE c0d0s4

globalzone# zfs create SPACE/zones

globalzone# zfs set mountpoint=/export/zones SPACE/zones

globalzone# zfs create SPACE/zones/disposable

globalzone# chmod 0700 /export/zones/disposable

globalzone# zfs set atime=off SPACE/zones/disposable

Disabling of “atime” above is a personal preference thing. Now we set up a simple zone. Yours can be as complicated or as simple as you want it to be.

globalzone# zonecfg -z disposable

zonecfg:disposable> set zonepath=/export/zones/disposable

zonecfg:disposable> add net

zonecfg:disposable:net> set physical=e1000g0

zonecfg:disposable:net> set address=192.168.24.133

zonecfg:disposable:net> end

zonecfg:disposable> verify

zonecfg:disposable> commit

zonecfg:disposable> exit

globalzone# zoneadm -z disposable install

cannot create ZFS dataset SPACE/zones/disposable: dataset already exists

Preparing to install zone .

Creating list of files to copy from the global zone.

Copying <9386> files to the zone.

Initializing zone product registry.

Determining zone package initialization order.

Preparing to initialize <1307> packages on the zone.

Initialized <1307> packages on zone.

Zone is initialized.

Installation of <1> packages was skipped.

Installation of these packages generated warnings:

The file contains a log of the zone installation.

For the eagle-eyed amongst you, the WebStackTooling failure is due to the fact that this is a sparse zone and I'm running beta software (Nevada Build 80). In a sparse zone the /usr file system is read-only and The WebStackTooling is trying to create or change some files. I'm just ignoring this error for now as it does not bother me.

So far, so good. Lets save a backup of what we've got so far.

globalzone# zfs snapshot SPACE/zones/disposable@freshly_installed

Now we perform the first boot and system identification. Below is an abbreviated copy-paste showing the flow of the process.

globalzone# zoneadm -z disposable boot; zlogin -C disposable

[Connected to zone 'disposable' console]

Configuring Services ... 150/150

Reading ZFS config: done.

>>> Select a Language

>>> Select a Locale

>>> What type of terminal are you using?

Creating new rsa public/private host key pair

Creating new dsa public/private host key pair

Configuring network interface addresses: e1000g0.

>>> Host name for e1000g0:1 disposable

>>> Configure Security Policy:

>>> Name Service

>>> NFSv4 Domain Name:

>>> Region and Time zone: Africa/Johannesburg

>>> Root Password

System identification is completed.

rebooting system due to change(s) in /etc/default/init

[NOTICE: Zone rebooting]

SunOS Release 5.11 Version snv_80 64-bit

Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.

Hostname: disposable

Reading ZFS config: done.

disposable console login: root

Password:

Feb 20 21:33:05 disposable login: ROOT LOGIN /dev/console

Sun Microsystems Inc. SunOS 5.11 snv_80 January 2008

You may want to make a few more changes now that the Zone is running. Some ideas may be to set up User-IDs, enable/disable some services, and set up some NFS and/or autmounter file systems.

# mkdir /export/home

# useradd -c "Joe Blogs" -d /export/home/joeblogs -m joeblogs

# passwd joeblogs

Assuming you've done all you want, this is the point where we have a cleanly built zone, running, and essentially the point that we would like to be able to return to after we did whatever make-and-break or sandbox testing. The Zone should be halted before we take the snapshot, even if only to close all open files.

# halt

Feb 20 21:33:12 disposable halt: initiated by root on /dev/console

Feb 20 21:33:12 disposable syslogd: going down on signal 15

[NOTICE: Zone halted]

~.

[Connection to zone 'disposable' console closed]

Now just take another ZFS snapshot:

globalzone# zfs snapshot SPACE/zones/disposable@system_identified

=================

Now the Zone is ready for you to let your users loose in it. Allow them to have full root access, go crazy, run "rm -r /", etc.

globalzone# zoneadm -z disposable boot; zlogin -C disposable

zoneadm: zone 'disposable': WARNING: e1000g0:1: no matching subnet found in netmasks(4) for 192.168.24.133; using default of 255.255.255.0.

[Connected to zone 'disposable' console]

Hostname: disposable

Reading ZFS config: done.

disposable console login: root

Password:

Feb 20 21:40:11 disposable login: ROOT LOGIN /dev/console

Last login: Wed Feb 20 21:33:05 on console

Sun Microsystems Inc. SunOS 5.11 snv_80 January 2008

Now perform some "work" - Create a few directories, modify some files, etc. I chose to run sys-unconfig.

# sys-unconfig

WARNING

This program will unconfigure your system. It will cause it

to revert to a "blank" system - it will not have a name or know

about other systems or networks.

This program will also halt the system.

Do you want to continue (y/n) ? y

sys-unconfig started Wed Feb 20 21:40:30 2008

sys-unconfig completed Wed Feb 20 21:40:30 2008

Halting system...

svc.startd: The system is coming down. Please wait.

svc.startd: 59 system services are now being stopped.

svc.startd: The system is down.

[NOTICE: Zone halted]

Then, back in the global zone, examine the available ZFS snapshots:

globalzone# zfs list

NAME USED AVAIL REFER MOUNTPOINT

SPACE 684M 14.1G 18K /SPACE

SPACE/zones 684M 14.1G 19K /export/zones

SPACE/zones/disposable 684M 14.1G 624M /export/zones/disposable

SPACE/zones/disposable@freshly_installed 790K - 523M -

SPACE/zones/disposable@system_identified 59.2M - 611M -

These four commands can go nicely into a little "revert" script.

globalzone# zfs clone SPACE/zones/disposable@system_identified \

SPACE/zones/reverted_temp

globalzone# zfs promote SPACE/zones/reverted_temp

globalzone# zfs destroy SPACE/zones/disposable

globalzone# zfs rename SPACE/zones/reverted_temp SPACE/zones/disposable

That took just a few seconds, and we are ready to start using the zone again...

global# zoneadm -z disposable boot; zlogin -C disposable

[Connected to zone 'disposable' console]

SunOS Release 5.11 Version snv_80 64-bit

Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved.

Use is subject to license terms.

Hostname: disposable

Reading ZFS config: done.

disposable console login:

As expected, you will find that all changes are reverted. Besides the normal application test environment, one other area where I think this would be quite handy is in a class-room situation, where you can allow the students full root access in the zone, and at the end of the day quickly recover the system to a sane state for the next day's class.

All in all that was Q-E-D. This principle, as well as the information from my previous blog posting will form the basis of the next few posts.

Saturday, February 16, 2008

15 minutes to your first Solaris zone

So you got OpenSolaris on your local machine and you would like to try out Zones. This tutorial will give you a 15 minute by-example guide to setting up your first zone, and after that explain a few things hands-on, showing you where what is.

There are a few things to understand about Zones before we start:

First thing is the concept of the Global Zone - the "base" operating system instance is the Global zone and all other "local zones" run under the control of the global zone.

Secondly you get three types of local Zones, being Full root zones, Sparse root zones, and Branded Zones. These differ in what gets copied into the zone's root file system, as well as in how they start up. Branded Zones, which starts up using custom scripts and provides special hooks into the operating system, can emulate a different kernel, but that is a topic for another day.

A Full root zone gets a complete copy of all the Solaris installed package files, which needs about 5 GB of storage. A Sparse zone only gets its own /etc and /dev, which saves on space but it uses a read-only loop-back mounted copy of other directories, such as /opt and /usr. Copying all the files into a Full root zone takes some 10 minutes longer than setting up a sparse zone and uses 5 GB more, but because /usr and /opt is writable it results in a more flexible (read more useful) example environment.

Normally before you set up a zone you would need to do thorough planning, taking care of a things like disk space/file systems, CPU and memory resource allocations, maybe even special boot options. But for a quick start lets just assume defaults for most things.

You do need to work out a few details before you start. 1. Select a name for the zone. This can also be the "hostname" for the virtual OS instance in the zone. 2. Select an IP address and identify on which interface it will live. 3. Identify a spot on your file system where the Zone root will be installed.

Lets call the zone "myfirstzone", let it use IP address 192.168.24.131 on interface e1000g0, and give it a directory /export/zones/myfirstzone

mkdir -m 0700 /export/zones/myfirstzone

Yes, you guessed right, later we can add more zones under /export/zones. Be aware, if you don't restrict the permissions on the directory, the zone installation later on will fail with a security error.

The commands for working with zones are:

zonecfg - Change a zone's configuration / setup. This will create and/or modify the xml base configuration of a zone. zoneadm - Manipulate running zones, eg rebooting and showing their status. zlogin - Connect to a zone (Create a shell session in a zone or connect to the zone's console)

Without further ado, creating the zone using zonecfg zonecfg -z myfirstzone zonecfg:myfirstzone> create zonecfg:myfirstzone> set zonepath=/export/zones/myfirstzone zonecfg:myfirstzone> add net zonecfg:myfirstzone:net> set address=192.168.24.131 zonecfg:myfirstzone:net> set physical=e1000g0 zonecfg:myfirstzone:net> end zonecfg:myfirstzone> commit zonecfg:myfirstzone> exit bash-3.2#

The basic Zone has now been set up. We can view it using this command zonecfg -z myfirstzone export create -b set zonepath=/export/zones/myfirstzone set autoboot=false set ip-type=shared add net set address=192.168.24.131 set physical=e1000g0 end

Now to populate it with Solaris packages/files, very simply use the zoneadm install command. This process takes a good 15 minutes or more, and you should see output like this:

bash-3.2# zoneadm -z myfirstzone install Preparing to install zone . Creating list of files to copy from the global zone. Copying <187209> files to the zone. Initializing zone product registry. Determining zone package initialization order. Preparing to initialize <1282> packages on the zone. Initialized <1282> packages on zone. Zone is initialized. Installation of <1> packages was skipped. The file contains a log of the zone installation.

The install process is highly disk IO intensive. If you're building a zone on the same disk from which the system is running, the resulting disk contention can cause it to run quite a bit longer, up to an hour on systems with slow disks.

Use this time to scan through the man pages for the zone commands. There is also an introductory man page, i.e zones(5).

Also while this install is running use the time to look at the output from these commands: zonecfg -z myfirstzone info

zoneadm list -vc

Once the installation process finishes, you can boot the zone and log in. During the first login, you need to login on the zone console and provide the system identification information prompted.

zoneadm -z myfirstzone boot; zlogin -C myfirstzone [Connected to zone 'myfirstzone' console] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone ... ... ... System identification is completed. rebooting system due to change(s) in /etc/default/init [NOTICE: Zone rebooting] SunOS Release 5.11 Version snv_80 64-bit Copyright 1983-2007 Sun Microsystems, Inc. All rights reserved. Use is subject to license terms. Hostname: zoneone Reading ZFS config: done. zoneone console login:

Now log in as root with the password you just set, and run a few commands in the zone; - Do a ps -ef in the zone, and another one in the Global zone, and compare the output. - Also compare the output from a few other commands, particularly df -h, uptime, zonename, hostname, ifconfig, and uptime. On the ifconfig -a command output, note the zone identifier for each interface, as well as the multiple loopback identifiers.

For now I will leave you to marvel at the differences and similarities, but I will point out one process running in the global zone, namely zoneadmd. It manages the zone environment, ensures that its file systems and network interfaces are mounted and plumbed, etc. There will be an instance of it for every running non-Global zone.

Disconnecting from the zone Console requires that you enter the "disconnect" escape sequence, by default ~. (A tilde followed by a full stop) If you're already working remotely using something like SSH, use ~~. - The extra tilde character will inform SSH that it should not itself act on the disconnect command, but rather pass it on down the server. It is also possible to change your escape sequences, but I'll reserve that for a small blog article on another day. Note that disconnecting the console is NOT the same as logging out of the zones - processes like prstat will continue to run on the zone's console and you can reconnect later by again by running "zlogin -C myfirstzone"

Now that the zone is running, lets do a few interesting things with it.

First, in the Global Zone, have another look at what you see from zoneadm list -vc. Other interesting command are "prstat -Z", as well as "df -Z"

Next try out zlogin sans the -C option which would put you on the console. Use exit, logout, or ^D to disconnect.

Create a new user in the zone zlogin myfirstzone mkdir /export/home useradd -d /export/home/joe -m joe passwd joe

Use SSH to login to the zone from another machine, using the zone's IP address. Confirm where you logged in to using hostname, and zonename.

Now lets start an FTP server in the zone: zlogin myfirstzone svcadm enable ftpd

zlogin as used above runs the specified command directly in the zone. You can check the results using by ftp-ing to the zone, though you need to use a user like joe created above, which can login via the network.

Finally, lets add some disk space to our existing zone. This could have been done during the configuration process, but that would have defeated the purpose of showing how to add something to a zone.

I don't have an extra disk device, so I'll use a loop-back (lofi) device. cd /export/zones mkfile 1g fakedev_for_myfirstzone lofiadm -a `pwd`/fakedev_for_myfirstzone newfs /dev/lofi/1 /dev/rlofi/1: Unable to find Media type. Proceeding with system determined parameters. newfs: construct a new file system /dev/rlofi/1: (y/n)? y /dev/rlofi/1: 2097000 sectors in 3495 cylinders of 1 tracks, 600 sectors 1023.9MB in 219 cyl groups (16 c/g, 4.69MB/g, 2240 i/g) super-block backups (for fsck -F ufs -o b=#) at: 32, 9632, 19232, 28832, 38432, 48032, 57632, 67232, 76832, 86432, 2006432, 2016032, 2025632, 2035232, 2044832, 2054432, 2064032, 2073632, 2083232, 2092832

There are at least 3 ways in which this file system can be added into the domain. I will mount it in the global zone, then loop it into "myfirstzone" (This allws you to mount it into multiple zones should you so wish.)

mkdir mountpoint_for_myfirstzone_space1 mount /dev/lofi/1 `pwd`/mountpoint_for_myfirstzone_space1 zonecfg -z myfirstzone zonecfg:myfirstzone> add fs zonecfg:myfirstzone:fs> set type=lofs zonecfg:myfirstzone:fs> set special=/export/zones/mountpoint_for_myfirstzone_space1 zonecfg:myfirstzone:fs> set dir=/space1 zonecfg:myfirstzone:fs> end zonecfg:myfirstzone> verify zonecfg:myfirstzone> confirm zonecfg:myfirstzone> exit

Make the change take effect zoneadm -z myfirstzone reboot

Log back into the zone, and look at the output of df again. Compare this with the output from df in the global-zone ...

I'll post a follow-up article explaining some of the more advanced zone configuration topics, but hopefully you'll be impressed with how easy this all is. If you want to know why zones work, thing chroot jail combined with ppriv's (man privileges)