Solaris Live Upgrade, ZFS and Zones

I've been working on this problem for a few days and have only just solved it, so thought it might be worth sharing...

Solaris is a very powerful operating system with some great features. Zones brought Docker-like containers to Solaris back in 2005, ZFS is one of the most advanced filesystems currently available, and the Live Upgrade capability is highly underrated and is a great way to patch a server while ensure you have a back out plan.

All good stuff, but when you put Live Upgrade into a mix of Zones and ZFS, things get a bit flakey.

The issue I was having was that when I ran the "lupc -S" (Live Upgrade Preflight Check) script on my zone, I'd get the following message:

# lupc -S
This system has Patch level/IDR  of 121430-92.
Please check MOS (My Oracle Support) to verify that the latest Live Upgrade patch is installed -- lupc does not verify patch versions.

Zonepath of zone is the mountpoint of top level dataset.
This configuration is unsupported

Oracle has a document on My Oracle Support: "List of currently unsupported Live Upgrade (LU) configurations (Doc ID 1396382.1)" which lists a lot of ways in which Live Upgrade won't work(!). On checking this document for the top level dataset issue, it gives the following text:

If ZFS root pool resides on one pool (say rpool) with zone residing on toplevel dataset of a different pool (say newpool) mounted on /newpool i.e. zonepath=/newpool, the lucreate would fail.

Okay, except that's not what I've got. My zone, s1, has a zonepath set to  /zones/s1. The zpool is called "zones" and "s1" is a separate ZFS filesystem in this dataset.

What the system is actually complaining about is that the zpool is called "zones" and is mounted as "/zones". The workaround is to set the ZFS mountpoint to be something different from the pool name.

For example,  I created a new ZFS filesystem under zones called "zoneroot":

# zfs create zones/zoneroot

Then (and this is the important bit), I set the mountpoint to something else:

# zfs set mountpoint=/zoneroot zones/zoneroot

Running zfs list for this dataset shows:

zones/zoneroot                  1.80G   122G    32K  /zoneroot

Now, I can create a zone, let's call it "s2":

# zonecfg -z s2
s2: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:s2> create
zonecfg:s2> set zonepath=/zoneroot/s2
zonecfg:s2> verify
zonecfg:s2> commit
zonecfg:s2> exit

On installing this zone, a new ZFS file system is created, /zoneroot/s2.

Now, when running the "lupc -S" command, Live Upgrade doesn't complain about unconfigured configurations!

Customising gVim

While other text editors may be available, I prefer to use vi for my editing needs when running on a Unix or Linux box. I get my vi fix on Windows by running the excellent gVim. When I upgraded to a new work laptop running 64bit Windows 7, installing gVim was one of my first tasks.

There are a couple of things that need to be done to make gVim work correctly. The first is a registry change to add an "Edit with VIM" context menu item in Explorer. The following is the contents of a gvim.reg file I ran to add this functionality:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\Edit with Vim]

[HKEY_CLASSES_ROOT\*\shell\Edit with Vim\command]
@="C:\\Program Files (x86)\\Vim\\vim72\\gvim.exe \"%1\""

The second thing to do is add some customisation. gVim can use a _vimrc file (the underscore is necessary at the start) and uses the HOME variable to locate it. I setup a HOME variable that was pointing to "%USERPROFILE%" (c:\users\jr) and created a text file in %HOME% with my desired settings:

colorscheme slate
set guifont=Lucida\ Console
set columns=132
set lines=50
set nobackup
set number

This doesn't do much apart from set a nice colour scheme, font, window size, prevents backup files being created whenever a file is edited, and turns on line numbers. There are hundreds of options that power users can add to customise gVim, but it's a good start.

With the final edition of adding the gVim icon to my task bar, I now have a comfortable working environment.

IBM pSeries (AIX) to Sun StorageTek 2540 - Part 2

The saga continues...

At the end of my previous port, the SAN LUN was being successfully seen by AIX as a single device using the Cambex driver.

With the multipathing fixed, it was time to build some WPARs. Everything went smoothly until we rebooted. At which point, hdisk10 was visible but I could no longer see the logical volumes on the disk. Furthermore, I couldn't activate the volume group I'd created "wparvg", getting the message:

bash-3.2# varyonvg wparvg
0516-013 varyonvg: The volume group cannot be varied on because there are no good copies of the descriptor area.

To cut a long story short (that primarily consists of me rebooting, removing the device in smit and running cfgmgr is various combinations), the Cambex install (/usr/lpp/cbxdpf) includes some useful commands. Running the dpfinfo command showed that hdisk10 was configured in the following way:

=== /usr/lpp/cbxdpf/dpfutil listall ===
# Device Active Standby
hdisk10 cbx1 (fscsi0 0x040200,1) cbx0 (fscsi0 0x030200,1)

This means that it's using path cbx1 as its active path, with cbx0 as the failover path. Some exploring with the dpfutil command showed it supports the following options:

dpfutil []
Commands may be abbreviated:
HELP - Display this message
LISTALL - List devices and path configuration
ACTIVATE [cbxN] - Manually switch virtual disk to path [cbxN]
VARYOFFLINE [cbxN] - Mark path [cbxN] unavailable
VARYONLINE [cbxN] - Mark path [cbxN] available
MARKFORDELETE [cbxN] - Force path off even if open (may crash)
LIST_HBAS - List HBAs with DPF paths
HBA_SET_WWN [cbxN] [no|yes] - Set WWN preferred path
TARGET_SET_WWN [cbxN] [yes|no] - Set target preferred path


I tried to manually switch over the paths:

bash-3.2# ./dpfutil activate cbx0
bash-3.2# ./dpfutil listall
# Device Active Standby
hdisk10 cbx0 (fscsi0 0x030200,1) cbx1 (fscsi0 0x040200,1)

With this done, I then tried the varyonvg again:

bash-3.2# varyonvg wparvg
bash-3.2# lsvg wparvg
VOLUME GROUP: wparvg VG IDENTIFIER: 00048ada0000d3000000012865034072
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 999 (255744 megabytes)
MAX LVs: 256 FREE PPs: 979 (250624 megabytes)
LVs: 2 USED PPs: 20 (5120 megabytes)
OPEN LVs: 0 QUORUM: 2 (Enabled)
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 1024 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
bash-3.2#

Result!

Not sure what this says about the failover capabilities of the driver... It appears that when the VG is active, manually failing over the paths works okay and the VG remains active.

Fortunately this isn't a mission critical production box (it's a development compile box for porting our code from Solaris to AIX).

IBM pSeries (AIX) to Sun StorageTek 2540

We have a IBM pSeries 505 running AIX 6.1 that we use for product compilation and testing. The 505 is a 1U, entry-level POWER server with the capacity for two internal disks. In order to provision additional disk space so we can run Workload Partitions (wpars), we've added a single port Fibre Channel HBA.

The Common Array Manager (CAM) software that Sun provides to manage the 25x0 series of arrays (that form the heart of our SAN) allows the administrator to define an initiator which has a "host type" (i.e., what OS the host is running). Among the list of supported host types are the following for AIX:

• AIX
• AIX (with Veritas DMP)
• AIX (Discretionary Access Control)

The "right" option depends on the software running on the server. As I don't have Veritas DMP, and don't know what Discretionary Access Control is, I opted for AIX. Note, this doesn't appear to be documented in any of Sun's manuals...!

So with a volume setup on the array and mapped to the AIX server, it was time to see what the AIX server discovered.

In theory, booting the AIX server should find the new disks, but if you don't want to reboot, run the cfgmgr command. This appears to scan for new devices and the output can be checked by running:

bash-3.2# lsdev -Cc disk

hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive

hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive

hdisk2 Available 01-08-01 Other FC SCSI Disk Drive

hdisk3 Available 01-08-01 Other FC SCSI Disk Drive

hdisk4 Available 01-08-01 Other FC SCSI Disk Drive

hdisk5 Available 01-08-01 Other FC SCSI Disk Drive

hdisk6 Available 01-08-01 Other FC SCSI Disk Drive

hdisk7 Available 01-08-01 Other FC SCSI Disk Drive

hdisk8 Available 01-08-01 Other FC SCSI Disk Drive

hdisk9 Available 01-08-01 Other FC SCSI Disk Drive

hdisk10 Available 01-08-01 Other FC SCSI Disk Drive

hdisk11 Available 01-08-01 Other FC SCSI Disk Drive

The first two disks are internal SCSI. The disks labelled hdisk2 to hdisk9 are the management LUNs for the two 2540 arrays we have and can be ignored. The final two disks, hdisk10 and hdisk11, are two views of the LUN published from the array.

The reason there are two LUNs is because the 2540 has two controllers. Although the array is asymmetric active/passive, the server can see the LUN through both controllers at the same time.

To work around this, we need some multipath I/O software. Hunting around Sun's website found the "Dynamic Path Failover (DPF) Drivers for AIX Operating System 63 General Availability" of which there was a download for AIX 6.1. The download requires a software licence (more on that below...).

With the file downloaded to /tmp, the package could be installed by running smit and selecting the download directory (/tmp) and installing the driver. The software installs into /usr/lpp/cbxfc.

Now, a little diversion into the licensing of this driver. You need to register for a licence with Sun, providing your serial number, contract number and site id which proves you own a 2540 array. After that, the licence is free. Quite why you'd want the driver if you didn't own the array is beyond my powers of comprehension! A 30 day licence is provided in /usr/lpp/cbxfc and can be activated by copying the file "license.30day" to "license". It might be interesting to see what happens if you registered the licence a long way in the future and then changing the date back to today... (purely academic interest only, but might be useful if Sun take their time sending the licence through!).

Having installed the driver, it's another trip into smit, devices and unconfiguring the previously discovered SAN disks, hdisk2 through to hdisk11. When removing, specify the option to remove from the device database as well. After doing this, lscfg should not show the SAN disks.

With the SAN disks no longer visible to the AIX server, re-run cfgmgr and then run lscfg again. If everything is working correctly, the disks should reappear, but be labelled differently:


bash-3.2# lsdev -Cc disk
hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk3 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk4 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk5 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk6 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk7 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk8 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk9 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk10 Available 01-08-01-01 StorageTek FlexLine with DPF V4.31P

Ignoring the management LUNs (hdisk2 to hdisk9), note how there is only one SAN LUN visible, hdisk10. Running /usr/lpp/cbxdpf/dputil utility shows the multipathed disk:

# Device Active Standby
hdisk10 cbx0 (fscsi0 0x010200,1) cbx1 (fscsi0 0x030200,1)

The new disk also appears as a physical volume:

bash-3.2# lspv
hdisk0 00048ada53748080 rootvg active
hdisk1 none None
hdisk10 00048ada40cd9d27 None

The disk can now be added to a volume group and used by the AIX system.

Special thanks to @cgibbo on Twitter who spotted my cry for help when I was struggling to get the multipathing working and got in touch. Thanks for the pointers Chris!
.

Upgrading the EeePC 701 to Eeebuntu

I don't tend to use my EeePC 701 4G very much; there's not much point when you have a pretty well setup PC and network. But when it comes to going on holiday, the Eee is a must-pack luggage item.

T and I have just been away for a week in Corfu. Weather: hot. Hotel Wi-Fi: not bad and free to use (guess which is the most important criteria... :-))

It was when using the Eee on holiday that I realised how dated the default Xandros-derived distro is. Some websites even encouraged us to to upgrade to a later release of Firefox. So upon returning, I purchased a 2GB RAM upgrade (from the default 512MB), an 8GB SD card to store my files on and a 4GB USB stick with which I installed Eeebuntu.

I've never been a serious Ubuntu user (or any of its derivatives), being quite happy with OpenSUSE, so installing Eeebuntu has been interesting. Fortunately the website had some decent documentation on building an install USB key (since the Eee doesn't have a CD drive). Once that was set up, it was simply a matter of booting the Eee off the USB stick and following the prompts.

The result is a modern, GNOME-based distro that can take advantage of all the Eee functionality including the Wi-Fi and webcam. It's also a very smart-looking setup with Compiz working out of the box. I took the opportunity to add some extra software that might be useful in the future, including Wireshark and Nessus.

I'm not going to pretend that the Eee is going to be my new, main machine, or that it will be heavily used on a daily basis, but it's a very capable little computer that will be far more useful with the updated OS on it. My initial foray into Eeebuntu has also been very positive. If you're looking to get something better than the default, dated Xandros version, it's worth a look.

Configuring Kerberos on CentOS 5

Kerberos is a ticket-oriented authentication system that was originally designed for Unix networks, but was also embraced (and extended) by Microsoft in Active Directory. I've been debugging a number of issues involving the Squid proxy server on Linux using Samba to authenticate against Active Directory, and as part of this I had to get familiar with Kerberos.

It's not trivial, so I've documented my workflow here. Hopefully it will useful to others.

The test environment consists of two virtual machines running CentOS 5, imaginatively named centos01 (krbserver) and centos02 (krbclient). For the purpose of this test, centos01 is the Kerberos server and centos02 is the client.

I followed the instructions here and broadly recommend them. These are my additional notes to clarify some parts of the install.

General notes

Make sure that you use the same time source for both client and server. I used NTP to keep the two VMs in sync. The notes do state this but it's worth stressing.

Remember how many IT problems are caused by name resolution errors! Make sure you have both the server and client registered in DNS (or have entries in /etc/hosts). If using /etc/hosts both the standalone hostname and the FQDN should be added:

192.168.192.26 krbserver.local.zone krbserver
192.168.192.108 krbclient.local.zone krbserver

Note the order of the hostname and the FQDN! This is important (see further below).

Configure the server

After installing the packages using YUM, configuring the database and ACL file, adding the first principal user and starting the three services, the server should be ready to go. Confirm this with kinit and klist. Now it's time to configure the client.

Configure the client

Install the packages using YUM and then run the kadmin command and add a new principal for the client machine. It's worth noting that this should be done using the kadmin interactive interface instead of trying to put the "addprinc" parameter on the command line. This is because the -randkey option will be interpreted by kadmin on the command line as "-r andkey" and it will try and authenticate against the "andkey" realm. So for me, the command looked like:

# kadmin -p julian/admin@LOCAL.ZONE
Password for julian/admin@LOCAL.ZONE: ********
kadmin: addprinc -randkey host/krbclient.local.zone

I assume that this is rougly analogous to adding a machine to an Active Directory domain.

Once this entry, export the principal to the workstation's /etc/krb5.keytab file.

In addition to the machine principal, I also created a normal (non-admin) a local user, julian@LOCAL.ZONE. On the client, I log in as my own non-root user ("julian") and type kinit:

$ kinit
Password for julian@LOCAL.ZONE: ********

If this succeeds, you should see the "ticket granting ticket" be assigned:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 11:14:20 07/18/09 11:14:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE

Kerberos 4 ticket cache: /tmp/tkt500 klist: You have no tickets cached

This process shows that communication between the client and server using Kerberos is successful.

Configuring telnet (for testing)

On the server, I then enabled the krb5-telnet service in /etc/xinetd.d and started xinetd. On the client, I then ran:

$ /usr/kerberos/bin/telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
[ Kerberos V5 refuses authentication because telnetd: krb5_rd_req failed: Key version number for principal in key table is incorrect ]
Password:

Problem. It was asking for a password which implied the Kerberos ticket was not being passed correctly. However, when I ran klist, it showed that the ticket for the host was passed correctly:

$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 10:38:20 07/18/09 10:38:20 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 10:38:31 07/18/09 10:38:20 host/krbserver.local.zone@LOCAL.ZONE

After running strace against the telnetd process, it appeared that the telnet server was failing when trying to read /etc/krb5.keytab. But all the documentation I had read stated that this should be run on the client and not the server. So, why does the Kerberos server need a keytab file?

Answer: The Kerberos server does not require a keytab file, but the telnet server does! Although they are both running on the same VM, the telnet server is itself a client to the Kerberos server. Simple when you work it out it would have semantically been easier to understand if my telnet server been different from the Kerberos server.

So I ran the kadmin command on the server and created a keytab file using the ktadd command. I restarted the Kerberos services for good measure and cleared my client and server caches using kdestroy, restarted xinetd and tried the telnet:

[julian@krbclient bin]$ ./telnet -a krbserver
Trying 192.168.192.26...
Connected to krbserver.local.zone (192.168.192.26).
Escape character is '^]'.
[ Kerberos V5 accepts you as ``julian@LOCAL.ZONE'' ]
Last login: Fri Jul 17 10:48:10 from krbclient
[julian@krbserver ~]$

Result!

Configuring SSH

The instructions state that GSSAPIAuthentication and GSSAPIDelegateCredentials need to be enabled. I did this and restarted the SSH daemon with -ddd (debug) enabled.

The first attempt at running ssh krbserver prompted for a password, but the server debug revealed the following:

debug1: Unspecified GSS failure. Minor code may provide more information
No principal in keytab matches desired name

Okay, so this is weird. Checking the output of klist showed this:

[julian@krbclient ~]$ ssh krbserver
julian@krbserver's password:
Connection closed by 192.168.192.26
[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached


Note that krbserver@ has no realm. This turned out to be because /etc/hosts (on the client) looks like this:

192.168.192.108 krbclient krbclient.local.zone
192.168.192.26 krbserver krbserver.local.zone

Putting the hostname after the FQDN like this:

192.168.192.108 krbclient.local.zone krbclient
192.168.192.26 krbserver.local.zone krbserver

fixes the problem!

[julian@krbclient ~]$ ssh krbserver
Last login: Fri Jul 17 13:54:40 2009 from krbclient.local.zone

Klist now shows:

[julian@krbclient ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500
Default principal: julian@LOCAL.ZONE

Valid starting Expires Service principal
07/17/09 13:42:27 07/18/09 13:42:27 krbtgt/LOCAL.ZONE@LOCAL.ZONE
07/17/09 13:42:33 07/18/09 13:42:27 host/krbserver@
07/17/09 13:54:38 07/18/09 13:42:27 host/krbserver.local.zone@LOCAL.ZONE


Kerberos 4 ticket cache: /tmp/tkt500
klist: You have no tickets cached

Summary

What you see above does not include the time spent trying things out and staring blankly at the screen. Getting Kerberos up and running is not the most trivial process and while there is some decent documentation, there are also a lot of people posting questions and asking for help when it doesn't work properly. Hopefully this will shed some light on it for others.

An OpenSUSE quickie

The command line tool for patch and package management on OpenSUSE is "zypper". I've used zypper to list patch updates using:

# zypper lu

The patches can be added (updated) using:

# zypper up

Because I never got around to reading the man page, I didn't realise that both the above commands have an implicit "-t patch". I also didn't realise that "-t package" applied to the above commands can be used to display and update packages to a later version.

# zypper lu -t package
# zypper up -t package

Currently installing 83 package updates...