Solaris Live Upgrade, ZFS and Zones

I've been working on this problem for a few days and have only just solved it, so thought it might be worth sharing...

Solaris is a very powerful operating system with some great features. Zones brought Docker-like containers to Solaris back in 2005, ZFS is one of the most advanced filesystems currently available, and the Live Upgrade capability is highly underrated and is a great way to patch a server while ensure you have a back out plan.

All good stuff, but when you put Live Upgrade into a mix of Zones and ZFS, things get a bit flakey.

The issue I was having was that when I ran the "lupc -S" (Live Upgrade Preflight Check) script on my zone, I'd get the following message:

# lupc -S
This system has Patch level/IDR  of 121430-92.
Please check MOS (My Oracle Support) to verify that the latest Live Upgrade patch is installed -- lupc does not verify patch versions.

Zonepath of zone is the mountpoint of top level dataset.
This configuration is unsupported

Oracle has a document on My Oracle Support: "List of currently unsupported Live Upgrade (LU) configurations (Doc ID 1396382.1)" which lists a lot of ways in which Live Upgrade won't work(!). On checking this document for the top level dataset issue, it gives the following text:

If ZFS root pool resides on one pool (say rpool) with zone residing on toplevel dataset of a different pool (say newpool) mounted on /newpool i.e. zonepath=/newpool, the lucreate would fail.

Okay, except that's not what I've got. My zone, s1, has a zonepath set to  /zones/s1. The zpool is called "zones" and "s1" is a separate ZFS filesystem in this dataset.

What the system is actually complaining about is that the zpool is called "zones" and is mounted as "/zones". The workaround is to set the ZFS mountpoint to be something different from the pool name.

For example,  I created a new ZFS filesystem under zones called "zoneroot":

# zfs create zones/zoneroot

Then (and this is the important bit), I set the mountpoint to something else:

# zfs set mountpoint=/zoneroot zones/zoneroot

Running zfs list for this dataset shows:

zones/zoneroot                  1.80G   122G    32K  /zoneroot

Now, I can create a zone, let's call it "s2":

# zonecfg -z s2
s2: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:s2> create
zonecfg:s2> set zonepath=/zoneroot/s2
zonecfg:s2> verify
zonecfg:s2> commit
zonecfg:s2> exit

On installing this zone, a new ZFS file system is created, /zoneroot/s2.

Now, when running the "lupc -S" command, Live Upgrade doesn't complain about unconfigured configurations!

Sun X4100 M2 firmware upgrade

This is a very short note that others might run into...

I was trying to upgrade the firmware on a Sun X4100 M2 server to the latest release and the System BIOS upgrade was failing. I was picking the firmware image up from a network drive which may have been the problem, as copying the image to my C: drive and then installing the upgrade worked fine.

Not sure why this should be the case, but the upgrade has now worked.

Configure Solaris 10 for mail relaying

We have a number of devices on our network that can send email alerts. It makes sense to have a central server that can act as a mail relay. We have a Solaris 10 server "sol10" that comes bundled with sendmail, but this is not configured to act as a mail relay.

In order to make the Solaris server relay messages to another host involves editing the /etc/mail/sendmail.cf file and setting the value:

# "Smart" relay host (may be null)
DSmailserver.my.domain

Obviously replace "mailserver.my.domain" with the FQDN of your real mail server that you want to send email through. Restart sendmail by running:

svcadm restart /network/smtp

This setting will allow mail that originates on "sol10" to be sent out, but does not help when you want other devices on your network to use sol10 as it's relay. The answer was surprisingly easy:

Create a new file /etc/mail/relay-domains. In this file, put the networks you want sol10 to accept email from. For example, if you have devices on the 10.0.0.0/8, 172.16.0.0/16 and 192.168.20.0/24 networks and want to use sol10 as the relay, enter the following lines in /etc/mail/relay-domains:

10

172.16

192.168.20

Once done, restart sendmail again (same command as above), configure your clients to use the Solaris server as their SMTP server and check the output in /var/log/syslog while you send a test message.

Book Review: OpenSolaris Bible

The relationship between OpenSolaris and Solaris is similar to that between Fedora and Red Hat Enterprise Linux. OpenSolaris is Sun's "in development" operating system that introduces many new features that will eventually become available in a Solaris 10 update or in a future Solaris 11 release.

So it would seem sensible for Solaris system administrators to have some familiarity with OpenSolaris and while it's possible to transfer a lot of existing Solaris knowledge across, having a comprehensive book alongside can be very useful.

Enter, the OpenSolaris Bible by Solter, Jelinek and Miner; a book I received just before Christmas and have been reading through recently.

The book covers the release of OpenSolaris as of 2008 which suggests the book was based around release 2008.05 or 2008.11 (OpenSolaris releases have a YYYY.MM version number). Since that date, there has been a 2009.06 release and 2010.02 is anticipated next month. However, do not let this put you off considering this book. OpenSolaris development is fast paced, but there is an awful lot of stuff in this book to absorb that still remains relevant in newer releases.

The book is broken into six parts:

• Introduction to OpenSolaris
• Using OpenSolaris
• OpenSolaris File Systems, Networking and Security
• OpenSolaris Reliability, Availability and Serviceability
• OpenSolaris Virtualization
• Deploying and Developing on OpenSolaris

The first part is a typical introduction and covers the history of Solaris, Open Source, as well as instructions on installing OpenSolaris and a basic "crash course" on using the GNOME desktop and the Unix shell. Experienced administrators will be able to skim this section.

Part two covers using the desktop in more detail, printing and software management using the Image Packaging System (IPS). This is an essential read as IPS is a new feature in OpenSolaris and printing can sometimes be a bit tricky.

Part three provides a very comprehensive introduction to Solaris disks, pseudo filesystems such as devfs, tmpfs, lofs and swap, UFS, Solaris Volume Manager, iSCSI, quotas, backups and restores, mounting and unmounting as well as a full chapter on ZFS, before moving onto network configuration including IPMP, link aggregation, virtual LAN interfaces, network services (DNS, DHCP, FTP, NTP, Mail, HTTP etc.), routing and the IP Filter firewall. Part three of the book then finishes with a chapter on network file systems and directory services (NFS, CIFS, NIS and LDAP) and security (PAM, RBAC, SSH, auditing and Kerberos). There is a lot of good content here.

Part four details the Fault Management architecture in OpenSolaris, the Service Management Framework (SMF) introduced in Solaris 10 as well as monitoring with conventional tools and Dtrace, ending with a chapter on high-availability clustering.

Part five covers resource management (projects, tasks, caps and pools) along with a number of Sun virtualisation technologies (Zones, xVM, LDOMs and VirtualBox). The xVM section is only relevant to x64 installs and the LDOM section requires Sun UltraSPARC T-series processors, while Zones can be used one either architecture and is certainly worth a read.

The final part consists of a chapter on deploying a web stack (Apache, PHP, MySQL, Tomcat and Glassfish) and a chapter on software development (Java, C/C++, etc.). I have no strong interest in these subjects at the moment, so haven't read this section.

While I have not read the whole book yet (Having ignored most of the coverage of GNOME desktop applications as if you are familiar with Linux, there's not a lot of new stuff to learn), there are plenty of sections that have made the book worthwhile. Whether this book is suitable for you or not, depends on where you're starting from:

If you are a Windows administrator looking to develop some Solaris experience, the OpenSolaris Bible is well worth a read. The first two parts provide a gentle introduction to the Unix operating system to get you started, and subsequent chapters dive pretty deep into the capabilities of OpenSolaris.

If you are experienced with Linux but have minimal Solaris experience, the OpenSolaris Bible is highly recommended! FMA, SMF, Zones, ZFS, UFS/SVM, Clustering, Dtrace and IPS are not found in Linux, so the OpenSolaris Bible provides a single point of reference for a whole lot of new learning.

Even experienced Solaris administrators will find things to like in this book. The IPS is certainly a new feature that I assume will impact us when Solaris 11 is released, and while ZFS, Zones, FMA, SMF etc are already present in Solaris 10, the book provides a very good overview of these technologies that can otherwise only be found by attending a course or reading the online documentation.

It's probably fair to say that if you read through the whole book, put into practice the features described, and you understand them, you'll have a wider understanding than many existing Solaris system administrators.

The OpenSolaris Bible can be bought at Amazon.

Highly recommended. 9/10.

Hands on Solaris IP Multipathing

Today I had a first look at IP multipathing on Sun Solaris. What is IP multipathing? It's a feature that can be used to provide additional network resilience to a server with multiple physical network interfaces. By ensuring that at least two interfaces are on the same subnet, IP Multi Pathing (IPMP) provides continuous uptime if one of the links goes down, migrating the IP address transparently over to the other interface. IPMP operates at the IP layer (layer 3) and does not do link aggregation (although Solaris does support this, I haven't tried that yet).

To test, I booted a Solaris 10 VM on VMware ESXi. I had assigned two NICs to the VM and ensured they were connected and plumbed in.

The first step is to create a "group" for the IPMP by assigning the interfaces e1000g0 and e1000g1 together. I unimaginatively called the group "mygroup".

bash-3.00# ifconfig e1000g0 group mygroup
bash-3.00# ifconfig e1000g1 group mygroup

I then assigned an IP address to each interface and brought the interface up:

bash-3.00# ifconfig e1000g0 192.168.192.105 up
bash-3.00# ifconfig e1000g1 192.168.192.106 up

The output of ifconfig -a looks similar to normal with the addition of the groupname flag:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

With IPMP setup, I setup a continuous ping from another machine and then edited the VM in the vSphere client, disconnecting the second NIC from the network. Immediately, the following was reported in /var/adm/messages:

Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 215189 daemon.error] The link has gone down on e1000g1
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 594170 daemon.error] NIC failure detected on e1000g1 of group mygroup
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g1 to NIC e1000g0

No packet loss so far. What does ifconfig -a now show?

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g0:1: flags=1000843 mtu 1500 index 2
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
e1000g1: flags=19000802 mtu 0 index 3
      inet 0.0.0.0 netmask 0
      groupname mygroup
      ether 0:c:29:f9:5d:ee

While e1000g1 was still present, it now had a status of FAILED and was no longer UP. The in.mpathd daemon had initiated a new virtual interface, e1000g0:1, on the other interface and assigned the failed IP address.

With the test complete, I then reattached the NIC in the vSphere client and noted the following in /var/adm/messages:

Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 820239 daemon.error] The link has come up on e1000g1
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 299542 daemon.error] NIC repair detected on e1000g1 of group mygroup
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g1

This looked good, and a final check of ifconfig -a showed:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

This was so easy I don't know why I didn't do this years ago...

An hour with Solaris Live Upgrade

Last week I had the opportunity to do some work with the Live Upgrade feature of Sun Solaris. I had been vaguely aware of it's capabilities and we had been including a provision for it on our customer server builds, but it was only yesterday that I sat down and tried to do an upgrade to the latest Solaris 10 update 8.

Live Upgrade is a capability where the system administrator can upgrade to a new version of Solaris while the existing operating system in running. The only downtime experienced is a scheduled reboot at the end of the process to initialise the new version. If something goes wrong, the original version of the OS is still available for booting.

The way it works is based around the concept of a "boot environment". The default environment is the operating system you're running at the moment. On our servers, we have been creating a 20GB root (/) filesystem. A second 20GB slice is also created, but not used (nominally mounted as /lu so we remember it's there).

The first step in running the live upgrade is to create a new boot environment. Firstly, the /lu partition was unmounted, and commented out in /etc/vfstab. Once that was done, the new boot environment was setup:

# lucreate -n osupgrade -m /:/dev/md/dsk/d30:ufs

Okay, so here we are creating a new environment called "osupgrade" and saying that the root ("/") filesystem will be installed on the device /dev/md/dsk/d30 and that the filesystem type will be UFS (it can also do ZFS but we didn't have the correct setup on my test system). This bit takes a while, depending on how much you have on your root filesystem.

For those unfamiliar with the /dev/md/ part, this is a Solaris Volume Manager (SVM) metadevice. In reality, "d30" is a mirror that contains two submirrors (probably called d31 and d32). These submirrors are comprised of one of more disk slices. In other words, in the above command, the new boot environment will be installed onto a new mirrored disk.

At the end of the lucreate command, you can actually look at the new, mounted boot environment and see that it's basically a copy of your existing root filesystem. The next step is to upgrade it. To do this, I mounted the install location of our Jumpstart server over NFS and initiated the live upgrade:

# luupgrade -u -n osupgrade -s /mnt/install_sol10_u8_sparc/

This bit takes a while (a bit like installing Solaris...) but basically upgrades the named boot environment using the media specified. At the end of this, all that needs to be done is for the boot environment to be activated:

# luactivate osupgrade

Before initialising the new environment, it's worth noting down your existing, working environment. For me, this was the root filesystem located on /dev/md/dsk/d10. Find out the underlying slices used by d10 (c2t0d0s0 and c2d1d0s0 in my case) Once done, reboot the server and the new boot environment should be loaded.

Now the coolness of this should be immediately apparent! Previously, operating system upgrades would require a backup of the system to tape (always a good idea!), followed by scheduled downtime as the system was upgraded "offline". This also meant a visit to a customer site, typically at a weekend.

Combined with the use of an ILOM interface (for network access to the console), it now becomes perfectly possible to upgrade a Solaris server during the day, while users are on the system. All that it required now is an out-of-hours reboot of the server to initialise the new release.

If there are problems with the upgrade, it's possible to rollback by setting the old boot environment to active. To do this, boot off cdrom or the network (I did the network), by typing the following at the PROM:

ok boot net -s

[wait for the OS to boot)

# mount -Fufs /dev/dsk/c2t0d0s0 /mnt
# /mnt/sbin/luactivate

Exit single user mode and reboot.

Obviously this is only scratching the surface of what Live Upgrade can do. It's possible to merge and split filesystems, detach and build disk mirrors, and much more. The use of Live Upgrade is also greater than the occasional update; it's perfectly possible to use Live Upgrade to apply system patches, with a very easy rollback capability.

Definitely a technology that needs to be investigated more fully...

Oracle to buy Sun

This blog is typically about stuff I'm doing, but having invested 10 years in working with Sun kit, I think this deserves a comment. I'm not particularly qualified to make an informed contribution to the Oracle buyout discussion, but since that doesn't stop most Slashdot commentators, here are my observations:

1) It could have been a lot worse. If IBM had bought Oracle, they would have two directly competing CPU architectures (POWER and SPARC), two directly competing Unix implementations (AIX and Solaris), directly competing storage, Java development tools, application servers, databases etc. The long term outcome would have been that some of these technologies would have been sunset and development consolidated.

2) It could have been much, much worse: HP could have bought Sun. See DEC/Digital and the Compaq merger to see how that would have played out.

3) Sun seems to get a bad rap from the Slashdot crowd, but the number of people who are dismayed that their favourite Sun product is under threat highlights the significance that Sun has in the market. This is especially telling when you see that the object of dismay is one of many different products: Solaris, Java, OpenOffice, Virtualbox, MySQL...

4) Oracle have now migrated from being a software solutions house to a total solutions provider. Although there is talk that Oracle will sell off SPARC to Fujitsu, or port everything to x86, the reality is that SPARC is a very lucrative platform for Sun and will be a revenue generator for Oracle. In fact, Oracle now own a very good portfolio.

5) In the last 12-18 months, Sun has "got" Open Source. I'm not sure Oracle has. Hopefully the Sun culture will impact Oracle (remember how NeXT "absorbed" Apple after being bought?).

So although I'm no expert in the area of business, the future for Sun might be okay after all. Time will tell...

Sun Certified System Administrator

Today I took the Sun Certified System Administrator for the Solaris 10 Operating System Upgrade Exam (snappy title huh?). I passed - but only just.

To aid me in my revision, I used the "Sun Certified System Administrator for Solaris 10 Study Guide" by Paul Sanghera published by Osborne, and "Exam Prep Solaris 10 System Administration" by Bill Calkins published by Que. This is on top of using Solaris on a daily basis for the last eight years.

After revising from both these books and taking the test questions, I was consistently getting over 80% from both books. Considering the passmark is 60% for the upgrade, it looked pretty good.

So imagine my surprise when I sat down in the exam and was confronted by a number of questions that are not covered in either book! The confidentiality agreement prevents me from detailing too much about the content, but I was also surprised to find that the questions were based on a small subset of topics.

This is apparently because Sun have upgraded the exams earlier this year and the books are not yet updated. Calkins has a update on his website detailing the changes (which I've only just found!).

So beware if you want to revise this certification! It's not a walk in the park and make sure you know your Flash from your Grub...!

Roundcube WebMail installation

Having got the Courier IMAP server up and with all my mail migrated onto the dedicated mail server zone, I decided to put a webmail interface on the front.

Having previously played with a few - Squirrelmail, Horde and Roundcube, I concluded that Roundcube was the one I wanted to use. The interface is very clean and it's simple.

I used Blastwave to setup the required components - Apache, PHP, MySQL, edited the httpd.conf to include PHP support, configured the database and we were off.

The final step was to configure sendmail so that I could use Roundcube to send emails as well. This proved more difficult than it should as sendmail was failing to resolve post.demon.co.uk (my upstream mail relay). This was fixed by specifying Demon's DNS servers in /etc/resolv.conf.

So now I have access to my email via Thunderbird (on Linux), Mail.app (on the Mac) and Roundcube (web browser). At some point I may investigate a commandline MUA for when I SSH into the network, but this will do for now.

Fixes to Solaris Express on VMware Server

Having installed the VMware Tools into my newly created Solaris Express VM, I discovered that the X screen resolution was absolutely massive (something like 2000x1000 pixels). The solution was to edit /etc/X11/xorg.conf from a command line login and remove the unwanted Modeline entries. As I wanted to run my VM in 1024x768, I removed all other lines, and then exited the shell. When the desktop manager reloaded, the screen was at the desired resolution.

The second problem was that installing the VMware Tools changed the network device from pcn0 to vmxnet0, but the /etc/hostname.pcn0 file remained and a new file was not created. I renamed this to /etc/hostname.vmxnet0 and restarted networking (rebooting will also work!).

Sendmail complains if it does not have a fully qualified domain name. The easiest fix it to edit /etc/hosts and add on a domain name alongside your host entry.

These are fairly minor changes than need to be made and the installation is looking pretty decent so far.

ZFS and RAID levels

As a builder of Ingres systems, we tend to use RAID 1+0 (aka RAID10) for most of our systems. RAID5 is not recommended for databases due to the read-parity-write that is required for every write operation. Furthermore, the ability of a mirrored system to read two separate data blocks in parallel can give RAID1 a significant read performance advantage.

With the arrival of ZFS in Solaris 10, I've been reading up a little to see if this accepted wisdom changes with RAIDZ. A very interesting article (http://blogs.sun.com/roch/entry/when_to_and_not_

t o) basically explains that the read potential for a RAIDZ filesystem can be significantly less than that of a RAID10. However, for non-data filesystems (such as the checkpoint backup area), RAIDZ can provide some advantages. In these areas, disk I/O is less critical (the Ingres archiver writing data to the journals is performed asynchronously to a users query execution) and sequential write operations are handled better than random I/O, making it ideal for checkpoints and journals, but less good for data areas.

Will now be putting this theory into practice...