Passing the VCP for vSphere 4

Tonight I took and passed the VCP410 exam, upgrading my VCP for VI3 to the latest release. The scoring is between 100 and 500, with 300 being the pass mark. I got 338 which wasn't great; I actually found the exam pretty tough (being at 6pm on a very warm day probably didn't help either!).

As with my CCNA post, I thought it might be useful to share some of the resources I used to study.

I used both Scott Lowe's Mastering VMware vSphere 4 and Mike Laverick's VMware vSphere 4 Implementation.  Both were very good at explaining the underlying technologies, but both had sections that were out of date. Always compare with the official VMware docs!

I also used the following sites:

• Simon Long's blog and (practice exam)
• vReference
• B3RG

The essential VMware exam blueprint and online documentation is a must read.

I also set up a home lab using virtualised ESX servers on my ML115 using the 60 day trial licences. Getting hands on is essential and having an environment where breaking things is not a problem makes revision much easier.

In addition to this, I work with VMware nearly every day (not every component and feature, but I get regular, hands on experience).

The NDA prevents me from talking about questions in the exam, but I will say this: I thought the mock exams were much easier than the real thing. I was getting > 90% in the mocks, so was slightly disappointed to get such as low pass score. Having said that, I'm extremely relieved that I don't have to go back to more revision!

Onto the next thing now...

Exporting and importing SharePoint sites

A number of our users have SharePoint (WSS 3.0) sites hosted in another office and wanted to move the contents down to our local WSS 3.0 install. This was not as straightforward as you might imagine. We hit a number of gotchas and had to provide workarounds that are documented here so that others can benefit from our experiences.

Running out of disk space on the C: drive

When running an export using stsadm, we kept filling up the C: drive despite exporting to a separate drive. The reason for this is that SharePoint writes temporary files to the location defined by the %TMP% variable. This defaults to the C: drive!

To fix, open a command prompt and type:

set TMP=E:\Tmp

(replace E:\Tmp with the drive and folder you want to use for your temporary storage). Then run the stsadm export and it should work!


Commands to export and import a named site

The command we used to export the site was:

stsadm -o export -url http://old-sharepointserver/hostedsites/development -filename e:\development.cab -includeusersecurity -versions 4 -overwrite

The above command will export the site called "development" referenced at http://old-sharepointserver/hostedsites/development to a file called development.cab. The security information will be included in the export as will all versions of documents.

To import, the following command was used on the new server:

stsadm -o import -url http://new-sharepointserver/development -filename development.cab -includeusersecurity

Note that we are importing the site "development" into the top level and not as a subsite beneath hostedsites. If the name of the site is omitted, the top level site is overwritten!

The gotchas

When running the import, we received the following message:

"The file cannot be imported because its parent web <site path> does not exist"

This error is not helpful and for us the problem was permission related. We had used users (albeit domain admin accounts) to export and import the data that were different from the site collection administrators. To fix we had to do the following:

Make sure the site collection administrator is the same on both the source and destination servers.

When running the export and import, make sure you are running the stsadm commands as the site collection administrator. This ensures the permissions are aligned and the import should work.

stsadm Import error: The 'ASPXPageIndexMode' attribute is not declared

Not sure what the cause of this error is, but we found a fix online:

To get round it I edited C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\XML\DeploymentManifest.xsd on the destination server:

under section

   <!-- SPWeb definition -->

I added the following.


<xs:attribute name="ASPXPageIndexMode" type="xs:string" use="optional"></xs:attribute>
<xs:attribute name="NoCrawl" type="xs:boolean" use="optional"></xs:attribute>
<xs:attribute name="CacheAllSchema" type="xs:boolean" use="optional"> </xs:attribute>
<xs:attribute name="AllowAutomaticASPXPageIndexing" type="xs:boolean" use="optional"></xs:attribute>

With these gotchas overcome, we were able to successfully import the new site.

IBM pSeries (AIX) to Sun StorageTek 2540 - Part 2

The saga continues...

At the end of my previous port, the SAN LUN was being successfully seen by AIX as a single device using the Cambex driver.

With the multipathing fixed, it was time to build some WPARs. Everything went smoothly until we rebooted. At which point, hdisk10 was visible but I could no longer see the logical volumes on the disk. Furthermore, I couldn't activate the volume group I'd created "wparvg", getting the message:

bash-3.2# varyonvg wparvg
0516-013 varyonvg: The volume group cannot be varied on because there are no good copies of the descriptor area.

To cut a long story short (that primarily consists of me rebooting, removing the device in smit and running cfgmgr is various combinations), the Cambex install (/usr/lpp/cbxdpf) includes some useful commands. Running the dpfinfo command showed that hdisk10 was configured in the following way:

=== /usr/lpp/cbxdpf/dpfutil listall ===
# Device Active Standby
hdisk10 cbx1 (fscsi0 0x040200,1) cbx0 (fscsi0 0x030200,1)

This means that it's using path cbx1 as its active path, with cbx0 as the failover path. Some exploring with the dpfutil command showed it supports the following options:

dpfutil []
Commands may be abbreviated:
HELP - Display this message
LISTALL - List devices and path configuration
ACTIVATE [cbxN] - Manually switch virtual disk to path [cbxN]
VARYOFFLINE [cbxN] - Mark path [cbxN] unavailable
VARYONLINE [cbxN] - Mark path [cbxN] available
MARKFORDELETE [cbxN] - Force path off even if open (may crash)
LIST_HBAS - List HBAs with DPF paths
HBA_SET_WWN [cbxN] [no|yes] - Set WWN preferred path
TARGET_SET_WWN [cbxN] [yes|no] - Set target preferred path


I tried to manually switch over the paths:

bash-3.2# ./dpfutil activate cbx0
bash-3.2# ./dpfutil listall
# Device Active Standby
hdisk10 cbx0 (fscsi0 0x030200,1) cbx1 (fscsi0 0x040200,1)

With this done, I then tried the varyonvg again:

bash-3.2# varyonvg wparvg
bash-3.2# lsvg wparvg
VOLUME GROUP: wparvg VG IDENTIFIER: 00048ada0000d3000000012865034072
VG STATE: active PP SIZE: 256 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 999 (255744 megabytes)
MAX LVs: 256 FREE PPs: 979 (250624 megabytes)
LVs: 2 USED PPs: 20 (5120 megabytes)
OPEN LVs: 0 QUORUM: 2 (Enabled)
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per VG: 32512
MAX PPs per PV: 1016 MAX PVs: 32
LTG size (Dynamic): 1024 kilobyte(s) AUTO SYNC: no
HOT SPARE: no BB POLICY: relocatable
bash-3.2#

Result!

Not sure what this says about the failover capabilities of the driver... It appears that when the VG is active, manually failing over the paths works okay and the VG remains active.

Fortunately this isn't a mission critical production box (it's a development compile box for porting our code from Solaris to AIX).

IBM pSeries (AIX) to Sun StorageTek 2540

We have a IBM pSeries 505 running AIX 6.1 that we use for product compilation and testing. The 505 is a 1U, entry-level POWER server with the capacity for two internal disks. In order to provision additional disk space so we can run Workload Partitions (wpars), we've added a single port Fibre Channel HBA.

The Common Array Manager (CAM) software that Sun provides to manage the 25x0 series of arrays (that form the heart of our SAN) allows the administrator to define an initiator which has a "host type" (i.e., what OS the host is running). Among the list of supported host types are the following for AIX:

• AIX
• AIX (with Veritas DMP)
• AIX (Discretionary Access Control)

The "right" option depends on the software running on the server. As I don't have Veritas DMP, and don't know what Discretionary Access Control is, I opted for AIX. Note, this doesn't appear to be documented in any of Sun's manuals...!

So with a volume setup on the array and mapped to the AIX server, it was time to see what the AIX server discovered.

In theory, booting the AIX server should find the new disks, but if you don't want to reboot, run the cfgmgr command. This appears to scan for new devices and the output can be checked by running:

bash-3.2# lsdev -Cc disk

hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive

hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive

hdisk2 Available 01-08-01 Other FC SCSI Disk Drive

hdisk3 Available 01-08-01 Other FC SCSI Disk Drive

hdisk4 Available 01-08-01 Other FC SCSI Disk Drive

hdisk5 Available 01-08-01 Other FC SCSI Disk Drive

hdisk6 Available 01-08-01 Other FC SCSI Disk Drive

hdisk7 Available 01-08-01 Other FC SCSI Disk Drive

hdisk8 Available 01-08-01 Other FC SCSI Disk Drive

hdisk9 Available 01-08-01 Other FC SCSI Disk Drive

hdisk10 Available 01-08-01 Other FC SCSI Disk Drive

hdisk11 Available 01-08-01 Other FC SCSI Disk Drive

The first two disks are internal SCSI. The disks labelled hdisk2 to hdisk9 are the management LUNs for the two 2540 arrays we have and can be ignored. The final two disks, hdisk10 and hdisk11, are two views of the LUN published from the array.

The reason there are two LUNs is because the 2540 has two controllers. Although the array is asymmetric active/passive, the server can see the LUN through both controllers at the same time.

To work around this, we need some multipath I/O software. Hunting around Sun's website found the "Dynamic Path Failover (DPF) Drivers for AIX Operating System 63 General Availability" of which there was a download for AIX 6.1. The download requires a software licence (more on that below...).

With the file downloaded to /tmp, the package could be installed by running smit and selecting the download directory (/tmp) and installing the driver. The software installs into /usr/lpp/cbxfc.

Now, a little diversion into the licensing of this driver. You need to register for a licence with Sun, providing your serial number, contract number and site id which proves you own a 2540 array. After that, the licence is free. Quite why you'd want the driver if you didn't own the array is beyond my powers of comprehension! A 30 day licence is provided in /usr/lpp/cbxfc and can be activated by copying the file "license.30day" to "license". It might be interesting to see what happens if you registered the licence a long way in the future and then changing the date back to today... (purely academic interest only, but might be useful if Sun take their time sending the licence through!).

Having installed the driver, it's another trip into smit, devices and unconfiguring the previously discovered SAN disks, hdisk2 through to hdisk11. When removing, specify the option to remove from the device database as well. After doing this, lscfg should not show the SAN disks.

With the SAN disks no longer visible to the AIX server, re-run cfgmgr and then run lscfg again. If everything is working correctly, the disks should reappear, but be labelled differently:


bash-3.2# lsdev -Cc disk
hdisk0 Available 06-08-01-5,0 16 Bit LVD SCSI Disk Drive
hdisk1 Available 06-08-01-8,0 16 Bit LVD SCSI Disk Drive
hdisk2 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk3 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk4 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk5 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk6 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk7 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk8 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk9 Defined 01-08-01 Sun StorageTek Universal Xport
hdisk10 Available 01-08-01-01 StorageTek FlexLine with DPF V4.31P

Ignoring the management LUNs (hdisk2 to hdisk9), note how there is only one SAN LUN visible, hdisk10. Running /usr/lpp/cbxdpf/dputil utility shows the multipathed disk:

# Device Active Standby
hdisk10 cbx0 (fscsi0 0x010200,1) cbx1 (fscsi0 0x030200,1)

The new disk also appears as a physical volume:

bash-3.2# lspv
hdisk0 00048ada53748080 rootvg active
hdisk1 none None
hdisk10 00048ada40cd9d27 None

The disk can now be added to a volume group and used by the AIX system.

Special thanks to @cgibbo on Twitter who spotted my cry for help when I was struggling to get the multipathing working and got in touch. Thanks for the pointers Chris!
.

OpenSolaris: Very slow boot times

Today I lost power to my servers due to a power outage. The UPS wasn't up to coping and almost instantly died (I was running five computers on the one, small UPS...).

Booting the OpenSolaris server back up reminded me of the painfully slow boot times that can occur. We're talking *hours* to get the server up.

The reason for this is due to the number of ZFS snapshots on a system. Here's the experiment:

I booted off the OpenSolaris 2009.06 CD and ran the format command. This displayed the disks on the system. I then imported the zpools into the running installation:

# zpool import rpool -f
# zpool import datapool -f

The -f is required because the system thinks the zpools have been assigned to another server (useful if the zpool is on a SAN LUN). The first command was relatively quick, the second was much, much slower.

Running prstat revealed that devfsadm was consuming an entire CPU. The purpose of devfsadm is to dynamically add and remove devices on the system. It was stating each of the snapshots in the datapool and creating entries in /dev. After running for a few hours, it had created over 4000(!) devices in /dev/zvol/dsk/datapool and /dev/zvol/rdsk/datapool.

The number of snapshots is thanks to the automatic snapshot service which takes frequent snapshots of the filesystems in the pool. This list is not automatically cleared down, so can grow huge. Not a problem usually because the uptime of OpenSolaris is fantastic, but is a real pain when you need the server to boot.

So, in order to keep your OpenSolaris boot times down, keep an eye on the number of snapshots on your system.

Book Review: OpenSolaris Bible

The relationship between OpenSolaris and Solaris is similar to that between Fedora and Red Hat Enterprise Linux. OpenSolaris is Sun's "in development" operating system that introduces many new features that will eventually become available in a Solaris 10 update or in a future Solaris 11 release.

So it would seem sensible for Solaris system administrators to have some familiarity with OpenSolaris and while it's possible to transfer a lot of existing Solaris knowledge across, having a comprehensive book alongside can be very useful.

Enter, the OpenSolaris Bible by Solter, Jelinek and Miner; a book I received just before Christmas and have been reading through recently.

The book covers the release of OpenSolaris as of 2008 which suggests the book was based around release 2008.05 or 2008.11 (OpenSolaris releases have a YYYY.MM version number). Since that date, there has been a 2009.06 release and 2010.02 is anticipated next month. However, do not let this put you off considering this book. OpenSolaris development is fast paced, but there is an awful lot of stuff in this book to absorb that still remains relevant in newer releases.

The book is broken into six parts:

• Introduction to OpenSolaris
• Using OpenSolaris
• OpenSolaris File Systems, Networking and Security
• OpenSolaris Reliability, Availability and Serviceability
• OpenSolaris Virtualization
• Deploying and Developing on OpenSolaris

The first part is a typical introduction and covers the history of Solaris, Open Source, as well as instructions on installing OpenSolaris and a basic "crash course" on using the GNOME desktop and the Unix shell. Experienced administrators will be able to skim this section.

Part two covers using the desktop in more detail, printing and software management using the Image Packaging System (IPS). This is an essential read as IPS is a new feature in OpenSolaris and printing can sometimes be a bit tricky.

Part three provides a very comprehensive introduction to Solaris disks, pseudo filesystems such as devfs, tmpfs, lofs and swap, UFS, Solaris Volume Manager, iSCSI, quotas, backups and restores, mounting and unmounting as well as a full chapter on ZFS, before moving onto network configuration including IPMP, link aggregation, virtual LAN interfaces, network services (DNS, DHCP, FTP, NTP, Mail, HTTP etc.), routing and the IP Filter firewall. Part three of the book then finishes with a chapter on network file systems and directory services (NFS, CIFS, NIS and LDAP) and security (PAM, RBAC, SSH, auditing and Kerberos). There is a lot of good content here.

Part four details the Fault Management architecture in OpenSolaris, the Service Management Framework (SMF) introduced in Solaris 10 as well as monitoring with conventional tools and Dtrace, ending with a chapter on high-availability clustering.

Part five covers resource management (projects, tasks, caps and pools) along with a number of Sun virtualisation technologies (Zones, xVM, LDOMs and VirtualBox). The xVM section is only relevant to x64 installs and the LDOM section requires Sun UltraSPARC T-series processors, while Zones can be used one either architecture and is certainly worth a read.

The final part consists of a chapter on deploying a web stack (Apache, PHP, MySQL, Tomcat and Glassfish) and a chapter on software development (Java, C/C++, etc.). I have no strong interest in these subjects at the moment, so haven't read this section.

While I have not read the whole book yet (Having ignored most of the coverage of GNOME desktop applications as if you are familiar with Linux, there's not a lot of new stuff to learn), there are plenty of sections that have made the book worthwhile. Whether this book is suitable for you or not, depends on where you're starting from:

If you are a Windows administrator looking to develop some Solaris experience, the OpenSolaris Bible is well worth a read. The first two parts provide a gentle introduction to the Unix operating system to get you started, and subsequent chapters dive pretty deep into the capabilities of OpenSolaris.

If you are experienced with Linux but have minimal Solaris experience, the OpenSolaris Bible is highly recommended! FMA, SMF, Zones, ZFS, UFS/SVM, Clustering, Dtrace and IPS are not found in Linux, so the OpenSolaris Bible provides a single point of reference for a whole lot of new learning.

Even experienced Solaris administrators will find things to like in this book. The IPS is certainly a new feature that I assume will impact us when Solaris 11 is released, and while ZFS, Zones, FMA, SMF etc are already present in Solaris 10, the book provides a very good overview of these technologies that can otherwise only be found by attending a course or reading the online documentation.

It's probably fair to say that if you read through the whole book, put into practice the features described, and you understand them, you'll have a wider understanding than many existing Solaris system administrators.

The OpenSolaris Bible can be bought at Amazon.

Highly recommended. 9/10.

Hands on Solaris IP Multipathing

Today I had a first look at IP multipathing on Sun Solaris. What is IP multipathing? It's a feature that can be used to provide additional network resilience to a server with multiple physical network interfaces. By ensuring that at least two interfaces are on the same subnet, IP Multi Pathing (IPMP) provides continuous uptime if one of the links goes down, migrating the IP address transparently over to the other interface. IPMP operates at the IP layer (layer 3) and does not do link aggregation (although Solaris does support this, I haven't tried that yet).

To test, I booted a Solaris 10 VM on VMware ESXi. I had assigned two NICs to the VM and ensured they were connected and plumbed in.

The first step is to create a "group" for the IPMP by assigning the interfaces e1000g0 and e1000g1 together. I unimaginatively called the group "mygroup".

bash-3.00# ifconfig e1000g0 group mygroup
bash-3.00# ifconfig e1000g1 group mygroup

I then assigned an IP address to each interface and brought the interface up:

bash-3.00# ifconfig e1000g0 192.168.192.105 up
bash-3.00# ifconfig e1000g1 192.168.192.106 up

The output of ifconfig -a looks similar to normal with the addition of the groupname flag:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

With IPMP setup, I setup a continuous ping from another machine and then edited the VM in the vSphere client, disconnecting the second NIC from the network. Immediately, the following was reported in /var/adm/messages:

Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 215189 daemon.error] The link has gone down on e1000g1
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 594170 daemon.error] NIC failure detected on e1000g1 of group mygroup
Jan  7 20:41:08 solaris10 in.mpathd[1208]: [ID 832587 daemon.error] Successfully failed over from NIC e1000g1 to NIC e1000g0

No packet loss so far. What does ifconfig -a now show?

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g0:1: flags=1000843 mtu 1500 index 2
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
e1000g1: flags=19000802 mtu 0 index 3
      inet 0.0.0.0 netmask 0
      groupname mygroup
      ether 0:c:29:f9:5d:ee

While e1000g1 was still present, it now had a status of FAILED and was no longer UP. The in.mpathd daemon had initiated a new virtual interface, e1000g0:1, on the other interface and assigned the failed IP address.

With the test complete, I then reattached the NIC in the vSphere client and noted the following in /var/adm/messages:

Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 820239 daemon.error] The link has come up on e1000g1
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 299542 daemon.error] NIC repair detected on e1000g1 of group mygroup
Jan  7 20:41:56 solaris10 in.mpathd[1208]: [ID 620804 daemon.error] Successfully failed back to NIC e1000g1

This looked good, and a final check of ifconfig -a showed:

bash-3.00# ifconfig -a
lo0: flags=2001000849 mtu 8232 index 1
      inet 127.0.0.1 netmask ff000000
e1000g0: flags=1000843 mtu 1500 index 2
      inet 192.168.192.105 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:e4
e1000g1: flags=1000843 mtu 1500 index 3
      inet 192.168.192.106 netmask ffffff00 broadcast 192.168.192.255
      groupname mygroup
      ether 0:c:29:f9:5d:ee

This was so easy I don't know why I didn't do this years ago...