EMC VNXe - diving under the hood (Part 1: Intro)

Recently we took delivery of an EMC VNXe 3100. The purpose of this unit is to provide a low cost alternative to our NetApp FAS2040, primarily for backups but also as a VMware datastore for some smaller workloads. Here is is:

If you want a decent primer on the VNXe, check out Henriwithani's blog.

The VNXe is an interesting unit and this mini series will look at what the VNXe 3100 provides and how it works. In order to understand some of the thinking behind the design, we need some background.

First, some history...

(It's worth stating here at the start that I'm not an expert on CLARiiON or Celerra and the below is my understanding based on reading the documentation. If I've made errors, please correct me!)

Go back a couple of years and EMC had two product lines for the midrange: CLARiiON and Celerra.

CLARiiON was their block storage array capable of supporting Fibre Channel and iSCSI. Each controller in a CLARiiON ran an operating environment called FLARE which actually ran on top of a Windows XP Embedded kernel.

Celerra took a CLARiiON, added one or more “X-Blades” (aka “Datamovers”) and a 1U, Linux-based “Control Station”. The Datamovers were basically NAS head units and added support for NFS and CIFS/SMB. They could also do iSCSI, albeit in a somewhat more abstracted way than running native block on CLARiiON. The operating system running on the Datamovers was a UNIX-lilke derivative called DART.

More information on Celerra’s architecture here.

In January 2011, EMC announced the successor to both CLARiiON and Celerra: The VNX. At the same time, the VNXe was announced as an entry-level sibling to the VNX. The VNX appears to be a fairly straightforward upgrade to CLARiiON and Celerra: Faster, bigger, more features etc.

However, the VNXe sports a new software architecture running on an “execution environment” called CSX. From what I can tell by looking at the VNXe support logs and poking around the command line, CSX runs on a Linux kernel. Relevant parts of the FLARE and DART stacks have been ported to run on CSX and each Storage Processor (SP) (aka controller) in the VNXe runs an instance of the Linux/CSX/FLARE/DART stack.

(More information on VNXe here.)

So what you get in a VNXe is a fusion of bits of FLARE and DART, but mixed up in a new way.

The VNXe configuration

Because it is aimed at the non-storage administrator, a lot of the technical details in the VNXe are hidden. This is a bit frustrating for those of us who like to know how things work, so I’ve tried to dive under the hood a bit.

The base VNXe 3100 is a 2U unit and has 12 drive bays and up to two controllers. Each controller (Storage Processor, or “SP” in EMC language) has a dedicated management Ethernet port, and two ports for data called Eth2 and Eth3. There is an option to install a SLIC module in each controller that adds an additional 4 x 10/100/1000Mbit Ethernet ports (copper). We didn’t buy this expansion, so I can’t comment further on those.

With two controllers, the VNXe 3100 has the ability to support up to 96 drives through the addition of Disk Array Enclosures (DAEs). The DAEs connect to the base unit via 6Gbps SAS.

As an entry level system, the VNXe has some limitations in terms of disk configuration options.  Disks are sold in packs and are designed to be added in one of the following ways:

• 300GB SAS: 5 pack in RAID5 (4+1)
• 300GB SAS: 6 pack in RAID10 (3+3)
• 600GB SAS: 5 pack in RAID5 (4+1)
• 600GB SAS: 6 pack in RAID10 (3+3)
• 1TB NL-SAS: 6 pack in RAID6 (4+2)
• 2TB NL-SAS: 6 pack in RAID6 (4+2)

The version we purchased has two controllers and 12 disks: 6x300GB SAS and 6x2TB NL-SAS.

Once installed in the system, the 6x300GB disks can be either configured as “High Performance” which is RAID10 (3+3), or as “Balanced Performance/Capacity” which is RAID5 (4+1) leaving 1 disk as a hot spare. We opted for the latter.

The 6x2TB NL-SAS can only be configured as RAID6 (4+2). Fine for our purposes (backups), but worth knowing as some use cases may require different configurations.

However, it's not as simple as popping in some disks and off you go (well, it is that simple, but there's a lot going on underneath!).

To understand what the VNXe does with the disks requires another background history lesson because there is a bunch of terminology carried over from both CLARiiON and Celerra.

CLARiiON terminology

The CLARiiON works on block storage in the following way:

Physical disks are added to a CLARiiON and a RAID set is created (such as a RAID5 4+1 configuration). The result is call a "RAID Group". Traditionally, up to 16 disks could belong to a single RAID Group.

From within the RAID Group, FLARE (the CLARiiON operating environment) would create LUNs. The most basic type of LUN is the cunningly named "FLARE LUN". Some FLARE LUNs have special purposes and are used internally (for snapshots, logging etc.). These are called Private LUNs.

The problem with a FLARE LUN is that you are limited to the maximum size of the RAID Group (16 disks worth of capacity minus overhead for parity or mirroring). To overcome this, EMC invented the MetaLUN. This construct combines multiple FLARE LUNs together either through striping or concatenation.

For the sake of completeness, later releases of FLARE introduced the concept of a Storage Pool, out of which Thick LUNs (with reserved space) and Thin LUNs (with no reserved space, also known as Virtual (aka Thin) Provisioning) could be created.

FLARE provides some additional features such as compression (which moves the LUN to a pool, converts it to a Thin LUN and compresses it) and Fully Automated Storage Tiering Virtual Provisioning (FAST VP) which combines multiple disk types (such as Flash, Fibre Channel and SATA) into a single pool and dynamically moves hot data to the fast disks). Pretty clever stuff.

Regardless of the type, each LUN is assigned to a preferred owning Storage Processor (SP), allowing the storage administrator the ability to manually balance LUNs for maximum performance. In the event of a controller failure, the peer controller would take ownership. The LUNs are then mapped to be visible to hosts.

Okay, so that's CLARiiON, what about Celerra?

Celerra terminology

To the CLARiiON, a Celerra is a host to which LUNs are presented. These LUNs must be configured in a specific way to be understood by the Celerra operating system, DART.

The LUNs that CLARiiON present are seen as single disks by the Celerra (regardless of the number of underlying physical disks that comprise the LUN). These disks are sometimes referred to as "dvols". You should assume a 1:1 mapping between CLARiiON LUNs and Celerra dvols.

In addition to the disk (dvol), Celerra offers some additional I/O constructs: A "slice" is a part of a dvol, and a single disk can be partitioned into many slices. Similarly, a "stripe" is an interlace of multiple dvols or multiple slices. The "meta" (not to be confused with a CLARiiON MetaLUN) is a concatenation of multiple slices, stripes, disks or other metas.

Finally, a "volume" is created. This is a block I/O construct into which a filesystem is created. And it's these filesystems that are made visible to hosts. The default filesystem is uxfs and is used by both NFS and CIFS fileservers.

As the above illustrates, the path between physical disk and filesystem is pretty complicated. How the VNXe is derived from this messy lineage will be the subject of part 2 (coming soon).

Passing the VCP for vSphere 5

With time running out on being able to certify as a VCP5 without having to take a course, I've spent the evenings of the last few weeks revising and testing in the home lab. As with previous exams, the scoring is emphatically not percentage based and is graded between 100 and 500 with a passing mark of 300(!).

Well, I passed, with a score of 378 (higher than when I did the VCP4) which I was very pleased with, but thought I'd share a few thoughts on the process:

Despite a lot of experience with vSphere and reading up on all the topics in the exam blueprint (this is a must if you want to pass!), I still found myself having to make guesses in a few of the questions. This is because some questions seemed to be more "trivia" oriented and you would only know the answer if you had done that exact operation outside of the exam.

On the plus side, you will note that the exam blueprint doesn't require you to remember a list of configuration maximums anymore!

There are 85 questions to complete within 90 minutes and I found that I answered them all with about 15 minutes to spare, but because I had marked quite a few for review, I used all the time available.

For revision, I used the aforementioned exam blueprint which contains pointers to various VMware documents. This is a huge amount to digest, but it's worth a skim read of all these docs.

I also used Scott Lowe's Mastering vSphere 5 book which currently seems to be the de facto book on the subject, although I did make reference to some parts of Mike Laverick's VMware vSphere 4 Implementation (yes, the previous release) as I feel it gave a better explanation to some sections.

Finally, there is no substitute to hands on experience, and the home lab was fully utilised with 3 nested ESXi servers, both the Windows vCenter Server and the vSphere Appliance tested, along with the VSA, Update Manager, Distributed vSwitches etc.

Special mention must go to the following two sites which have written notes on each of the blueprint sections, both are excellent and really helped in my revision:

• Jason Langer's blog
• Mike Preston's blog

It's worth noting that as a regular user of the "Enterprise" version, getting experience with the "Enterprise Plus" features in the lab (thanks to the 60 day trial) is very important as otherwise there are many things that I would never otherwise see. Hopefully the community effort to reignite the VMTN program is successful and we can all benefit from using the full feature set in our home labs.

So that's the VCP out the way until the next big release. More certs later this year...

HP Microserver SAN in a box: Benchmarks

In my previous post, I detailed the build of my new "SAN" using an HP Microserver and the Nexentastor virtual storage appliance. Interested in knowing what the server was capable of, I installed the bonnie++ benchmarking software on the Nexenta VM and ran a number of tests to see how well it would perform as a datastore host for the VMware lab:

Each test was run four times and the average of each result taken. All testing was performed against a four disk RAIDZ parity stripe with an SSD read cache.

For the first test, the ZFS filesystem was configured sync=standard and compression=off. This resulted in the following averages:

• Block Writes: 65MB/sec
• Rewrites: 39MB/sec
• Block Reads: 173MB/sec
• Random Seeks: 451.55

For the second test, sync=standard and compression=on:

• Block Writes: 148MB/sec
• Rewrites: 107MB/sec
• Block Reads: 218MB/sec
• Random Seeks: 2036.1

As the results show, enabling compression results in a huge performance boost.

Although not recommended in situations where data integrity is important, ZFS supports the option of disabling synchronous writes. The third test was run with sync=disabled and compression=on:

• Block Writes: 164MB/sec
• Rewrites: 113MB/sec
• Block Reads: 216MB/sec
• Random Seeks: 2522.76

As expected, disabling synchronous writes improved the write performance of the server and had a corresponding knock-on effect for the rewrites. Block reads, although marginally slower than the second test, were close enough to suggest the difference was environmental.

Although running with synchronous writes disabled resulted in the highest performance, in order to get the best possible data integrity, I opted to run with sync=standard and compression=on.

While running the benchmark with compression=on, I noted in the vSphere client that both the CPU cores in the Microserver ran at nearly 100% (the Nexenta VM has 2 x vCPUs assigned). This suggests that the performance here was limited, not by the disks, but by the rather weak CPU in the Microserver.

HP Microserver: Building a SAN in a box

[Update 05/2013: I've since migrated my Nexenta install to a dedicated server. See here for the details.]

Following a recent upgrade to my home lab, my storage now looks like this:

• A unified SAN capable of providing both block (iSCSI) and file (NFS/CIFS) data.
• Eight disks:
• 2 x SSD
• 6 x SATA
• Two discrete RAID groups: 
• a 400GB (usable capacity) two disk mirror
• a 1.2TB (usable capacity) four disk parity stripe
• Both RAID groups have dedicated 20GB flash read caches.
• LUNs can be configured to support compression and/or deduplication
• Copy on Write (COW) snapshots for all filesystems and LUNs
• Support for replicating filesystems and LUNs to a second SAN

All sounds pretty funky. Must be expensive right?

Actually, the above is all achieved using a very cheap HP Microserver running VMware ESXi and the Nexenta virtual storage appliance. I've assigned the Nexenta VM 4GB RAM, but it would happily use more for it's L1 read cache.

The HP Microserver has 4 x SATA disks (2 x 1TB and 2 x 500GB) with a single 60GB SSD disk.

The Nexenta virtual machine is then assigned VMDK files. The first RAID group is a mirror: one VMDK file on SATA disks 1 and 2. The second RAID group is a RAIDZ parity stripe: one VMDK file on SATA disks 1, 2, 3 and 4. The flash read caches are 20GB VMDK files on the SSD.

The compression, deduplication, snapshot and replication features are provided by the ZFS filesystem.

This is a pictorial representation of the configuration:

And this is what it looks like physically:

Oops. No, that's the NetApp at work. But functionality-wise they are quite similar (obviously the vastly more expensive NetApp is much faster!).

This is the real physical hardware (on the far right, next to the ML110 and ML115):

Pretty small for such a setup. I've currently only got the built-in NIC in the Microserver, but will look at adding another to create a dedicated storage network.

Cisco SG200-26 review

Until recently I was using a Netgear GS108 switch for my home lab. This eight port, unmanaged switch performed well, but with the addition of a couple of HP Microservers, I ran out of free ports and needed something bigger.

Although not essential to the lab, I wanted a switch with a few more features. I initially looked at the Cisco SG200-18, the HP V1810-24G and a couple of other makes that I hadn't come across before (TP-Link and ZyXEL). The one requirement was that the new switch should be silent. The fans of a Cisco Catalyst switch would dominate the home office and was unacceptable.

I discounted the switches from TP-Link and ZyXEL because I couldn't find any decent reviews of them online. The HP V1810 was then discounted because the price hiked up to over £230. This left the Cisco SG200-18. I then noticed that the SG200-26 was only £3 more expensive at £188 (from Ebuyer), so buying the smaller switch would not have made financial sense. You can't have too many ports, right?

The first thing to say about the Cisco SG200-26 is that it is not an IOS switch. I assume it's the result of the purchase of Linksys. Having said that, the build quality is good, the switch is absolutely silent in operation but doesn't get hot (in contrast, the Netgear was hot to touch). The SG200-26 is a managed, layer 2 switch.

The SG200-26 has 24 standard 10/100/1000 ports, plus another two ports for uplinks. These can be RJ-45 10/100/1000 ports or SFP fibre ports (SFP modules not included). The form factor is standard rack-mount 1U (rack mount kit included) but also has attachable rubber feet for desktop use.

Configuration is through the web interface only (no SSH or serial interface), but does support external logging to a syslog server.

Be sure to upgrade to the latest firmware. This enabled the Cisco Discovery Protocol (CDP) which is very useful in vSphere networking for identifying which physical ports a NIC is plugged into.

In the web interface, ports can be given a description and those of us with OCD can spend a happy evening mapping this information into the switch. The port settings can also be used to state the speed and duplex setting of each port.

The SG200-26 supports up to four Link Aggregation Groups (LAGs) and can load balance based on either MAC address or IP/MAC address. Both static and dynamic (LACP) LAG groups can be configured. Up to eight ports can be assigned to a static LAG and sixteen ports to a dynamic LAG.

Multiple VLANs can be setup and managed as the switch supports 802.1q. Ports can be setup as trunk, general, access or Q-in-Q mode. VLAN pruning can be applied to trunk ports so that only specific VLANs are accessible to particular ports. The interface for this wasn't immediately obvious to me (and setting up the same in IOS initially seemed easier), but once I'd spent some time with it, the VLAN configuration was fairly straightforward. These VLAN options can be applied to either individual ports or a LAG.

In addition to these features, the SG200-26 can also be configured for QoS, there are numerous security features including 802.1X, Smartport macros to configure the port type (e.g, Printer, Desktop, Guest, Server etc.). Jumbo frames can be enabled, although this applies is a global setting that affects all ports (most switches, even expensive Cisco switches, work the same way). A "Green Ethernet" function reduces the power requirements of the switch by calculating the length of cable, and also by turning off unused ports to save energy.

As a lab switch, the SG200-26 is ideal. Personally, I would have liked to see a command line option for configuration as some tasks can be repetitive (e.g., setting up VLANs). Beyond that though, there is little to complain about. The SG200-26 is an excellent entry-level switch, with plenty of ports and a good range of options.

Some useful links:

The Cisco Small Business 200 Series Smart Switch Administration Guide

The Cisco Small Business Online Device Emulators page has a demo of the web interface for the SF300. The 300 series has additional layer 3 functionality, but you can get a good idea what the interface is like on the 200 series.

* Update 10/07/2012 * I experienced an issue where traffic between two ports (e.g., ports 1 and 2) would cause significant latency issues on other, unrelated ports. This was demonstrated by putting a ping on a host and watching the timings when there was significant network load (such as VM backups). This was resolved by upgrading the firmware to 1.1.2.0.

ISP router ARP cache problems when replacing servers

I experienced a problem today that took a while to understand so figured it was worth sharing...

Our external mail gateway was due for replacement and a new virtual machine was built, configured and tested alongside the old production server. Happy that everything was functioning as expected, the only remaining task was to disconnect the old server from the network and rename the IP address of the new server from its test IP to that of the old server. This would require no changes to DNS and total downtime would be about a minute.

The change was made and... nothing. No traffic to the new server.

Huh? I tested it from another IP on the public network and it was fine. We tried from another network and... nothing.

I changed the IP back to the test address and the server sprang into life.

After a significant amount of time brainstorming with colleagues as to what was happening, we hit upon the possible problem being an ARP cache issue on the ISP provided router. Unfortunately, we don't have administrative access to this router.

Fortunately, the ISP hadn't locked down the console port of the Cisco router and I was able to connect in and run a "show ip arp" command. Sure enough, it showed the MAC address of the old server. This meant that when packets arrived from the Internet the router was trying to forward them to the old server that was no longer on the network. If I had administrative access to the router, I would have been able to flush the ARP cache and all would have been good. But because this was a "managed" router, I wasn't able to do this. I could see the problem, I knew the solution, but couldn't fix it.

I did some research online to see what the default ARP cache timeout was: typically 4 hours.

I logged a call with the ISP which was not a particularly useful experience. The ISP is a subsidiary of Cable & Wireless, and if you've ever had the misfortune of working with that company you'll understand what I'm talking about! I was told I'd get a call back in 8 hours. Brilliant! Not.

There were a couple of other options: Pulling the Ethernet cable from the router would down the interface which I *think* will cause the ARP cache to flush. I didn't have the luxury of doing this in hours.

The final option was to try and get the new server to send a gratuitous ARP request. This is an ARP request that a server broadcasts about itself. The idea is that other devices on the network will update their ARP caches with the information.

My server however was hidden behind a Cisco ASA firewall.

As I was searching for ways to get this working, the ARP cache timed out (possibly due to the router configuration being lower than the default, although I can't see the config to confirm this) and the new server sprang into life.

At first I wasn't sure whether it was the gratuitous ARP that fixed it, but within the next hour, the ISP called and confirmed they cleared the cache. So fair play to them for getting on with it and sorting the problem.


It's been a learning experience in that even the simplest and quickest network change can have unforeseen side effects!

Slow Windows 2008 install in an ESXi VM

This is just a quick note on a problem I've just experienced (and found a fix for!).

Having just built a new Windows 2008 VM and mounting the ISO from my NFS ISO datastore, I was surprised to see that the actual install was crawling along very slowly at the "Expanding Windows files" section:

As the above link indicates, the problem was the ESX hosts NIC configuration. Set to "auto-negotiate" (which it should be on gigabit connections), the port had managed to negotiate down to 10Mb half-duplex(!). I changed this to 1000Mbit full-duplex and then back to auto-negotiate (where it stayed at 1000Mbit full-duplex).

And the performance became speedy again!