How a disposable email address shows who is sharing your details...

I bought my iPhone 3G from an O2 store on the day of its release. A couple of years later I upgraded it to an iPhone 4, again from the same O2 store.

Over the last couple of weeks, I've had a number of missed calls to my mobile. No voicemail, just a missed call from a number I didn't recognise. I entered the number into Google and it appears to be associated with the Carphone Warehouse. The search results revealed a lot of people complaining that Carphone Warehouse were cold calling their mobiles to try and sell an upgrade.

Annoying, but since I'd missed the calls, no big deal.

Then, today, I received an email from Carphone Warehouse telling me I was eligible for an upgrade. Now, I'm not a customer of the Carphone Warehouse, so how did they know I was eligible?

The answer is in the email...

I use Yahoo mail for non-important stuff because it has disposable addresses. Basically you create a new address such as "mymail-" and you can then append a word to it depending on who you are dealing with. In this instance, the email was addressed to "mymail-o2@yahoo.co.uk". This address (and I've modified the first bit as it's not really "mymail-") was used when I signed up with O2.

I don't use it for anything else.

So if Carphone Warehouse are sending me emails to this address, it means that either a) I've given Carphone Warehouse my details using my O2 address (I haven't), or b) that O2 have passed/sold my details to a third party.

To be fair, it's probably hidden in the small print, but it's not cool. Who here likes receiving cold calls and junk email?

No, didn't think so.


So I complained on Twitter:

And got a reply, which started a conversation:

I'm not sure why me being a PAYG or Pay Monthly customer matters, or why recently upgrading would make a difference. I'm going to give the O2 Twitter operator the benefit of the doubt here. It's possible he/she doesn't know that the data is passed on.

But the reality is that somehow Carphone Warehouse have an address that should only exist in O2's customer database.

The last message I received today from O2 was:

That I will. And update this post.

Of course, if the data hasn't been intentionally passed to a third party, then there is always the possibility that O2 have experienced a data breach... (I don't honestly expect this to be the case).

Perhaps someone from O2 would like to clarify how Carphone Warehouse know the private address that only O2 and I know, unless the data is passed on?

Remember, O2, actions that result in your customers getting cold called and spammed is BAD.

And Carphone Warehouse? No. Not a chance.

*** UPDATE 19th December 2012 ***

Carphone Warehouse continue to spam me:

O2: Please make this stop. I have sent a STOP message and forwarded the text to the O2 spam reporting number.

VMworld Europe 2012: Wrap up and the next steps...

Having blogged about VMworld Europe 2012 as it happened, I felt it was important to summarise the week and highlight the key points that will shape the next 12 months:

1. Get the basics right

A solid infrastructure foundation is essential. As vSphere deployments grow, both in terms of number of VMs and complexity of applications provisioned, the need to ensure best practices for storage, networks and vSphere configurations is a pre-requisite of any new projects. This can be accomplished through the following of vendor white papers and policy management tools such as Host Profiles.

However, a pragmatic approach needs to be taken to balance any performance gains against operational complexity. For example, while it may be possible to tweak the round robin path selection parameter, or implement jumbo frames on storage switches, the additional complexity can introduce additional management problems later on. So, while tuning the number of IOPS per path may result in a performance improvement for a small number of VMs, the benefit may be negligible when dealing with a large number of hosts, datastores and VMs. Similarly, jumbo frames may provide a minor throughput improvement, but if a new switch is added later, the system administration team must remember to apply the same settings to the new switch or else experience frame fragmentation.

In other words, keep it simple.

2. Learn to automate

VMware vCenter Orchestrator is moving from being a peripheral product to a core part of a complex cloud infrastructure. Therefore, I'd consider it a must-have skill to acquire in the coming year. The knock on requirement is that vSphere admins will need to have a basic understanding of programming languages and development methodologies.

What about other scripting approaches such as PowerCLI? In many ways, Orchestrator and PowerCLI are complimentary. PowerCLI scripts can be used to make repetitive individual tasks easier to perform, while Orchestrator is more about enabling workflow (of larger tasks). It's probable that PowerCLI (or other PowerShell) scripts will be called as part of an Orchestrator workflow.

Bottom line: Learn both.

3. Virtualise more

VMware's "Software Defined Data Center" (SDDC) strategy extends the existing virtual infrastructure and introduces the automation and orchestration of edge devices. If implemented well, this means that provisioning new services should be faster and less error prone. The opportunity to upgrade Enterprise Plus to vCloud Suite Standard enables the rollout of vCNS and vCloud Director, thereby forming the first step in moving to this new "agile" data centre, and also provides the foundations to build a private cloud on top of vSphere.

In order to ensure that a private cloud implementation is "done right", the VMware vCloud Architecture Toolkit will be followed. This will dictate some changes to the existing infrastructure, further extending the use of a dedicated management cluster and separate production clusters.

There is one challenge to consider: The added complexity that comes with the additional functionality makes the learning curve significantly steeper. Today, it is possible for the IT generalist to work with VMware in addition to their regular day job (which may involve supporting everything else in the IT estate). At first glance, the SDDC makes this a lot more difficult. There are more virtual devices to manage and more places to configure more settings.

The responsibility is therefore on the VMware senior administrators/architects to design using clear principles, document and create the orchestration workflows to ensure that complex tasks are well managed so that the IT staff responsible for operations can do their job without needing to fully understand the complexity of the environment.  

IT architects: Be prepared to invest time learning how all this fits together.

4. Make it easier for users

The truth is that today, end users have to jump through too many hoops to get their VMs provisioned and the cost model is still unclear.

Although VMware have demonstrated vCloud Automation Manager, capable of handling the provision request/approval process, by limiting it to customers running vCloud Suite Enterprise, the rest of us are left with no ability to provide a request/approval mechanism for VM creation. It is likely that something will need to be written in-house, perhaps using something like WaveMaker Studio front-ending a series of Orchestrator workflows.

There are other ways to make life easier for our users. As admins, we currently use vSphere templates to make VM deployments simple, but this should be extended to create a full vCloud Director application catalogue for our end users. For example, if our end users want a Red Hat Enterprise Linux server with Apache Tomcat already installed, we should be in a position to make a catalogue item available for this purpose.

Using the vFabric suite to deploy these applications is probably overkill, but there are alternatives. It would be a useful exercise to do some work with Puppet to look into the provisioning of applications.

5. Monitor and plan

Finally, as things get more abstracted and complex, there is a real need to manage this infrastructure and proactively plan for future growth. VCOPS Foundation will be the starting point for this, but it's possible (probable?) that a fuller featured edition may be required over time.

There is going to be a real need to perform capacity planning to ensure that existing infrastructure resources are utilised properly, and that additional hardware is available when required.

Rationale

The above needs to be done in order to allow IT to keep up with the business demands of our users.  We need to stop being a stumbling block and instead come up with ways to deliver services faster.

The amount of servers and applications we now manage is greater than ever before and there is no sign that this will change in the near future (probably the opposite, and demand will continue to increase!).

We can't keep doing things the same way (it doesn't scale and it's too slow). We need to be smarter and more proactive about managing our infrastructure and providing the applications and environments our users need.

VMworld Europe 2012: Day Three

The final day of VMworld began and I again opted to spend the majority of my time in the hands on labs.

Back to the Hands on Lab

The first lab of the day was based around the new features and capabilities built into the vCloud Networking and Security (vCNS) product, previously known as vShield Edge and App. This product is a virtual appliance based firewall with some advanced capabilities such as IPsec VPN and load balancing. Previous versions of vShield sat at the border of a vSwitch port group and provided only two interfaces (inside and outside). The new version can support multiple networks with the example given being a traditional three legged design for external, DMZ and internal networks.

Another useful feature of vCNS is the ability to put two appliances in an active/passive configuration. As part of the lab, the active instance was powered off and the failover kicked in, losing only three ping packets before the peer picked up the load. As previous versions of vShield Edge represented a single point of failure, this addition is welcome for mission critical or highly available environments.

The second lab was an epic example of tying a number of products together to illustrate the automation and provisioning of applications. Understatedly titled "Deliver Your IT Services in the Cloud", the lab utilised VMware Service Manager to request a new vApp as a consumer, Zimbra for the administrator to receive the request and then action it in VMware Service Manager, which then kicked off a workflow in Orchestrator that provisioned a VM in vCloud Director and then installed the web application using vFabric Application Manager. This advanced level of automation and integration across the various products provided perhaps the best illustration of where VMware are going with its Software Defined Datacenter strategy. In contrast, it's interesting to see how little focus there is on the "traditional" VMware strengths of virtualising compute, network and storage. A hint to competing hypervisor vendors: the world has moved on and it's now about building and managing automated application stacks built on top of public/private/hybrid clouds.

The third lab was a focus on the new features of the Distributed vSwitch. As new Enterprise Plus users, the Distributed vSwitch will be a new addition to our infrastructure and the lab provided some useful troubleshooting tips.

The final lab covered the new features in the latest release of VCOPS. The analytics engine and presentation of data in VCOPS is amazing (no change there) and it was interesting to see how this is becoming the management platform for VMware.

vCenter Operations Manager

By which point my brain was struggling to assimilate any new information! There was no big bang event to close the conference, just a steady stream of people finishing up and heading back to their hotels.

There is still much to reflect on, and my deliberate strategy of tackling the Hands on Labs at the expense of attending the sessions means there will be significant ongoing catching up online over the coming weeks and months.

The value of attending a conference such as VMworld is that it gives an opportunity to deeply dive into the technology we use everyday, familiarise ourselves with the developments that will be with us soon, and allows us the space to think, plan, learn, discuss and ultimately equips us to do our jobs. For those of us who are passionate about the work we do, it's an amazing experience and I'm personally very grateful to my company for making it possible to attend.

VMworld Europe 2012: Day Two

Day two started with another keynote, this time focused on "end user computing". VMware demonstrated a number of new features to View, Horizon and application delivery to iOS and Android devices.

None of this really interests me, so, meh.

My first session of the day was around the vCloud Architecture Toolkit, which I only stayed in for the first ten minutes. The pace of the presentation, along the presenter providing a definition of cloud (really? It's 2012!), led to me bailing and going the Hands On Lab instead.

Hands on Lab

At the lab, I took several sessions throughout the day. Two sessions were based around the vFabric Application Director and vFabric Data Director. These two products are designed to facilitate the provisioning of applications to a cloud environment. In the first lab, you had to provision a database (MySQL), application (Tomcat) and web (Apache HTTP) server, connect them dynamically, and then scale out the middle tier. The resulting blueprint is then built as a vApp which is available through vCloud Director. Clever stuff, but quite a steep learning curve (following the guides were easy, but starting from scratch would be a formidable challenge!).

Of interest was the list of options of components that can be used in building an application blueprint. In addition to the open source stacks were objects for SQL Server and IIS, although the lab didn't touch on these components.

The vFabric Data Director lab was similarly interesting, providing a single web front end into the management and deployment of Oracle and Postgres databases. The database instances could be cloned, resources hot-added and SQL run through the interface. Of interest to me was that the underlying mechanism to manage these instances is to manipulate and clone VMs. In other words, the object that is being managed is a VM. The application doesn't appear to be doing anything to enable multi tenancy within a single database installation (or if it does, this wasn't covered in the lab). It was very impressive to be able to take a Postgres 9.0 installation, rapidly copy the VM using linked clones, and then apply an upgrade to 9.1!

For more information (and to see what the above looks like):

A short video introduction to vFabric Application Director
A short video introduction to vFabric Data Director

The final lab of the day was on the new vCenter Orchestrator, a product most of us VMware admins have had for years, but few of us have used. This lab went through the creation of workflows that are then run against objects in vCenter. The lab then extended this with the use of plugins, showing how Orchestrator can be used to manipulate Active Directory. This is something that deserves a greater look and as a result, I bought the VMware Press book, Automating vSphere with VMware vCenter Orchestrator.

While in the lab, I noticed a plasma with a display that looked as if it was showing a Splunk dashboard. On wandering over to the screen, I discovered that this was labelled "VMware Strata". On enquiring with the lab staff, no one appeared to know what it was, but they thought it may be a new product based on a VMware acquisition that would eventually make it into that all-powerful monitoring and reporting tool, vCenter Operations Manager.

Mysterious VMware Application

Following the labs was the "Hall Crawl", providing another opportunity to meet the vendors. This included a very useful discussion with HP on the use of 3PAR storage and the unique features it provides. The thin provisioning and zero reclaim are very impressive ways to optimise and manage storage usage, although there is currently no deduplication or compression as found in NetApp filers.

The day was finished with the VMworld Party, a huge event with a cool 80s retro computer games theme, many activities and games, live acts and a constant supply of food and drink that ran late into the evening.

The VMworld Party

VMworld Europe 2012: Day One

Arrived yesterday afternoon in sunny, hot and humid Barcelona in preparation for VMworld Europe 2012. Having registered, taken a lab session (on the vCloud Suite), settled into the hotel and eaten an evening meal in a small, local cafe (where the owner knew no English and we knew no Spanish, which made the whole affair very interesting), I made sure I had an early night because this conference is going to be packed.

And so it is!

And so it begins

The Keynote

The keynote kicked off at 9am. Not a huge amount of new stuff announced after the US show, but there is a new version of vCenter Operations Manager (aka VCOPS) which now includes application awareness so you can drill into a VM and get the state of the underlying application (such as the RAM utilisation of an individual Oracle instance). While this is very impressive, unfortunately, VCOPS remains reassuringly expensive (but see below for some good news).

A new plugin was shown that allows the vSphere client to manage third party hypervisors. This, along with both DynamicOps and Nicira supporting multiple hypervisors, is a new approach by VMware and probably reflects the reality in many datacenters.

The vCloud Automation Center was revealed, which is based on VMware's acquisition of DynamicOps. This showed the ability for end users to request new VMs from a catalogue and to have management over their VMs (power on, off, restart etc.) through a web portal. This looks like an essential component in the provisioning process, fulfilling the self-service aspect of the cloud while facilitating an approval process, but according to the VMware website, it's priced at $400 per managed VM(!), which makes it too expensive for all but the largest customers.

The new vFabric Application Director was also shown, with the introduction of the idea of "application blueprints" which take the concept of VM templates to the next level, allowing a drag and drop approach to building an application stack. The Cloud Application Marketplace (aka App Store) was shown where third parties can publish their own blueprint components, allowing for even more sophisticated application stacks. As an example, the screenshot showed some Riverbed components, so you could imagine integrating WAN acceleration into your one-click-to-deploy vApp. The list price for this is $6250, which, like the above, is going to be difficult for many of us to justify, no matter how cool.

A couple of screens were shown of the IT Business Management Suite, which would make a CIO very happy. Lots of cost related metrics and pretty dashboards, but not something I'll personally have any need for.

The keynote ended with a demo of a social networking project. In the demo, hosts, clusters and VMs are social network entities that can be "followed" and "liked" in the same way our Facebook friends can be "followed" and "liked". If a host loses a datastore, it posts a message to the news stream. Other hosts that have the same problem "like" the original message which can be used for analytics. If dozens of hosts all report that they have lost access to a datastore, then there's probably a serious issue. All sounds a bit gimmicky, but it did look pretty decent and got the biggest, most spontaneous applause of the keynote(!).

The main things I took away from the keynote are:

VMware offer so much more than just hypervisor-based virtualisation. Where other hypervisor vendors sometimes compare vMotion/Live Migration capabilities, play "My VM is bigger than your VM" (VMware are guilty of this too), and fight to see who offers the most features for free, the VMware strategy is now about the datacentre and everything in it, automating and orchestrating according to policy.

In VMware language, this policy based automation is referred to as the "Software Defined Datacenter" (SDDC) and is a logical continuation of the ability to virtualise resources. We're now comfortable virtualising compute, storage and network connectivity, but VMware are looking to extend this to network edge devices (load balancers, WAN compression, firewalls etc.). While virtual appliances already exist for some of these, the big difference is that the SDDC is built on the idea that these devices can be scripted, orchestrated and therefore automated by policy. This further extends (through the vFabric suite) into the application layer.

Stop and think about this for a bit, because this is actually a big deal. What happens when your entire datacentre is software defined? You get to throw the word "agile" into the mix...

Want to know more about the SDDC? Check this short video: http://www.youtube.com/watch?v=Jf0KdjpxgCI

A negative comment regarding the keynote announcements, it is frustrating that customers who opted for the top end solution in the VI3.5 days with the Enterprise edition were left in the cold with 4.0's introduction of Enterprise Plus and subsequently upgraded, are now migrated to vCloud Suite Standard which still leaves many useful features out, only available in vCloud Suite Advanced and Ednterprise. I understand that each edition adds many new features, but it seems like a ploy by VMware to constantly generate new revenue by making customers pay out for ever bigger software solutions.

Post keynote

Following the keynote, I had a quick wander around the Solutions Exchange and got into a conversation with a Cisco or NetApp employee about the new ExpressPod which appears to be a low cost FlexPod built around the C-series rack mount server and Nexus 3000 switches (with NetApp storage). This may be worth investigating further...

The Solution Exchange

Also in the Solution Exchange, I had a conversation with Veeam about support for vCloud Director and end user, self-service restores on backed up VMs. Nothing yet announced, but Backup and Replication release 7 should be interesting next year. There was also the suggestion that tape support was coming, but on later reflection, this may have limitations for those of us running a purely virtual Veeam deployment.

The Solution Exchange was also the venue for an unexpected encounter with Brent Spiner who was signing photos and chatting to people!

It's Data!

And, to the probable dismay of my wife, I now own a red fedora, courtesy of Red Hat (and she thought last year's bandana was bad...).

The rest of the day was filled with sessions, including an excellent multi-vendor presentation on storage best practices by Chad Sakac (EMC) and Vaughn Stewart (NetApp). Storage is a topic of interest to me and I've read up a fair amount on it, but was pleased to find myself noting down many "to-do" items for the infrastructure I manage. Very useful!

The second session attended was on future storage developments, including vVols, Virtual SAN and Virtual Flash. Interesting technology and it's useful to know this stuff is coming down the road.

While sitting in the Hang Space reading up on the new developments, I came across a reference on VMware's website that a new, entry level version of VCOPS called vCenter Operations Manager Foundation, will be FREE to all vSphere customers. While this removes about 90% of the product's feature set, it will be a good starting point for infrastructure management and alerting.

Part of the Hang Space

The early evening was spent in the Solutions Exchange for the Welcome Reception, basically an opportunity to engage with vendors while eating canapés and drinking free wine/beer/juice/water.
And that, for me, was enough for the day! There are parties running late into the evening, but tomorrow promises to be an equally busy day, so time to get some head down time.

Other thoughts...

On a less IT related note, this venue is very impressive (but not noticeably better or worse than Copenhagen). The conference centre is huge, requiring the use of those automated walkways normally found in airports to get around. There is a constant availability of free food and drink on offer and plenty of space to sit, relax and reflect on all the new developments. Credit to the event organisers who really understand their audience and do everything to keep attendees happy and focused!

As with last year's conference, it's interesting to note the use of smartphones and tablets, alongside laptops. It seems that most people have converged down onto two devices, not one. Of all laptops being used by conference attendees, I'd guess that easily 50% are MacBooks, implying that either VMware admins prefer to spend time doing VMware stuff and not installing Windows patches, or they are easily seduced by shiny hardware...

Passing the CCNA Security exam

The CCNA certification is valid for 3 years and mine was due to expire at the end of August 2012. I could either retake the same exam and recertify, or take another CCNA "concentration" exam that would give me a new certification and renew the original certification at the same time. I opted to tackle the CCNA Security exam, IINS 640-553.

I'd originally bought the Cisco Press "Authorized Self-Study Guide", Implementing Cisco IOS Network Security by Catherine Paquet back in 2010, but the material is a bit dry and I didn't have the motivation to get into it very far. By booking the exam, I suddenly acquired the motivation required.

As things happen, the 640-553 exam is due to be retired in September 2012, to be replaced by 640-554. The main difference in the new exam appears to be an additional focus on the Cisco ASA platform, as well as de-emphasising the Cisco Secure Device Manager (SDM). This means that any advice I give here will be redundant soon, and also I'm bound by the NDA, so can't obviously comment on what is in the exam.

What I can do though is give some general thoughts on the revision process:

The Good

The Implementing Cisco IOS Network Security book is very thorough. It covers a lot of detail and assumes little prior knowledge of security. Some of it is dry, especially the first chapter which weighs in at about 100 pages and gives an introduction to the world of security. Once that's passed, the content gets better and even the chapter on cryptography was interesting(!).

I also bought the Cisco CCNA Security Lab manual for the CCNA Security course. This gave some very good exercises to run through which were very useful in grounding the theory in the practical.

All of this was made possible using the amazing GNS3 router simulation software. I installed this on a meaty Windows Server VM and was able to run the 3 routers and 2 XP images (in Virtualbox, under ESXi) without any problems. The ability to save configurations and easily re-import them was a great time saver. GNS3 doesn't do everything (specifically switches, due to the custom silicon in them), but it made the whole process of learning the syllabus a lot easier.

There is some very good material at the Cisco Learning Network including free study chapters, training videos and discussions. Highly recommended.

The Bad

Cisco sell the book for self-study, but make it very difficult to practice because IOS images are not available without having the correct support contract. If you work for a large company with either old routers sat on a shelf or a contract with the ability to download the image then you'll be okay. Otherwise I guess you'll be searching the Internet for a dodgy copy of an old image. Seriously Cisco, how about making them freely available? You can do the study material with a 2600 series router and how old is that?

The same is true of the IPS signatures. A valid contract is required just to learn how the IPS works and again, this could mean a trip to the darker parts of the Internet to find them.

The Cisco Press book covers the Cisco Access Control Server software but it's not in the syllabus or lab manual. It can be used to learn about AAA and specifically authentication and authorization with RADIUS and TACACS+. Unfortunately Cisco don't have a trial version to help self-studying students.

The Ugly

Getting the Cisco Security Device Manager (SDM) working requires jumping through a number of hoops. To cut a long story short, you need an old version of Java (1.4 worked for me) and Windows XP. I'm guessing the latter requirement is due to Internet Explorer 6 as I couldn't get it working on Server 2008 R2 no matter what settings I tried.

Conclusion

Having worked through the labs a number of times and then setting things up "blind" (without referring to any notes), I felt fairly confident as I went into the exam. I passed with a good mark well above the passing level, so I'm naturally very pleased with this. It's a good subject to read up on since security requirements impact on so much of what we design these days. The CCNA Security should demonstrate I now have a solid grounding in the subject, even if I'm still a long way from being an expert.

Home lab upgrade: HP ML110 G7

My home vSphere infrastructure has, until very recently, consisted of:

• 2 x HP Microserver N36L
• 1 x HP ML110 G5
• 1 x HP ML115 G5

The Microservers run my domain controllers, DNS servers, Nexenta NAS/SAN VMs and vCenter Server, vMA etc., leaving the ML servers to run any test servers I needed.

Unfortunately, the ML115 G5 PSU died recently, leaving me with a single server. While looking at the options (the PSU was really expensive to replace!), I came across the ML110 G7 cash back offer (£150!). Having counted the pennies and done some reading up, I ordered two of these to replace the existing ML servers. I replaced the included 2GB DIMM with 4x4GB DIMMs. (side question: what do people do with the 1GB and 2GB DIMMs that come with servers that we immediately replace with something bigger???).

This arrived today:

The inside of the ML110 G7 is pretty clean and fitting the extra RAM was very straightforward:

The G7 is roughly the same size as the G5:

And with both of them in place (with the Cisco SG200-26 and Belkin SOHO 4 port KVM on top):

In order to make the ML110 G7 work with VMware ESXi, the BIOS needs to be configured with the correct "C State". I used this very useful blog post to set mine.

I installed ESXi using a USB CD-ROM drive and everything went very smoothly. Added both to vCenter Server, mounted an NFS datastore, setup iSCSI and configured the vSwitch for correct VLAN membership and everything is now ready to go. This upgrade means I'll be in a better position to test vCloud Director, vShield etc.

Mac Mini upgrade

My main computer is a mid-2007 Mac Mini. While it's performed very well in the time I've had it, recently I've become frustrated with how slow it is running with Mac OS X Lion.

The cost of a new Mac is beyond what I have to spend at the moment, so I looked into ways to upgrade the Mini. I've previously been hesitant to do this as getting inside the Mini requires some effort. But these are difficult times, so I bit the bullet and opted to upgrade the memory and replace the internal hard disk.

RAM upgrade

I wanted to upgrade the RAM from the 2GB that it came with. This is the maximum that Apple officially support, but you can put more in and it will work. There is a caveat: Due to the chipset used, only 3GB can be addressed by the operating system (even though it will detect more). This meant I could either go for 1GB + 2GB solution, or 2GB + 2GB and effectively waste 1GB of it. I eventually went for the latter because I had read that putting equal sized DIMMs enabled the RAM to run at full speed. I don't know if this is really the case, but the cost difference was very small.

The RAM I bought from Ebuyer was the Crucial 2GB DDR2 667Mhz/PC2-5300 Laptop Memory SODIMM CL5 1.8V (quickfind code: 142421) and cost just over £20 per stick. I bought two sticks.

Disk upgrade

The hard disk that came in the Mac Mini was a 120GB SATA 5400RPM drive. The rotational speed of the disk is at the low end (typical desktops are 7200RPM, with servers having 10000RPM or 15000RPM drives). It was never particularly speedy, but did the job. With the cost of SSDs falling, I could replace the disk with something solid state and much faster.

Ebuyer had an offer on the OCZ 120GB Agility 3 SSD (AGT3-25SAT3-120G) (quickfind code: 268244) costing £80.

There is a lot of discussion about the support for TRIM in Mac OS X. TRIM is a command that the operating system can send to the device to notify it that disk blocks are free and can be wiped. Without TRIM support, disk writes can slow down over time. Official Apple SSDs support TRIM, third party SSDs don't. There is a kernel extension that hacks third party devices, but that was a bit too risky for my liking.

Reading up on the OCZ Agility 3, it includes on-chipset garbage collection that performs  reclamation of blocks in the background. It may not be as good as TRIM, but it does a similar job.
 So, I'm running without TRIM and will see over time how it performs.

Performing the upgrade

Due to the way that the Mac Mini is constructed, getting into the case can be a challenge. You will need a putty knife to get inside the case. Once opened, a very small cross-head screwdriver is required to detach the optical and hard drives. Inserting the RAM is straightforward once this is done. Similarly, replacing the hard disk was pretty easy once inside. There are plenty of guides online that show how to do this.

With the internal drive replaced, there is no operating system to boot from. I had burned a DVD image of the Lion installer and used this to boot the machine (hold down "c" when powering up and until the Apple logo and spinning wheel appears). The DVD has the option to restore a Time Machine backup which I selected. You need to load the Disk Utility to format the SSD, at which point Time Machine can be used to restore a backup (held on an external USB drive) to the internal disk.

The restore process started and I walked away. When I came back, the Mini was asleep and I couldn't get it to wake without performing a power off/on. At which point I didn't know whether it had completed or not. I repeated the process and watched it, moving the mouse every couple of minutes in case power management was sending it to sleep. With six minutes of restore remaining, the machine put itself to sleep. This wasn't power management. This was a crash.

I then tried booting off the original Leopard DVD that came with the machine. I performed the Time Machine restore from here and it completed successfully without incident. I'm not sure if this is a bug in the Lion DVD, but it's worth noting if you have the same problem!

With the machine restored, I booted off the SSD and loaded a few apps. The extra memory and SSD make it feel like a new computer! Where the original configuration was taking up to 2 minutes from power on to the desktop, it now does it in 48 seconds (including Finder opening network drives)! Most applications load very quickly with no noticeable delay.

I previously used Google Mail through a browser as Mail.app felt like unnecessary overhead and was quite slow. Now it's fast and I can keep it open all the time in the background (along with iCal which was another application I had given up on).

Spending £120 on a couple of upgrades has given the Mac Mini a new lease of life. Although the hardware won't be up to running Mountain Lion when it's out later this year, I should be able to get another couple of years out of the Mini, which is definitely worth it.

Personal Cloud: What I use (May 2012 edition)

This blog originally started because I was looking for ways to move much of my online life to cloud services. It's since grown to encompass some of my technical projects, but I still have an interest in what is now referred to as "personal cloud" services. So, as of May 2012, these are the services I use:

Email

I use Google Mail for my main, personal email. For signing up to web sites etc., I also have a Yahoo email address which supports disposable addresses, very useful for creating per-domain, unique addresses that can be removed if I start getting spam. Just for completeness, I also have a Hotmail account which gets limited use and is used primarily for logging into Microsoft services.

Calendar

Google Calendar synchronises with my iPhone/iPad and my Outlook calendar using Google Calendar Sync.

News

Google Reader remains my main application for RSS feed aggregation. I use it with Reeder on the iPhone and iPad for mobile reading.

Notes

I absolutely love Evernote. I use it to store multiple notebooks containing all my research, notes, web clipping (especially useful now Evernote Clearly has been released) and it acts as a single "dumping ground" in which to throw my thoughts and anything I find interesting. I run the client on Windows, Mac, iPhone and iPad. It's so good I pay for it as a premium user.

Backups

I have a Crashplan+ subscription and use it to backup my Windows PC, T's Windows PC and my Mac. Data is encrypted before leaving my home network so is secure online. There is a real peace of mind knowing that a copy of all our family documents - and photos - are backed up online. It's worth paying for.

Passwords

I've recently started using LastPass to manage all my website passwords as well as provide a place to store other private data (such as computer account credentials). LastPass encrypts data locally, but stores the results in the cloud. There is an app for the iPhone, but you need to be a premium user to get it. I'm only using the free version at the moment, but I like what I've seen with it so far and may subscribe.

[Update: I liked LastPass enough to pay for the Premium version. Recommended!]

File sharing

I've had a Box account for a few years (with 5GB of free space) and it's recently changed its focus to become more of a SharePoint alternative. It's pretty good, but this space is getting crowded with 25GB free with Microsoft SkyDrive, 5GB with Google Drive and 2GB with Dropbox. I wouldn't use any of these services to store my important, personal data (especially since it's unencrypted), but for non-sensitive data, it's good to have options, especially if you need to collaborate with someone. It's too early to determine what service I'll end up focusing on, so watch this space as the products mature.

EMC VNXe - diving under the hood (Part 5: Networking)

A quick look at the EMC Community Forum for the VNXe will show a lot of questions around the best way to configure networking. This is partly due to the way that networking is handled by the VNXe.

Put simply, it's different from the CLARiiON and Celerra.

As the previous post illustrated, the VNXe operating system is based on Linux with CSX Containers hosting FLARE and DART environments. To understand how networking is handled in the VNXe requires looking at various parts of the stack.

The physical perspective

Each SP has, by default, four network interfaces:

• 1 x Network Management
• 1 x Service Laptop (not discussed here)
• 2 x LAN ports (for iSCSI/CIFS/NFS traffic)

Additional ports can be added in the form a a SLIC module, but I don't have access to any of these, so won't discuss that here.

The Linux perspective

Running ifconfig in an SSH session reveals a number of different devices:

• bond0
• cmin0
• eth2
• eth3
• eth_int
• lab (???)
• lo (loopback)
• mgmt
• mgmt:0

The Linux "mgmt" device maps to the physical management NIC port, but does not have an IP address assigned to it. A virtual interface, "mgmt:0" is created on top of "mgmt" and this is assigned the Unisphere management interface IP address. This is almost certainly due to the HA capabilities built into the VNXe. In the event of a SP failure, the virtual interface will be failed over to the peer SP.

End user data is transferred over "eth2" and "eth3". The first thing to note from the output of ifconfig is that the MAC addresses of both interfaces are the same.

Another device, "bond0", is created on top of eth2. If link aggregation is configured, eth3 is also joined to bond0. This provides load balancing of network traffic into the VNXe.

There is also a "cmin0" device which connects to the internal "CLARiiON Messaging Interface" (CMI). The CMI is a fast PCIe connection to the peer SP and is used for failover traffic and cache mirroring. The cmin0 device does not have an IP address. It's possible the CMI communicates using layer 2 only and therefore doesn't require an IP address, but that's only speculation.

Finally, there is an eth_int device that has an IP address in the 128.221.255.0 subnet. This is used to communicate with the peer SP and either uses the CMI or has an internal network connection of some kind.

A quick check of a client machines ARP cache reveals that the IP addresses of Shared Folder Servers in the VNXe do not have a MAC address that is mapped to any of the Linux devices. So how does IP traffic reach the Shared Folder Servers if the network ports are not listening for those MAC addresses? Running the "dmesg" command shows the kernel log, including boot information. The answer to the question can be found here:

[  659.638845] device eth_int entered promiscuous mode
[  659.797273] device bond0 entered promiscuous mode
[  659.797283] device eth2 entered promiscuous mode
[  659.799994] device eth3 entered promiscuous mode
[  659.805381] device cmin0 entered promiscuous mode

On start up, all the Linux network ports are put into promiscuous mode. This means that the ports listen to all traffic passed to them regardless of the destination MAC address and can therefore pass traffic up the stack to the DART container.

The DART perspective

The CSX DART Container sits on top of the Linux operating system and provides its own network devices:

• DART vnic0 maps to Linux bond0
• DART vnic1 maps to Linux eth3 (presumably unless eth3 is joined to bond0)
• DART vnic0-b maps to the Linux cmin0 device
• DART vnic-int maps to the Linux eth_int device

DART also creates some "Fail Safe Network" devices on top of the vnics:

• fsn0 maps to vnic0
• fsn1 maps to vnic1

While ifconfig provides information from within Linux, to get the DART information requires the "svc_networkcheck -i" command. This reveals some useful information.

The DART fsnX devices are virtual devices that map to the underlying DART devices in an active/standby configuration:

• fsn1 active=vnic1 primary=vnic1 standby=vnic1-b
• fsn0 active=vnic0 primary=vnic0 standby=vnic0-b

If that wasn't enough layers of indirection, DART creates some additional interfaces on top of the fsnX devices:

• rep30 - IP on the internal 128.221.255.0 network and maps to vnic-int
• el30 - IP address of DART instance "server_2" on the vnic-int interface, 128.221.255.0 network
• if_12 - maps to device fsn0 and contains the user configured IP address of the Shared Folder Server

This appears to suggest that a Shared Folder Server (which is basically equivalent to a Data Mover in the Celerra, albeit now implemented in software and not hardware) has a front end external network interface and a back end internal network interface. I'm not sure what the significance of "el30" or "if_12" is, but seems to be carried over from the Celerra.

I don't know what the rep30 interface is, but guess it could be an address for replication to use if licensed and configured.

This might be more clearly explained with a diagram:

Failover

So how does failover work?

It would appear that there are different failover technologies used. The Linux-HA software is used within Linux to provide management interface failover to the peer SP.

It's also likely that DART is doing some form of HA clustering as well. On a Celerra, DART redundancy was handled by setting up a (physical) standby Data Mover. Given that DART is running as a CSX Container, does the peer SP actually run two instances of the DART CSX Container, one active for the SP, the second running standby for the peer? I don't know, but it would make some sense if it did, and would also help explain what the 8GB of RAM in each SP is being used for.

Network Configuration

This document has a good overview on how to best configure networking for a VNXe. The following hopefully explains "why" networks should be configured in a particular way.

The "best" approach does depend on whether you are using NFS or iSCSI. The important thing to understand is that they make use of multiple links in different ways:

Stacking switch pair with link aggregation

If both eth2 and eth3 are aggregated into an Etherchannel, then the failure of one link should not cause a problem. Redundancy is handled at the network layer (through the bond0 device) and DART should not even notice that the physical link is down.

With an aggregation, traffic is load balanced based on a MAC or IP hash. With multiple hosts accessing the VNXe, the load should be balanced pretty evenly across both links. However, if you only have a single host accessing the VNXe, chances are you will be limited to the throughput of a single link. Despite this limitation, you will still have the additional redundancy of the second link.

Separate switches (no link aggregation)

If eth2 and eth3 are connected to separate non-stacking switches then eth3 would not be joined to bond0, but would connect directly to vnic1 and be used by a different Shared Folder Server or iSCSI Server to one using eth2.

Therefore, the only connection on the SP is via eth2 and if it fails, DART would detect that vnic0 has failed and the fsn0 device will failover from vnic0 to vnic0-b. This will then route traffic via the peer SPs physical Ethernet ports via the cmin0 device. Presumably a gratuitous ARP request is sent from the peer SP to notify the upstream network of the new route.

This is why eth0 on SPA must be on the same subnet (and VLAN) as SPB. If a failover occurs, then the peer SP must be able to impersonate the failed link.

This is a big difference from the CLARiiON which passes LUN ownership to the peer SP, or Celerra which relies on standby data movers to pick up the load if the active fails (although as noted above, it might still do this in software). In contrast, there should be little performance hit if traffic is directed across the CMI to the peer SP (although the peer SP network links may be overloaded as a result).

Conclusion

This pretty much concludes this mini-series into the VNXe!

It goes to show that even the simplest of storage devices have a fair amount of complexity under the hood and despite its limitations in some areas, the VNXe is a very good entry level array. It is impressive how EMC have managed to virtualise the CLARiiON and Celerra stacks and it makes sense that this approach will be used in other products in the future.

Thanks for reading! Any comments and/or corrections welcome.

EMC VNXe - diving under the hood (Part 4: CSX)

After the last post, I was pointed in the direction of the "VNXe Theory of Operations" online training available from education.emc.com (just do a search for it). This free course provides some interesting details into the VNXe architecture.

With knowledge gained from the course in mind, let's see if we can get a better understanding of what's happening under the hood...

C4LX and CSX

When the VNXe was announced, Chad Sakac at EMC referred to the it as "using a completely homegrown EMC innovation called C4LX and CSX to virtualize, encapsulate whole kernels and other multiple high performance storage services into a tight, integrated package."

In the same blog post, Chad also illustrated the operating system stack which showed the C4LX and CSX components are built on a 64bit Linux kernel.

CSX (short for "Common Software eXecution") is designed to provide a common API layer for EMC software that is not tied to the underlying operating system kernel. As a portable execution environment, CSX can run on many platforms in either kernel or user space. So when some functionality is written within the CSX framework (e.g., data compression), it can be easily ported to all CSX supporting platforms, regardless of whether the underlying operating system is DART, FLARE or something else. Steve Todd has some more details about CSX on his blog.

So if you read that CSX instances are similar to Virtual Machines, think of it in terms more like a Java Virtual Machine rather than a VMware Virtual Machine. It's an API abstraction and runtime environment, not a virtualisation of physical resources such as CPU and memory.

There aren't many details on what C4LX is, but here's my conjecture: There are some functions that CSX needs the underlying operating system to perform that may not be easily possible "out of the box". If that's true, then C4LX is the Linux kernel along with a bunch of kernel modules and additional software that provide this functionality. Or another way to describe it might be to call it EMC's own internal Linux distribution...

Data Path

Like the data plane and control plane in a network switch, software in the VNXe appears to be designed to operate on the "Data Path" or the "Control Path".

CSX creates various "Containers" that are populated with "CSX Modules". A Container is either a user space application or a kernel module. CSX Containers implement functionality within the Data Path.

The FLARE functionality described in part 2 of this series is implemented as a CSX Module, as is the DART functionality described in part 3. Both these modules run in the Linux user space. There is a degree of isolation between Containers in that they be terminated and restarted without interfering with other Containers. However, some Containers (such as DART) have dependencies on other Containers (FLARE).

In addition to the FLARE and DART Containers, a Global Memory Services (GMS) Container provides memory management functionality and services other Containers. As an example, the FLARE Container takes 500MB memory, while the DART Container takes 2.5GB, all allocated by the GMS.

A kernel space Container is responsible for allocating resources on behalf of user space Containers. The Linux Upstart software provides a means to start, stop and restart Containers.

Control Path

The Control Path is a implemented using technology derived from the Celerra Control Station (itself a Linux-based server) and the CLARiiON NaviSphere software. The Control Path is also where the Common Security Toolkit (CST) is found. The CST appears to be RSA technology and is used in multiple EMC products for security-relation functions. In contrast to the Data Path which consists of functionality directly relating to the transferring of data, the Control Path is concerned with management functionality.

Within the Control Path of the VNXe is the EMC CIM (Common Interface Module) Object Manager (ECOM) management server. ECOM interfaces with "Providers" which are essentially plug-ins. Within the VNXe, ECOM runs on the master SP.

There are a number of different Providers. These include Application Providers for Exchange, iSCSI, Shared Folders and VMware software provision, a Virtual Server Provider, Pools Provider, CLARiiON Provider and Celerra Provider. There are also providers for Registration, Scripting, Scheduling, Replication etc. As plug-ins to the ECOM server, additional services can be written to extend the functionality within the VNXe.

With the use of Providers, ECOM implements a middleware subsystem that can be called by front end applications such as Unisphere or the VNXe command line.

In addition to running Providers, ECOM also provides basic web server functionality used by the Unisphere GUI and CLI via the Apache web server.

Pulling it together

The VNXe uses some additional Linux software along with the custom CSX and ECOM components. High availability is implemented through the open source Pacemaker cluster resource manager and using the Softdog software timing kernel driver. CSX components are resource managed using the cgroups feature of the Linux kernel. The Logging system uses the Postgres database. Although this is covered in the EMC training, it's also possible to see this by checking the output of "ps" from an SSH session.

To understand how the various components hang together, the boot sequence looks a bit like this:

1. BIOS/POST
2. Linux boots and initiates run level 3
3. The "C4" stack is loaded by the Linux Upstart software:
• CSX infra
• Log daemon
• GMS Container
• FLARE Container
• admin
4. Pacemaker is loaded and automatically starts:
• Logging
• DART Container
• Control Path software (ECOM on the master SP based on mgmt network status)

At which point, all the components are initialised and ready to go. Obviously there are additional details that I'm not covering (check the training if you want some more information on functionality such as replication, vault-to-flash, more on HA, licensing, logging and deduplication).

Hopefully this gives some insight into the complexity that underpins the VNXe. We're going to look at one more topic to conclude this mini series, and it's a subject that is the source of many questions on the EMC VNXe Community forum. In the next post we'll have a look at VNXe networking...

EMC VNXe - diving under the hood (Part 3: DART)

In the previous post, we looked at the parts of the VNXe that are derived from the FLARE (CLARiiON) code. The result is a number of LUNs that are presented up the stack to the DART (Celerra) part of the system.

Using the "svc_storagecheck -l" command, we can see that a total of 20 disks are found. These map to the two FLARE LUNs from the 300GB SAS RAID5 RAID Group and the sixteen FLARE LUNs from the 2TB NL-SAS RAID6 RAID Group, plus two other disks: root_disk and root_ldisk.

root_disk and root_ldisk appear to map to the internal SSD on the Service Processors and are not visible to the end user for configuration. These disks appear to have root filesystems, panic reservation and UFS log filesystems.

The FLARE LUNs are seen as disks to DART and are commonly referred to as "dvols".

The dvols are grouped into Storage Pools. The following are defined by the system, along with a subset of their parameters:

Name	Description	In use	Members	Volume Profile
clarsas_archive	CLARiiON RAID5 on SAS	False		clarsas_archive_vp
clarsas_r6	CLARiiON RAID6 on SAS	False		clarsas_r6_vp
clar_r1_3d_sas	3 disk RAID-1	False		clar_r1_3d_sas_vp
clar_r3_3P1_SAS	RAID-3 (3+1)	False		clar_r3_3P1_SAS_vp
performance_dart0	performance	True	d18,d19	N/A
capacity_dart1	capacity	True	d23,d24,d25,d26 d27,d28,d29,d30 d31,d32,d33,d34 d35,d36,d37,d38	N/A

As the above table shows, the LUNs presented from the FLARE side of the VNXe are assigned to the performance_dart0 and capacity_dart1 pools.

The Volume Profile should be familiar to Celerra administrators and is the set of rules that define how a set of disks should be configured.

On a Celerra, disks could be configured manually (if you know exactly what you want) or automatically using the "Automatic Volume Manager" (AVM). Because the VNXe is designed to be simple, AVM does all the work.

An AVM group called "root_avm_vol_group_63" (the svc_neo_map command refers to this as the "Internal FS name") has been created and consists of two dvols, d18 and d19 that corresponds to the performance_dart0 storage pool. These two dvols map to the two LUNs presented from the 300GB SAS disk RAID Group. It appears when a filesystem is created, the first disk is partitioned into a number of slices (sixteen on d18). Each slice then has a volume created on it and finally, another volume is created that spans across all the other volumes. It's this top level volume, called v139 in the diagram below, on which a filesystem is created:

Note that d19 in the above diagram isn't used. If the filesystem is expanded beyond the capacity of the single disk, then presumably the next disk is used. For some reason, slice 68 doesn't have a corresponding volume. I would welcome any explanation as to why this is.

The configuration for the capacity_dart1 pool is very similar, albeit with many more disks (sixteen instead of two) and many more slices. Unfortunately it's too big to show here. As an example, the first disk, d23, has 40 slices of its own that form part of the pool.

The use of all these smaller slices presumably means that a filesystem can grow incrementally from the pool (and possibly shrink?).

When the filesystem is created, it isn't visible to an external host. On a Celerra or VNX, this functionality would be handled by a physical data mover. The VNXe uses a software "Shared Folder Server" (SFS) which acts as the server to the other hosts on the network.

Multiple Shared Folder Servers can be created (apparently up to 12 Shared Folder Servers (file) and/or iSCSI Servers (block) are supported), each with its own network settings and sharing its own filesystems out over NFS or CIFS. Note that while a SFS can handle both NFS and CIFS, a single filesystem within a SFS can support either NFS or CIFS, but not both at the same time.

From a disk perspective, EMC have done well to hide a lot of legacy cruft away from the user and the encapsulation of FLARE and DART, along with the software implementation of the data mover idea is a neat evolution of an aging architecture.

There is more to look into such as networking (which has provoked a significant number of questions on the EMC forums) and I'd like to find out more about the CSX "execution environment" that underpins much of the new design. I'd be sure to post more if/when I get more information, but hopefully you've found this a useful dive under the hood of the VNXe.

EMC VNXe - diving under the hood (Part 2: FLARE)

In part one we looked at how the EMC midrange CLARiiON and Celerra combination (now replaced by the VNX) presented physical disks to a host as a filesystem. The VNXe is a new architecture physically, but dive beneath the hood and bits of FLARE and DART from the CLARiiON and Celerra are still visible.

The best way to get under the hood and see how the VNXe works is to enable SSH through the Unisphere web interface (it's under Settings > Service System). Then open an SSH session to the VNXe as user "service". First piece of information: It's running SUSE Linux Enterprise Server 11 (64bit).

The most interesting command to find out information about the VNXe storage system is "svc_storagecheck". This takes a number of parameters.

If we start at the bottom of the stack with the "svc_storagecheck -b" (for backend information), we get a lot of information about the array. Note that the word "Neo" crops up a lot and was the codename for the VNXe.

Some of the useful details revealed include the supported RAID types:

RAID Type	Min Disks	Max Disks
NEO_RG_TYPE_RAID_5	3	16
NEO_RG_TYPE_DISK	1	1
NEO_RG_TYPE_RAID_1	2	16
NEO_RG_TYPE_RAID_0	3	16
NEO_RG_TYPE_RAID_3	5	9
NEO_RG_TYPE_HOTSPARE	1	1
NEO_RG_TYPE_RAID_1_0	2	16
NEO_RG_TYPE_RAID_6	4	16

It's worth noting that not all of the above are directly accessible by the user (there is no way to manually create a RAID3 group to my knowledge). Also, the limit of 16 disks per RAID Group is also found on the CLARiiON.

The svc_storagecheck command also outputs details on "Flare's Memory Info" which shows two objects (one per SP?), each having a total capacity of 900 (MB?) with 128 (MB?) Read Cache and 768 (MB?) Write Cache. This might be a surprise if you were expecting the entire 8GB of memory to be available. A lot of this 8GB is used by the FLARE and DART subsystems, along with the Linux operating system itself.

There is also information on "2 Internal Disks" which presumably refer to the internal SSD on each SP which is used to store the operating environment.

Eight Ethernet ports are listed, as are two Link Aggregation Ports that I have setup within Unisphere.

Each of the 12 disks in the array are also detailed, including manufacturer (Seagate), capacity, speed, the slot number etc. Each disk is assigned a type of "NEO_DISK_TYPE_SAS" and also contains a reference to the RAID Group that it belongs to.

There are 2 Private RAID Groups, but no LUNs are presented from it and I cannot determine what this is for. I assume it's used by the operating system.

On my VNXe, there are an additional 3 RAID Groups:

Number	RAID Type	Number of Disks	Number of LUNs
0	NEO_RG_TYPE_RAID_5	5	2
1	NEO_RG_TYPE_HOTSPARE	1	1
2	NEO_RG_TYPE_RAID_6	6	16

The first of these RAID Groups is the RAID5 (4+1) of the 300GB SAS disks, the second isn't really a RAID Group and is the SAS hot spare disk. The final RAID Group comprises the six 2TB NL-SAS disks in a RAID6 (4+2) configuration.

The 2 LUNs are presented from the SAS disks in a way that looks similar to that done on CLARiiON (except on the CLARiiON, each LUN would typically be assigned to a different SP, whereas on the VNXe, this isn't the case and both LUNs are on the same SP).

I'm not sure why the NL-SAS RAID Group presents 16 LUNs, possibly due to the size of each disk. Each of these LUNs is striped across the RAID Group as follows:

The next part of the "svc_storagecheck -b" command details the 19 LUNs that have been defined above.

Each LUN has a name, prefixed with "flare_lun_" which gives a big clue to its history. The default owning and current controller is also defined.

The final part of the "svc_storagecheck -b" command details the Storage Groups used by the array. A Storage Group is an intersection of LUNs and servers. For example. LUN0, LUN1 and LUN2 could be added to a SG with ServerA and ServerB. In this example, both ServerA and ServerB can see LUNs 0, 1 and 2.

There are some in-built Storage Groups (~management, ~physical, ~filestorage and control_lun4) as well as dart0, dart1, dart2 and dart3. The 2 LUNs from the 300GB SAS RAID Group belong to Storage Group "dart0" along with the IDs of the Storage Processors. Similarly, the 16 LUNs from the 2TB NL-SAS RAID Group are mapped to "dart1" along with the two Storage Processors. Storage Groups dart2 and dart3 are unused and presumably for future use.

We can also get some more disk information by using the "svc_neo_map" command, specifying a LUN number:

# svc_neo_map --lun=0

This command can be used to help map the FLARE side to the DART side of the VNXe.

And this pretty much concludes the FLARE side of the VNXe. The resulting LUNs have been carved from the RAID Groups and presented to the Storage Processors in a configuration that the DART side of the VNXe will be able to understand. We'll look in more detail at the DART side in the next post.

EMC VNXe - diving under the hood (Part 1: Intro)

Recently we took delivery of an EMC VNXe 3100. The purpose of this unit is to provide a low cost alternative to our NetApp FAS2040, primarily for backups but also as a VMware datastore for some smaller workloads. Here is is:

If you want a decent primer on the VNXe, check out Henriwithani's blog.

The VNXe is an interesting unit and this mini series will look at what the VNXe 3100 provides and how it works. In order to understand some of the thinking behind the design, we need some background.

First, some history...

(It's worth stating here at the start that I'm not an expert on CLARiiON or Celerra and the below is my understanding based on reading the documentation. If I've made errors, please correct me!)

Go back a couple of years and EMC had two product lines for the midrange: CLARiiON and Celerra.

CLARiiON was their block storage array capable of supporting Fibre Channel and iSCSI. Each controller in a CLARiiON ran an operating environment called FLARE which actually ran on top of a Windows XP Embedded kernel.

Celerra took a CLARiiON, added one or more “X-Blades” (aka “Datamovers”) and a 1U, Linux-based “Control Station”. The Datamovers were basically NAS head units and added support for NFS and CIFS/SMB. They could also do iSCSI, albeit in a somewhat more abstracted way than running native block on CLARiiON. The operating system running on the Datamovers was a UNIX-lilke derivative called DART.

More information on Celerra’s architecture here.

In January 2011, EMC announced the successor to both CLARiiON and Celerra: The VNX. At the same time, the VNXe was announced as an entry-level sibling to the VNX. The VNX appears to be a fairly straightforward upgrade to CLARiiON and Celerra: Faster, bigger, more features etc.

However, the VNXe sports a new software architecture running on an “execution environment” called CSX. From what I can tell by looking at the VNXe support logs and poking around the command line, CSX runs on a Linux kernel. Relevant parts of the FLARE and DART stacks have been ported to run on CSX and each Storage Processor (SP) (aka controller) in the VNXe runs an instance of the Linux/CSX/FLARE/DART stack.

(More information on VNXe here.)

So what you get in a VNXe is a fusion of bits of FLARE and DART, but mixed up in a new way.

The VNXe configuration

Because it is aimed at the non-storage administrator, a lot of the technical details in the VNXe are hidden. This is a bit frustrating for those of us who like to know how things work, so I’ve tried to dive under the hood a bit.

The base VNXe 3100 is a 2U unit and has 12 drive bays and up to two controllers. Each controller (Storage Processor, or “SP” in EMC language) has a dedicated management Ethernet port, and two ports for data called Eth2 and Eth3. There is an option to install a SLIC module in each controller that adds an additional 4 x 10/100/1000Mbit Ethernet ports (copper). We didn’t buy this expansion, so I can’t comment further on those.

With two controllers, the VNXe 3100 has the ability to support up to 96 drives through the addition of Disk Array Enclosures (DAEs). The DAEs connect to the base unit via 6Gbps SAS.

As an entry level system, the VNXe has some limitations in terms of disk configuration options.  Disks are sold in packs and are designed to be added in one of the following ways:

• 300GB SAS: 5 pack in RAID5 (4+1)
• 300GB SAS: 6 pack in RAID10 (3+3)
• 600GB SAS: 5 pack in RAID5 (4+1)
• 600GB SAS: 6 pack in RAID10 (3+3)
• 1TB NL-SAS: 6 pack in RAID6 (4+2)
• 2TB NL-SAS: 6 pack in RAID6 (4+2)

The version we purchased has two controllers and 12 disks: 6x300GB SAS and 6x2TB NL-SAS.

Once installed in the system, the 6x300GB disks can be either configured as “High Performance” which is RAID10 (3+3), or as “Balanced Performance/Capacity” which is RAID5 (4+1) leaving 1 disk as a hot spare. We opted for the latter.

The 6x2TB NL-SAS can only be configured as RAID6 (4+2). Fine for our purposes (backups), but worth knowing as some use cases may require different configurations.

However, it's not as simple as popping in some disks and off you go (well, it is that simple, but there's a lot going on underneath!).

To understand what the VNXe does with the disks requires another background history lesson because there is a bunch of terminology carried over from both CLARiiON and Celerra.

CLARiiON terminology

The CLARiiON works on block storage in the following way:

Physical disks are added to a CLARiiON and a RAID set is created (such as a RAID5 4+1 configuration). The result is call a "RAID Group". Traditionally, up to 16 disks could belong to a single RAID Group.

From within the RAID Group, FLARE (the CLARiiON operating environment) would create LUNs. The most basic type of LUN is the cunningly named "FLARE LUN". Some FLARE LUNs have special purposes and are used internally (for snapshots, logging etc.). These are called Private LUNs.

The problem with a FLARE LUN is that you are limited to the maximum size of the RAID Group (16 disks worth of capacity minus overhead for parity or mirroring). To overcome this, EMC invented the MetaLUN. This construct combines multiple FLARE LUNs together either through striping or concatenation.

For the sake of completeness, later releases of FLARE introduced the concept of a Storage Pool, out of which Thick LUNs (with reserved space) and Thin LUNs (with no reserved space, also known as Virtual (aka Thin) Provisioning) could be created.

FLARE provides some additional features such as compression (which moves the LUN to a pool, converts it to a Thin LUN and compresses it) and Fully Automated Storage Tiering Virtual Provisioning (FAST VP) which combines multiple disk types (such as Flash, Fibre Channel and SATA) into a single pool and dynamically moves hot data to the fast disks). Pretty clever stuff.

Regardless of the type, each LUN is assigned to a preferred owning Storage Processor (SP), allowing the storage administrator the ability to manually balance LUNs for maximum performance. In the event of a controller failure, the peer controller would take ownership. The LUNs are then mapped to be visible to hosts.

Okay, so that's CLARiiON, what about Celerra?

Celerra terminology

To the CLARiiON, a Celerra is a host to which LUNs are presented. These LUNs must be configured in a specific way to be understood by the Celerra operating system, DART.

The LUNs that CLARiiON present are seen as single disks by the Celerra (regardless of the number of underlying physical disks that comprise the LUN). These disks are sometimes referred to as "dvols". You should assume a 1:1 mapping between CLARiiON LUNs and Celerra dvols.

In addition to the disk (dvol), Celerra offers some additional I/O constructs: A "slice" is a part of a dvol, and a single disk can be partitioned into many slices. Similarly, a "stripe" is an interlace of multiple dvols or multiple slices. The "meta" (not to be confused with a CLARiiON MetaLUN) is a concatenation of multiple slices, stripes, disks or other metas.

Finally, a "volume" is created. This is a block I/O construct into which a filesystem is created. And it's these filesystems that are made visible to hosts. The default filesystem is uxfs and is used by both NFS and CIFS fileservers.

As the above illustrates, the path between physical disk and filesystem is pretty complicated. How the VNXe is derived from this messy lineage will be the subject of part 2 (coming soon).

Passing the VCP for vSphere 5

With time running out on being able to certify as a VCP5 without having to take a course, I've spent the evenings of the last few weeks revising and testing in the home lab. As with previous exams, the scoring is emphatically not percentage based and is graded between 100 and 500 with a passing mark of 300(!).

Well, I passed, with a score of 378 (higher than when I did the VCP4) which I was very pleased with, but thought I'd share a few thoughts on the process:

Despite a lot of experience with vSphere and reading up on all the topics in the exam blueprint (this is a must if you want to pass!), I still found myself having to make guesses in a few of the questions. This is because some questions seemed to be more "trivia" oriented and you would only know the answer if you had done that exact operation outside of the exam.

On the plus side, you will note that the exam blueprint doesn't require you to remember a list of configuration maximums anymore!

There are 85 questions to complete within 90 minutes and I found that I answered them all with about 15 minutes to spare, but because I had marked quite a few for review, I used all the time available.

For revision, I used the aforementioned exam blueprint which contains pointers to various VMware documents. This is a huge amount to digest, but it's worth a skim read of all these docs.

I also used Scott Lowe's Mastering vSphere 5 book which currently seems to be the de facto book on the subject, although I did make reference to some parts of Mike Laverick's VMware vSphere 4 Implementation (yes, the previous release) as I feel it gave a better explanation to some sections.

Finally, there is no substitute to hands on experience, and the home lab was fully utilised with 3 nested ESXi servers, both the Windows vCenter Server and the vSphere Appliance tested, along with the VSA, Update Manager, Distributed vSwitches etc.

Special mention must go to the following two sites which have written notes on each of the blueprint sections, both are excellent and really helped in my revision:

• Jason Langer's blog
• Mike Preston's blog

It's worth noting that as a regular user of the "Enterprise" version, getting experience with the "Enterprise Plus" features in the lab (thanks to the 60 day trial) is very important as otherwise there are many things that I would never otherwise see. Hopefully the community effort to reignite the VMTN program is successful and we can all benefit from using the full feature set in our home labs.

So that's the VCP out the way until the next big release. More certs later this year...