Understanding NFS and ZFS interactions

This post was originally going to document the benchmarking of my NexentaStor VSA. Although most of this work has been done (and will be posted soon - promise!), the results were somewhat confusing and required me to dive into the guts of how Solaris (on which NexentaStor is based) handles filesystem operations and NFS. This might be useful if you are trying to debug some performance issues:

ZFS is designed so that all writes are transactional. A write is either successful and the data is written to disk, or it fails and no data gets written. This means that the data on-disk is always consistent.

From an application's perspective, there are two types of write operations to a POSIX compliant filesystem: asynchronous and synchronous.

An asynchronous write passes the data to the filesystem and then continues, effectively assuming that the data is safe. In comparison, a synchronous write passes the data to the filesystem and then waits until the filesystem acknowledges that the data is safe.

Asynchronous Writes

Synchronous Writes

The difference can be seen in what happens if the filesystem or server has a problem (e.g., power fail). In an asynchronous write, the data may only be buffered in RAM and is therefore lost, although the application believes it is safe. With synchronous writes, the data will be definitely be safe because the filesystem only reports back to the application when it is definitely written.

So while the ZFS on-disk data is consistent, it's possible, using asynchronous writes, to lose data in transit. For synchronous writes, ZFS uses a feature called the ZFS Intent Log (ZIL).

When a synchronous write request is made to the filesystem, the data is first written to the ZIL. This ensures the data is safe on disk and the application is free to continue. ZFS will then flush the contents of ZIL to the filesystem at a specified interval (roughly every 5 seconds).

By default, the ZIL is allocated disk blocks from the pool. This leads to a situation where data is written to disk twice, once to the ZIL and secondly to the actual filesystem. This double writing can slow things down.

One option to speed things up is to use a Separate Log (SLOG in ZFS terminology) on which to locate the ZIL. This is typically a flash/SSD drive. Synchronous writes can be logged quickly to the SLOG and then written to slower disk later, speeding up the response to the application. When using a SLOG, the recommendation is to use a mirrored ZIL to ensure that the data is truly safe before being committed to disk.

Synchronous Writes with separate ZIL

How does this impact on NFS?

It is common for an NFS client to write synchronously to an NFS server in order to get an acknowledgement that the data is safely committed to disk. This means that an NFS server with a ZFS filesystem will be performing ZIL writes and come with the associated overheads. This also means the NFS client will be blocked until it receives an acknowledgement back from the server. This can result in "poor performance" when running ZFS and NFS together.

NFS with synchronous writes

There are a couple of options that can speed things up:

The NFS client may opt to mount the filesystem asynchronously. This means that the data will sit in the server's RAM buffers until a transaction group commit and is therefore vulnerable to power failure. NFS clients mounting asynchronously may therefore lose data.



Another option is to disable the ZIL on the NFS server. This effectively makes all synchronous writes asynchronous from the ZFS perspective. Again, data in transit may be lost. Recent versions of ZFS refer to this as sync=standard (default, synchronous writes are written to the ZIL), sync=always (paranoid, everything is written synchronously) or sync=disabled (makes all writes asynchronous).

NFS with async or ZIL disables

Whether NFS async or disabling the ZIL is a risk worth taking depends on the nature of your data. VMware vSphere appears (based on my reading which I'm assuming to be true) to use synchronous writes when using NFS datastores, which can impact on performance. Applying some of the tuning detailed above may help improve performance in a virtual environment.

NexentaStor Community Edition - first impressions

During the Christmas break I took the opportunity to upgrade my HP ML110 G5 from the sadly future-less OpenSolaris to another platform. I opted to turn it into a VMware ESXi 4.1 install to run alongside my existing HP ML115 G5 lab server.

The ML110 G5 was fitted with 2 x 1TB SATA drives and a 60GB SSD drive. All three were presented as datastores to ESXi.

For file and block level storage, I opted to use NexentaStor Community Edition. This operating system is derived from the OpenSolaris code base and builds on many Solaris technologies, including ZFS. The enterprise version is pay-for, but the free Community Edition supports datasets up to 18TB, which is easily enough for a home lab environment.

I installed NexentaStor CE on a fairly small volume and created a larger (400GB) VMDK which I then added to the ZFS pool. I assigned 4GB of RAM to the VM, the majority of which will be used as the ARC cache (see below for details).

A (brief) ZFS Primer

In ZFS, physical disks are grouped together in pools. Writes to a pool are striped across all disks in the pool by default, but disks within the pool can be mirrored to each other, or configured in parity RAID comprising of one, two or three parity disks (called RAIDZ, RAIDZ2 and RAIDZ3 respectively) to provide additional resilience.

ZFS filesystems are created from space in the pool and can have many properties applied including size reservations, quotas, compression and deduplication. Filesystems can be shared over NFS, CIFS, or both concurrently.

In addition to ZFS filesystems, Zpools can also contain Zvols. These are basically ZFS filesystems without the filesystem formatted. Zvols provide many of the same properties as a ZFS filesystem including compression and deduplication. Zvols can be shared over iSCSI and formatted by the initiator to hold a server's native filesystem (such as VMFS, NTFS, Ext3, HFS+ etc.).

NexentaStor CE VSA data integrity

With a single 400GB VMDK created and assigned to the VM, I create a new zpool (called Datasets by Nexenta and configured through the web interface - command line mojo not required) and started creating new ZFS filesystems (called Shares, one to hold software installers, another for ISO images, a third for documents etc.).

Obviously a single disk is no good if there is a problem with the underlying drive, so I created a second 400GB VMDK on the other physical disk and presented it to the appliance (all disk rescanning is done without a reboot necessary). The second 400GB was then added to the zpool as a mirror. The process of copying data from the original disk to the mirror is called resilvering and can take some time.

This mirroring is within the VSA and will not help if the primary disk fails as the VM configuration files and boot VMDK are not mirrored. So why mirror the data?

ZFS stores a checksum for the data it writes and when configured in a mirror or RAID-Z, the filesystem is able to reconstruct the data in the event of disk write errors using the redundant data. See here for more information on the end-to-end checksumming and data integrity.

This means that while the VSA will not survive the primary disk physically dying, any corruptions that occur as a disk starts to die will be caught and corrected. A scheduled housekeeping job called a scrub runs weekly to ensure the checksums and data are correct.

NexentaStor CE VSA performance tuning

SATA disks are slow and SSD is fast. Unfortunately SSD is much more expensive than SATA. While one option is to put performance critical data on the SSDs and less important VMs on SATA, the alternative is to use flash disk as cache.

ZFS utilises an in-memory cache called the "Adaptive Replacement Cache" (ARC). This is very fast (being in RAM) and speeds up disk reads, but is limited to the physical memory in the machine (approximately 3GB in a 4GB VM). However, ZFS can support two additional caches: The L2ARC (Level 2 Adaptive Replacement Cache) and ZIL (ZFS Intent Log). The L2ARC is designed to speed up reads, while the ZIL speeds up metadata writes. The best practice for creating a ZIL is to use mirrored flash drives on devices separate from the L2ARC, but as I only had one SSD, I opted to create a single L2ARC.

The L2ARC was created as a 20GB VMDK disk on the SSD datastore and added to the VM. The new volume was then added to the zpool as a cache device. While 20GB is not huge in terms of disk, it represents a significant amount of cache memory.

The performance advantages of the cache are not immediately obvious given that it takes time for the cache to populate. However, once data has been read, future reads will be taken from SSD instead of SATA. I've not had the chance to do meaningful benchmarks yet, but plan to do so soon.

NexentaStor CE VSA snapshots and replication

On top of the data resilience provided by the checksum, ZFS supports copy-on-write snapshots. These can be automatically scheduled on a per-filesystem basis to provide a point in time snapshot. This can be configured so document data is snapshotted daily (or hourly), while more static data such as the ISO store taken weekly or monthly.

The final step was to add even more resilience to the configuration. For this, I created a second NexentaStor CE VM on my HP ML115 G5 lab machine. This VM is smaller with only 1GB RAM. I created a 400GB disk but did not bother with mirroring. Using the NexentaStor web interface, I paired the machines and configured some scheduled jobs to replicate specific filesystems from the primary VSA to this secondary VSA (using snapshot copies over SSH). Nexenta refers to this as a "tiering service". This means that in the event the original server dies, the important data will still be available.

Overkill? Perhaps, but part of this work was to see what could be done with ZFS and the result is a very powerful storage setup.

There are a couple of concerns. One surrounds the longterm viability of ZFS given the Oracle takeover. Although NetApp have settled with Oracle, I don't know if the agreement covers other users of ZFS. Secondly, there will be a performance overhead by running the NexentaStor CE as a VSA on top of the ESXi storage subsystem. While it might be possible to squeeze a bit more performance by running NexentaStor CE directly on the bare-metal, ESXi allows me to run a few other VMs alongside the VSA. The trade-off is worth it in my mind.

In summary, NexentaStor Community Edition is a very powerful piece of software (and this post only scratches the surface - no mention of its AD integration, iSCSI functionality etc.) that gives some high-end functionality *for free* and is certainly worth considering for your home lab.

Home network update

My home network has been faithfully served by an installation of OpenSolaris on my HP ML110 G5. Unfortunately Oracle's actions towards the open source community have sadly killed what was an excellent project. While there is a small hope that the Illumos folks might get a fork organised, remaining on OpenSolaris was not practical.

To replace OpenSolaris, I would need an environment that provided all the features I was currently running. This meant I needed a CIFS and NFS server, iSCSI target server, internal DNS server, CUPS print server, private IMAP server (for my old pre-Gmail mail archive) and Windows 7 virtual machine courtesy of VirtualBox. Yeah, OpenSolaris was a *very* capable platform.

The solution I opted for was to turn the ML110 G5 into another VMware ESXi server, running a number of virtual machines the provide the above services. I would also take this opportunity to fix a couple of niggling problems with the way it was setup.

This change coincided with a number of new purchases for the home network:

• Acer Revo Aspire
• Netgear ReadyNAS Duo
• OCZ Vertex 2 SSD

The Acer Revo was bought because I was fed up with using Windows 7 over RDP on the Mac. Little issues like the backslash key not working with a UK keyboard (sounds trivial, but you try using Windows seriously without entering backslash) meant I wanted something I could connect to directly via my KVM. The Revo won't win any awards for high performance, is a capable enough machine (with 2GB RAM) and can run a number of apps (including the vSphere client and Office 2010) without any problems.

I bought the Netgear ReadyNAS Duo after looking at some of the alternatives. I was originally tempted by the Iomega ix2-200d, but was put off by the fact the filesystem is proprietary and requires you to send the unit back if there is a problem. In contrast, the ReadyNAS Duo uses Ext3, but the real dealmaker was an offer to get a second 1TB disk *for free* (via mail-in coupon). My initial playing with this unit has been positive and it's nice and quiet, but I've not spent a huge amount of time with it yet.

The OCZ Vertex 2 SSD (60GB) was purchased because I wanted to experiment with the NexentaStor [Community Edition] virtual storage appliance. Built on top of the open-sourced Solaris codebase, Nexenta have built a storage solution around ZFS. Although the SSD is pretty small for a disk, it can be added as an "L2ARC" (Level 2 Adaptive Read Cache) to boost performance. This will require a block post of it's own to detail.

Finally, although I am not pleased with the way that Oracle have gutted Sun, the Solaris operating system remains excellent and I will be using it in my work for the foreseeable future. The preview release of Solaris 11 Express demanded a look.

I'll be blogging about some of these developments in future posts, coming soon...

Building a vLab Part 6: Configuring Lab Networking

In the last post in this series, we configured our two vESXi servers to connect to an OpenFiler storage appliance. This was done by creating a dedicated vSwitch connecting to a Storage LAN. We need to now finish the network configuration to add resilience to our VM network, and add another vSwitch for vMotion.

At the moment, our vESXi network configuration looks like this:

Although we could separate our Management Network from our VM Network (and in the real world, there are some good arguments for doing this), in this vLab, we will use vSwitch0 for both of these functions. vSwitch0 also has a single vmnic which would be unacceptable in a real-world environment.

To setup vSwitch0 for VM traffic, edit the Properties and click Add. Select the Connection Type of "Virtual Machine" and complete the wizard with the default options. Back at the vSwitch0 Properties window, select the Network Adapters tab and add the unclaimed adapter that is attached to the same LAN subnet. When this is setup, close the Properties window.

Next, to setup a private vMotion network, create a new vSwitch by clicking "Add Networking" and specifying the Connection Type of VMkernel. Assign both remaining unassigned vmnic adapters to the new vSwitch. Label the network (e.g., "vMotion LAN") noting that this is case sensitive! Tick the box for "Use this port group for vMotion". Enter an IP address on a new subnet (e.g., 192.168.30.0/24). Don't worry about the VMkernel Default Gateway as the vMotion network does not need to be routable. The resulting configuration should look like this:

Repeat the network configuration on the second vESXi host.

The networking is setup and the vLab should be ready for some VMs!

Building a vLab Part 5: Configuring The Lab Storage

At this point in the vLab build, we have done the following:

• Installed ESXi on a physical host
• Created VMs for Active Directory and vCenter Server
• Created a Vyatta VM to act as a router
• Created an OpenFiler VM to act as a SAN/NAS storage array
• Created two virtual (nested) ESXi server

At this point in the build, we need to connect everything together.

In order to allow the vESXi hosts to access the "SAN", they need to be connected to the correct LAN. To do this, first note down the MAC addresses of the interfaces you have assigned to the Storage LAN. These details can be obtained by using the vSphere Client connected to the pESXi server and editing the settings of the vESXi VM:

With the MAC addresses noted, switch to the vSphere Client connected to the vCenter Server, select one of the vESXi hosts, Configuration, Network Adapters. Identify the vmnics that correspond to the Storage LAN adapters.

Staying on the Configuration screen, select Networking and Add Networking.


Specify the Connection Type as VMkernel, Create a virtual switch and select the two vmnics that are connected to the Storage LAN. Give the network a suitable label (e.g., "Storage LAN"), then assign an IP address on the storage LAN subnet (e.g., 192.168.20.101).



The end result should appear similar to the following:

Repeat this for the second vESXi host.

At this point, the hosts are ready to connect to the storage. The next step is to configure our OpenFiler "SAN/NAS appliance" and share out some storage.



Setup SAN storage

Log into the OpenFiler web interface (https://192.168.20.1:446) as the openfiler user (default password is "password"). In the original build, I added two 100GB hard disks in addition to the 8GB install disk. We will create one as an NFS share and the second as an iSCSI target.

OpenFiler is built on Linux, so an understanding of Linux LVM is useful. A very basic summary of Linux LVM:

• Physical disks are encapsulated and referred to as "Physical Volumes" (PVs)
• One of more PVs are combined together into a Volume Group (VG)
• A VG is carved up into Logical Volumes (LVs)
• A filesystem is created on an LV

Click the Volumes button
Click the link to "Create new physical volumes"
Select the first non-OS disk (/dev/sdb on my configuration)
Create a partition with the following properties:

• Mode: Primary
• Partition Type: Physical Volume

Accept the start and end cylinders and click Create
Click Volume Groups
Under the "Create a new volume group" section, enter a name (e.g., "vmware"), put a tick next to the newly created physical volume and click "Add Volume Group".

First, let's create an NFS datastore. To do this, click Add Volume.
Under "Create a volume in ", enter a name (e.g., "ds01"), a description (e.g., "NFS datastore"), select the size (e.g., 50GB) and choose a filesystem type (I used ext3). Then click Create. This creates a new Logical Volume in the "vmware" volume group that is 50GB in size and formats an ext3 filesystem onto it. The create operation may take a couple of minutes.

When this is complete, create an iSCSI datastore by clicking Add Volume again.
Enter a new name (e.g., "ds02"), a description (e.g., "iSCSI datastore"), assign all remaining space in the volume group and select the partition type as iSCSI. Then click Create. The result should look similar to this:

The OpenFiler appliance will now have the two datastores configured, but they are not published yet. Click the Services button and enable the "NFSv3 server" and the "iSCSI target server".

Click the System button and scroll down to the section titled "Network Access Control". In order for a host to see the OpenFiler storage, it needs to match an ACL entry. The most secure way to do this is to enter the IP address of each VM host. The easiest way is to specify the entire storage LAN subnet (192.168.20.0/24):

Map the iSCSI volume to the ACL by clicking the Volumes button, then select "iSCSI Targets". The system will present a new iSCSI target name. Click Add. To assign the iSCSI volume to the target, click the LUN Mapping button and click Map.

Click the Network ACL button and change the host access configuration to Allow and Update.

Share the NFS volume by clicking the Shares button. Click the NFS Datastore and create a sub-folder (e.g., "VMs"):

Click on the new sub-folder and select "Make Share". Scroll to the bottom of the new window and change the NFS setting to RW. Click the Edit button and set the UID/GID mapping to no_root_squash:

With these options set, click Update.

Finally, change Share Access Control Mode to "Public guest access" and click Update.

Switch back to the vSphere Client connected to the vCenter Server, select the first VM host, select Configure, Storage Adapters. Select the iSCSI Software Adapter (probably vmhba33) and select Properties. Click the Configure button and put a tick next Enabled to turn on iSCSI. Click OK and then select the Dynamic Discovery tab. Click Add and enter the IP address of the OpenFiler server. With iSCSI enabled and configured, click Rescan All, scanning for new storage devices and new VMFS volumes. If all is successful, you should see the new iSCSI volume appear.

Click Storage and Add Storage. Select Disk/LUN and the iSCSI disk should appear. Select it, click Next and create a new partition. Enter a datastore name (e.g., OpenFiler iSCSI) and choose a maximum file size (doesn't matter which since the disk is only small). Finish clicking through the wizard and the new datastore should appear in the list.

 Click Add Storage again. Select Network File System and click next. Enter the IP address of the OpenFiler server (192.168.20.1) with a folder path in the format of /mnt/volumegroup/logicalvolume/sharename (e.g., /mnt/vmware/ds01/VMs). Give the Datastore Name something suitable (e.g., OpenFiler NFS).

The datastore view in vSphere client should now look similar to the following:

Repeat the adding of storage on the second vESXi host. After scanning for the iSCSI storage, the datastore should appear automatically. NFS storage will still need to be entered manually.

The next step will be to finish configuring out networks and setting up vMotion...

Building a vLab Part 4: Virtual ESXi install

In the last post in this mini-series, we created a VM for vCenter Server and set it up in such a way that the install could be rapidly rebuilt in the future. The next step is to create our virtual ESXi instances (called vESXi in this blog).

When sizing the VMs, it's worth thinking of the vESXi hosts as if they were physical. For example, most physical ESXi servers will have multiple network cards, so we'll put several in our vESXi servers.

Create the virtual machine as you normally would, specifying the following parameters:

• Guest OS: Other (64-bit)
• CPU: 2 vCPU
• Memory: 4096MB

Use the datastore available to the pESXi server and not the OpenFiler VM. Accept the default virtual disk size of 8GB and tick "Allocate and commit space on demand (Thin Provisioning)".

Edit the settings of the VM and add more network adapters (remember, we need this VM to appear similar to a physical host). For the vLab, use a total of 6 network adapters in pairs for the LAN, Storage LAN and vMotion LAN respectively.

The other networking step required is to edit each vSwitch, select the port group and edit the security settings:

• Promiscuous Mode: Accept
• MAC Address Changes: Accept
• Forged Transmits: Accept

Do this for all the vSwitches.

While this is enough to get ESXi installed, you won't be able to power on any nested VMs without making the following change. When the vESXi VM is created (but not powered on), right click on the VM and select "Edit Settings". On the Options tab, select Advanced, General, Configuration Parameters. Add a new row and give it the name monitor_control.restrict_backdoor with a value of true:

Now insert/connect the ESXi install CD/ISO and power on the vESXi VM. To be honest, this next bit should be simple (especially as you've already installed the physical ESXi server).

Once installed into the VM, assign a static IP address.

Now open a second vSphere client instance and connect to the vCenter Server. Within the vCenter Server, create a new Datacenter and add a new host, pointing to the IP address of the newly created vESXi VM. When added, the new server should look similar to this:

There's not much point in having a lab with only a single server, so repeat the process detailed in this post and create a second VM. In the next post, we'll configure it all up...

Building a vLab Part 3: vCenter Server

Previously on "Building a vLab": Part 1: The Design and Part 2: Infrastructure Build.

For a production environment, many people run vCenter on a server that connects to a SQL Server database on another server (possibly as part of a cluster).  However, as part of this vLab, we're going for the default install of a single VM using a local SQL Server Express database.

The vCenter VM has 1 vCPU, 4GB RAM, 40GB hard disk, 1 vNIC connecting to the vLab LAN with an IP address of 192.168.10.2/24. This specification is smaller than that recommended by VMware, but it's enough to get started with. The vCenter server is running Windows Server 2008 R2 as vCenter 4.1 requires a 64bit version of Windows.

Once built, the vCenter Server is assigned the default gateway of the Vyatta VM (192.168.10.33) and the DNS server of the domain controller. The vCenter Server is named "vcenter" and then joined to the vLab domain.

As I do not have permanent VMware vSphere licences in my home lab, I wanted to create an environment where rebuilding from scratch would be a fairly painless experience.

I first created a new 64bit Windows Server 2008 R2 virtual machine. After it was assigned a static IP address and given the correct hostname, I created a small command script called install-vcenter.cmd based on the VMware "Performing a Command-Line Installation of vCenter Server" and copied it to the administrator's desktop.

Having got the base Windows install done, I then exported the VM as an OVF template for future use.

The next step was to mount the vCenter ISO and run the install-vcenter.cmd script. This performed a silent default installation of vCenter including the installation of the .NET runtime and SQL Server Express install. There are many customisable options that can be passed to the setup but these work well enough for my needs:

set EXE=D:\vpx\VMware-vcserver.exe
start /wait %EXE% /q /s /w /L1033 /v" /qr DB_SERVER_TYPE=Bundled FORMAT_DB=1 /L*v \"%TEMP%\vmvcsvr.txt\""

This means that when the vCenter licence expires, I can wipe it out, re-import the template VM, rejoin the domain and run the install-vcenter.cmd script to rebuild a new vCenter installation. It won't keep all my previous settings, and won't configure all the VMs, but it's a start.

UPDATE (22 Jan 2011): If the lab isn't used for a couple of months, the Active Directory trust relationship will fail and the installation will fail. To fix, export the VM when built, but before it is on the domain. I then wrote a short cmd file to automatically join the domain with the following command:

netdom join %computername% /Domain:VSPHERE /UserD:Administrator /PasswordD:mypassword /REBoot:20

In the next part of this series, we'll build our virtual ESXi servers.