Home network update

My home network has been faithfully served by an installation of OpenSolaris on my HP ML110 G5. Unfortunately Oracle's actions towards the open source community have sadly killed what was an excellent project. While there is a small hope that the Illumos folks might get a fork organised, remaining on OpenSolaris was not practical.

To replace OpenSolaris, I would need an environment that provided all the features I was currently running. This meant I needed a CIFS and NFS server, iSCSI target server, internal DNS server, CUPS print server, private IMAP server (for my old pre-Gmail mail archive) and Windows 7 virtual machine courtesy of VirtualBox. Yeah, OpenSolaris was a *very* capable platform.

The solution I opted for was to turn the ML110 G5 into another VMware ESXi server, running a number of virtual machines the provide the above services. I would also take this opportunity to fix a couple of niggling problems with the way it was setup.

This change coincided with a number of new purchases for the home network:

• Acer Revo Aspire
• Netgear ReadyNAS Duo
• OCZ Vertex 2 SSD

The Acer Revo was bought because I was fed up with using Windows 7 over RDP on the Mac. Little issues like the backslash key not working with a UK keyboard (sounds trivial, but you try using Windows seriously without entering backslash) meant I wanted something I could connect to directly via my KVM. The Revo won't win any awards for high performance, is a capable enough machine (with 2GB RAM) and can run a number of apps (including the vSphere client and Office 2010) without any problems.

I bought the Netgear ReadyNAS Duo after looking at some of the alternatives. I was originally tempted by the Iomega ix2-200d, but was put off by the fact the filesystem is proprietary and requires you to send the unit back if there is a problem. In contrast, the ReadyNAS Duo uses Ext3, but the real dealmaker was an offer to get a second 1TB disk *for free* (via mail-in coupon). My initial playing with this unit has been positive and it's nice and quiet, but I've not spent a huge amount of time with it yet.

The OCZ Vertex 2 SSD (60GB) was purchased because I wanted to experiment with the NexentaStor [Community Edition] virtual storage appliance. Built on top of the open-sourced Solaris codebase, Nexenta have built a storage solution around ZFS. Although the SSD is pretty small for a disk, it can be added as an "L2ARC" (Level 2 Adaptive Read Cache) to boost performance. This will require a block post of it's own to detail.

Finally, although I am not pleased with the way that Oracle have gutted Sun, the Solaris operating system remains excellent and I will be using it in my work for the foreseeable future. The preview release of Solaris 11 Express demanded a look.

I'll be blogging about some of these developments in future posts, coming soon...

Building a vLab Part 6: Configuring Lab Networking

In the last post in this series, we configured our two vESXi servers to connect to an OpenFiler storage appliance. This was done by creating a dedicated vSwitch connecting to a Storage LAN. We need to now finish the network configuration to add resilience to our VM network, and add another vSwitch for vMotion.

At the moment, our vESXi network configuration looks like this:

Although we could separate our Management Network from our VM Network (and in the real world, there are some good arguments for doing this), in this vLab, we will use vSwitch0 for both of these functions. vSwitch0 also has a single vmnic which would be unacceptable in a real-world environment.

To setup vSwitch0 for VM traffic, edit the Properties and click Add. Select the Connection Type of "Virtual Machine" and complete the wizard with the default options. Back at the vSwitch0 Properties window, select the Network Adapters tab and add the unclaimed adapter that is attached to the same LAN subnet. When this is setup, close the Properties window.

Next, to setup a private vMotion network, create a new vSwitch by clicking "Add Networking" and specifying the Connection Type of VMkernel. Assign both remaining unassigned vmnic adapters to the new vSwitch. Label the network (e.g., "vMotion LAN") noting that this is case sensitive! Tick the box for "Use this port group for vMotion". Enter an IP address on a new subnet (e.g., 192.168.30.0/24). Don't worry about the VMkernel Default Gateway as the vMotion network does not need to be routable. The resulting configuration should look like this:

Repeat the network configuration on the second vESXi host.

The networking is setup and the vLab should be ready for some VMs!

Building a vLab Part 5: Configuring The Lab Storage

At this point in the vLab build, we have done the following:

• Installed ESXi on a physical host
• Created VMs for Active Directory and vCenter Server
• Created a Vyatta VM to act as a router
• Created an OpenFiler VM to act as a SAN/NAS storage array
• Created two virtual (nested) ESXi server

At this point in the build, we need to connect everything together.

In order to allow the vESXi hosts to access the "SAN", they need to be connected to the correct LAN. To do this, first note down the MAC addresses of the interfaces you have assigned to the Storage LAN. These details can be obtained by using the vSphere Client connected to the pESXi server and editing the settings of the vESXi VM:

With the MAC addresses noted, switch to the vSphere Client connected to the vCenter Server, select one of the vESXi hosts, Configuration, Network Adapters. Identify the vmnics that correspond to the Storage LAN adapters.

Staying on the Configuration screen, select Networking and Add Networking.


Specify the Connection Type as VMkernel, Create a virtual switch and select the two vmnics that are connected to the Storage LAN. Give the network a suitable label (e.g., "Storage LAN"), then assign an IP address on the storage LAN subnet (e.g., 192.168.20.101).



The end result should appear similar to the following:

Repeat this for the second vESXi host.

At this point, the hosts are ready to connect to the storage. The next step is to configure our OpenFiler "SAN/NAS appliance" and share out some storage.



Setup SAN storage

Log into the OpenFiler web interface (https://192.168.20.1:446) as the openfiler user (default password is "password"). In the original build, I added two 100GB hard disks in addition to the 8GB install disk. We will create one as an NFS share and the second as an iSCSI target.

OpenFiler is built on Linux, so an understanding of Linux LVM is useful. A very basic summary of Linux LVM:

• Physical disks are encapsulated and referred to as "Physical Volumes" (PVs)
• One of more PVs are combined together into a Volume Group (VG)
• A VG is carved up into Logical Volumes (LVs)
• A filesystem is created on an LV

Click the Volumes button
Click the link to "Create new physical volumes"
Select the first non-OS disk (/dev/sdb on my configuration)
Create a partition with the following properties:

• Mode: Primary
• Partition Type: Physical Volume

Accept the start and end cylinders and click Create
Click Volume Groups
Under the "Create a new volume group" section, enter a name (e.g., "vmware"), put a tick next to the newly created physical volume and click "Add Volume Group".

First, let's create an NFS datastore. To do this, click Add Volume.
Under "Create a volume in ", enter a name (e.g., "ds01"), a description (e.g., "NFS datastore"), select the size (e.g., 50GB) and choose a filesystem type (I used ext3). Then click Create. This creates a new Logical Volume in the "vmware" volume group that is 50GB in size and formats an ext3 filesystem onto it. The create operation may take a couple of minutes.

When this is complete, create an iSCSI datastore by clicking Add Volume again.
Enter a new name (e.g., "ds02"), a description (e.g., "iSCSI datastore"), assign all remaining space in the volume group and select the partition type as iSCSI. Then click Create. The result should look similar to this:

The OpenFiler appliance will now have the two datastores configured, but they are not published yet. Click the Services button and enable the "NFSv3 server" and the "iSCSI target server".

Click the System button and scroll down to the section titled "Network Access Control". In order for a host to see the OpenFiler storage, it needs to match an ACL entry. The most secure way to do this is to enter the IP address of each VM host. The easiest way is to specify the entire storage LAN subnet (192.168.20.0/24):

Map the iSCSI volume to the ACL by clicking the Volumes button, then select "iSCSI Targets". The system will present a new iSCSI target name. Click Add. To assign the iSCSI volume to the target, click the LUN Mapping button and click Map.

Click the Network ACL button and change the host access configuration to Allow and Update.

Share the NFS volume by clicking the Shares button. Click the NFS Datastore and create a sub-folder (e.g., "VMs"):

Click on the new sub-folder and select "Make Share". Scroll to the bottom of the new window and change the NFS setting to RW. Click the Edit button and set the UID/GID mapping to no_root_squash:

With these options set, click Update.

Finally, change Share Access Control Mode to "Public guest access" and click Update.

Switch back to the vSphere Client connected to the vCenter Server, select the first VM host, select Configure, Storage Adapters. Select the iSCSI Software Adapter (probably vmhba33) and select Properties. Click the Configure button and put a tick next Enabled to turn on iSCSI. Click OK and then select the Dynamic Discovery tab. Click Add and enter the IP address of the OpenFiler server. With iSCSI enabled and configured, click Rescan All, scanning for new storage devices and new VMFS volumes. If all is successful, you should see the new iSCSI volume appear.

Click Storage and Add Storage. Select Disk/LUN and the iSCSI disk should appear. Select it, click Next and create a new partition. Enter a datastore name (e.g., OpenFiler iSCSI) and choose a maximum file size (doesn't matter which since the disk is only small). Finish clicking through the wizard and the new datastore should appear in the list.

 Click Add Storage again. Select Network File System and click next. Enter the IP address of the OpenFiler server (192.168.20.1) with a folder path in the format of /mnt/volumegroup/logicalvolume/sharename (e.g., /mnt/vmware/ds01/VMs). Give the Datastore Name something suitable (e.g., OpenFiler NFS).

The datastore view in vSphere client should now look similar to the following:

Repeat the adding of storage on the second vESXi host. After scanning for the iSCSI storage, the datastore should appear automatically. NFS storage will still need to be entered manually.

The next step will be to finish configuring out networks and setting up vMotion...

Building a vLab Part 4: Virtual ESXi install

In the last post in this mini-series, we created a VM for vCenter Server and set it up in such a way that the install could be rapidly rebuilt in the future. The next step is to create our virtual ESXi instances (called vESXi in this blog).

When sizing the VMs, it's worth thinking of the vESXi hosts as if they were physical. For example, most physical ESXi servers will have multiple network cards, so we'll put several in our vESXi servers.

Create the virtual machine as you normally would, specifying the following parameters:

• Guest OS: Other (64-bit)
• CPU: 2 vCPU
• Memory: 4096MB

Use the datastore available to the pESXi server and not the OpenFiler VM. Accept the default virtual disk size of 8GB and tick "Allocate and commit space on demand (Thin Provisioning)".

Edit the settings of the VM and add more network adapters (remember, we need this VM to appear similar to a physical host). For the vLab, use a total of 6 network adapters in pairs for the LAN, Storage LAN and vMotion LAN respectively.

The other networking step required is to edit each vSwitch, select the port group and edit the security settings:

• Promiscuous Mode: Accept
• MAC Address Changes: Accept
• Forged Transmits: Accept

Do this for all the vSwitches.

While this is enough to get ESXi installed, you won't be able to power on any nested VMs without making the following change. When the vESXi VM is created (but not powered on), right click on the VM and select "Edit Settings". On the Options tab, select Advanced, General, Configuration Parameters. Add a new row and give it the name monitor_control.restrict_backdoor with a value of true:

Now insert/connect the ESXi install CD/ISO and power on the vESXi VM. To be honest, this next bit should be simple (especially as you've already installed the physical ESXi server).

Once installed into the VM, assign a static IP address.

Now open a second vSphere client instance and connect to the vCenter Server. Within the vCenter Server, create a new Datacenter and add a new host, pointing to the IP address of the newly created vESXi VM. When added, the new server should look similar to this:

There's not much point in having a lab with only a single server, so repeat the process detailed in this post and create a second VM. In the next post, we'll configure it all up...

Building a vLab Part 3: vCenter Server

Previously on "Building a vLab": Part 1: The Design and Part 2: Infrastructure Build.

For a production environment, many people run vCenter on a server that connects to a SQL Server database on another server (possibly as part of a cluster).  However, as part of this vLab, we're going for the default install of a single VM using a local SQL Server Express database.

The vCenter VM has 1 vCPU, 4GB RAM, 40GB hard disk, 1 vNIC connecting to the vLab LAN with an IP address of 192.168.10.2/24. This specification is smaller than that recommended by VMware, but it's enough to get started with. The vCenter server is running Windows Server 2008 R2 as vCenter 4.1 requires a 64bit version of Windows.

Once built, the vCenter Server is assigned the default gateway of the Vyatta VM (192.168.10.33) and the DNS server of the domain controller. The vCenter Server is named "vcenter" and then joined to the vLab domain.

As I do not have permanent VMware vSphere licences in my home lab, I wanted to create an environment where rebuilding from scratch would be a fairly painless experience.

I first created a new 64bit Windows Server 2008 R2 virtual machine. After it was assigned a static IP address and given the correct hostname, I created a small command script called install-vcenter.cmd based on the VMware "Performing a Command-Line Installation of vCenter Server" and copied it to the administrator's desktop.

Having got the base Windows install done, I then exported the VM as an OVF template for future use.

The next step was to mount the vCenter ISO and run the install-vcenter.cmd script. This performed a silent default installation of vCenter including the installation of the .NET runtime and SQL Server Express install. There are many customisable options that can be passed to the setup but these work well enough for my needs:

set EXE=D:\vpx\VMware-vcserver.exe
start /wait %EXE% /q /s /w /L1033 /v" /qr DB_SERVER_TYPE=Bundled FORMAT_DB=1 /L*v \"%TEMP%\vmvcsvr.txt\""

This means that when the vCenter licence expires, I can wipe it out, re-import the template VM, rejoin the domain and run the install-vcenter.cmd script to rebuild a new vCenter installation. It won't keep all my previous settings, and won't configure all the VMs, but it's a start.

UPDATE (22 Jan 2011): If the lab isn't used for a couple of months, the Active Directory trust relationship will fail and the installation will fail. To fix, export the VM when built, but before it is on the domain. I then wrote a short cmd file to automatically join the domain with the following command:

netdom join %computername% /Domain:VSPHERE /UserD:Administrator /PasswordD:mypassword /REBoot:20

In the next part of this series, we'll build our virtual ESXi servers.

Building a vLab Part 2: Infrastructure Build

The journey begins! In order to build the vLab as detailed in part one, I'll be using my HP ML115 G5. This is a quad core, single CPU tower server in which I've installed 8GB RAM. It's also got 2 x 500GB SATA drives of which I'll be using one for the vLab environment (the other will be used for other projects). The ML115 G5 has an internal USB socket and ESXi can easily be installed on it, reserving the disk space for the VMs.

There is little point in recreating the same installation instructions over and over again when there is a perfectly good reference point. In this case, I'm using the excellent "Installing VMware ESXi 4.0 on a USB Memory Stick The Official Way" post from TechHead (the install for 4.1 is pretty much the same).

As the only physical VMware server, I'll be referring to it as the pESXi box.

At the end of the install process, I have an empty VM host connected to my physical network on which to build the VMs that represent the "physical" items in my virtual infrastructure: The router and the SAN/NAS and the Active Domain controller that we'll need when vCenter Server is installed.


Building the LAN


VMware best practice is to use multiple networks for different types of traffic. The vLab will require four different VLANs (virtual machine network traffic, vMotion traffic, IP storage and access to the physical network). In order to enable this, the pESXi server needs four switches. All four of these switches contain "Virtual Machine" port groups. The switch containing the management interface to the pESXi box also has a VMkernel port.

• vSwitch0: Connects to the physical (non-vLab) network
• vSwitch1: The vLab LAN for management access and connecting VMs
• vSwitch2: The vLab storage network for iSCSI and/or NFS traffic
• vSwitch3: The vLab vMotion network

On the pESXi box, the networks look as follows:

You will notice that vSwitches1-3 are not connected to any physical adapter yet. We will use the Vyatta router to accomplish this.

For reference, I'll be using the following subnets:

• 192.168.192.0/24 - main network connected to the physical network
• 192.168.10.0/24 - vLab Virtual Machine network
• 192.168.20.0/24 - vLab storage network
• 192.168.30.0/24 - vLab vMotion network
  

Add the routes to the network on your management PC. Alternatively, it might be more useful to add these on your main router so that traffic to these networks route correctly.

The router is necessary because I want to give my vLab a completely separate IP range to the rest of my kit. Therefore, in order for my non-lab kit to communicate with the vLab, I need a layer 3 router. Vyatta have a free "Core" edition that can be installed. For this, I created a new VM with the following sizings:

• 1 vCPU
• 256MB RAM
• 8GB Hard Disk (thin provisioned)
• 3 x Network Interfaces
• 1 connecting to the default VM network (i.e., the physical LAN)
• 1 connecting to the "vSphere Lab in a Box LAN"
• 1 connecting to the "vSphere Lab in a Box Storage LAN"

For information on installing Vyatta, see this guide. My Vyatta configuration (as displayed using the show -all command) is:

 interfaces {
     ethernet eth0 {
         address 192.168.192.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:5b
         speed auto
     }
     ethernet eth1 {
         address 192.168.10.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:65
         speed auto
     }
     ethernet eth2 {
         address 192.168.20.33/24
         duplex auto
         hw-id 00:0c:29:50:0a:6f
         speed auto
     }
     loopback lo {
     }
 }
 system {
     gateway-address 192.168.192.1
     host-name vyatta
     login {
         user root {
             authentication {
                 encrypted-password $1$VBYqK71jAsu3bsoAznh22mx0pqp31nU/
             }
             level admin
         }
         user vyatta {
             authentication {
                 encrypted-password $1$FdjsdebjGneXOIVw9exHrXRAcaN.
             }
             level admin
         }
     }
     ntp-server 69.59.150.135
     package {
         auto-sync 1
         repository community {
             components main
             distribution stable
             password ""
             url http://packages.vyatta.com/vyatta
             username ""
         }
     }
     time-zone GMT
 }

I mapped the Vyatta ethernet adapters (eth0, eth1 and eth2) to the correct network by comparing the MAC address with those listed against each adapter in the vSphere client. The default route (pointing to my Internet router) will make things like downloading software from within the vLab easier.

Building the SAN

VMware vSphere shines when the hosts have access to shared storage. The vLab ESXi servers will connect to an IP based (iSCSI) SAN. There are multiple ways to achieve this and one of the most common ways for home lab users to get shared storage is to use the Linux-based OpenFiler distribution.

Again, in an attempt to avoid reinventing the wheel, I'll point to the excellent Techhead post on configuring OpenFiler.

The specifics for the vLab is that the OpenFiler VM is connected to a the vLab storage LAN and not the vLab VM LAN. The IP address for the OpenFiler VM is 192.168.20.1. In addition to the install disk of 8GB, I've also created a 100GB thin provisioned disk on which to install VMs. The OpenFiler storage will be used for the VMs that I'll install on the virtual ESXi servers. The Active Directory Domain Controller, Vyatta VM and vCenter Server will also install directly onto the 500GB SATA datastore.


Installing the Active Directory Domain Controller

VMware vCenter Server requires Active Directory, so we'll need a domain controller for the vLab. Best practice requires at least two domain controllers for resilience, but we'll make do with just the one (this is a VM lab not a Windows lab). I sized the DC VM to be very small: 1 vCPU with 256MB RAM, 8GB hard disk and 1 vNIC connecting to the vLab LAN with an IP address of 192.168.10.1/24. Although I prefer Windows Server 2008, the DC will run Windows Server 2003 because of it's lower footprint.

The setup was a standard Windows Server 2003 install, followed by running dcpromo. I called the host "labdc", the domain "vsphere.lab" and we're off.

Okay, so we have everything ready apart from our virtual ESXi hosts and the vCenter server. We'll continue the journey in part 3.

Building a vLab Part 1: The Design

Like many in the VMware community, having a home lab on which to try things out is something I've been working on for some time. As I prepared to update my VCP3 to the VCP4, I thought it would be good to build myself a new "vLab" to test - and break - things without worrying about production systems and complaining users.

This mini series is partly inspired by a posting on TechHead's blog. I think there was originally going to be a series there, but for whatever reason (I'm guessing time was the issue), it didn't really develop.

In order to be able to run through the VCP4 syllabus, I needed the following:

• 2 x ESX servers
• 1 x VMware vCenter server
• 1 x Active Directory Domain Controller
• 1 x SAN/NAS shared storage array
• 1 x router/firewall that isolates the vLab from the rest of my home network

This is what I'm aiming for:




Obviously this is going to take up a fair amount of space, will be noisy and hot. So the approach I'm going to aim for is to run the entire vLab on a single HP ML115 G5 with 1 x quad core CPU and 8GB RAM. I'll run VMware ESXi as the base hypervisor and then install the following VMs:

• 1 x Windows Server 2008 R2 VM to run vCenter Server (4.1 requires a 64bit OS)
• 1 x Windows Server 2003 VM to run as an Active Directory Domain Controller
• 1 x OpenFiler VM to run as an iSCSI and NFS server
• 1 x Vyatta VM to run as a router/firewall
• 2 x VMware vSphere Hypervisor (ESXi) VMs to run the lab VMs

For the Windows Server 2003/2008 licences I'll use my Technet subscription and the OpenFiler and Vyatta installs are free. For the vCenter Server and enterprise features of vSphere, I'll have to use the evaluation licences.

With the software components downloaded and ready to do, it's time to do the build...http://livingonthecloud.blogspot.com/2010/10/building-vlab-part-2-infrastructure.html