The new Microsoft iSCSI Software Target 3.3 includes support for a new feature called differencing virtual hard disks (VHDs). This feature helps deploy diskless boot for servers running Windows, especially in a Windows HPC Server 2008 R2 compute cluster environment.

We worked closely with the HPC team to deliver a simple management experience. The deployment process is tightly integrated within the Microsoft HPC management pack, which manages the iSCSI target using an HPC provider. Support for differencing VHDs and iSCSI boot in iSCSI Software Target 3.3 is useful in other deployments beyond HPC and this post will focus on how to deploy iSCSI boot outside of the HPC environment. You can get more details about how this works in this blog post.

Advantages of using diskless boot include:

1. Fast deployment: A single golden image can be used to deploy many machines in parallel. At Microsoft we tested deployments as large as 256 nodes in 30 minutes.
2. Quick recovery: Since the operating system image is not on the server, you could simply replace the blade and point to the old remote VHD, and boot from it. No operating system installation is required.
3. Cost reduction: Because many servers can boot off of a single image, CAPEX and OPEX are directly reduced by having lower storage requirements for the operating system images. This also reduces the power, cooling, and space required for the storage.

This post will explain how diskless boot works and how you can try it out yourself!

Terminology

Before going any further, let’s clarify a few terms:

Client: The server that will boot from the image stored on the iSCSI target server. It can also be referred as an iSCSI initiator or a “diskless node”. Note: Since iSCSI boot is only supported on Windows Server SKU’s, the term “Client” refers to the iSCSI client machine which runs Windows Server OS.

Golden image: A generalized (sysprep’d) image containing an operating system and any other required applications. It is prepared on an iSCSI LUN and set to be a read-only master image. A section below describes one way to create the golden image.

HPC: High-performance computing (see here).

iSCSI target: The end point where the iSCSI initiator will establish the connection. Once the initiator connects to the iSCSI target, all the VHDs associated with that target will be accessible to that initiator.

iSCSI Software Target: Software application which provides iSCSI storage to clients (iSCSI initiators).

Differencing VHD: One of the supported VHD formats used by Microsoft iSCSI Software Target. (For a definition, please see here.) When using diskless boot, the clients will read from the golden image and write to a differencing VHD. This allows multiple clients to boot off of the same golden image.

Boot loader: Refers to the software that can bootstrap the operating system. In the iSCSI boot scenario, it contains a basic iSCSI initiator that can connect to the iSCSI target and mount a disk. It is an alternative solution to an iSCSI boot-capable NIC/HBA.

Overview

There are two phases when managing diskless clients:

1. Deployment: Enabling a diskless physical server (bare metal) to boot Windows, involves:
• Preparing a golden image. If you are planning to reuse the image, you can store it in a folder, and copy it out later when you do the deployment.
• Create a one or more differencing VHDs, and set the golden image as the parent VHD
• Create an iSCSI target for each differencing VHD and assign the target to the client.
• Power on the client.
• Final operating system customization – This is the Windows setup process. It can be automated using an unattended setup file.
  
2. Boot process: This is the normal machine boot process, that is handled by the iSCSI boot capable hardware or software. (For diagrams on how the boot process works, see the boot process sections below).

See this TechNet article on iSCSI Boot which covers the more information on this topic. It includes a step-by-step guide for deployment using options 1 & 2 mentioned below.

Hardware/Software options

You will need one of the following options to enable iSCSI boot on a physical client:

1. An iSCSI boot-capable NIC or
2. An iSCSI boot-capable HBA or
3. A software-based iSCSI boot loader

An iSCSI boot-capable NIC

The NIC needs to be configured in the server BIOS to provide the iSCSI target IQN, iSCSI target IP address, and credentials if set (examples of this type include Intel or Broadcom).

Figure 1 - iSCSI Boot capable NIC configuration

An iSCSI boot-capable HBA

An iSCSI boot capable HBA requires similar information to the iSCSI NIC. The follow example shows a QLogic HBA.

Figure 2 - iSCSI boot capable HBA configuration

An iSCSI boot loader

A software boot loader needs more configuration and details are in the workflow section of this post. Examples of a boot loader are gPXE (open source) and Doubletake Netboot/I (commercial). This post will use the gPXE as an example. You can also use iPXE instead of gPXE.

Boot Process

There are two stages during machine boot up:

1. Pre-boot
2. Windows boot

The pre-boot phase can be executed by any of the three options described above (NIC, HBA, or software boot loader).

• For hardware, the boot loader is built in the firmware of the NICs or HBAs, and it can connect directly to the iSCSI Target and mounted to the assigned disk containing the Windows operating system image.
• With a software boot loader, you can put the boot loader image on a CD or a USB drive, and have the computer boot to the device. Once the computer logs on to the iSCSI target, it enters the Windows boot phase.

The following diagram describes the components involved on the client computer during the boot process:

Figure 3 - Boot process components

When the pre-boot loader starts, it loads a real-mode network stack driver. The loader contains an iSCSI initiator which logs on to the iSCSI target and mount the boot disk to the system. Another function of the loader is to populate the iSCSI Boot Firmware Table (iBFT), which is required for iSCSI boot. The Boot Parameter driver in Windows will load the parameters from the iBFT, and the Microsoft iSCSI Software Initiator will be able to connect to the iSCSI target using the parameters set in the iBFT. The importance of the iBFT is to be able to share the parameters between the iSCSI boot initiator (which establishes the session in the preboot phase) and the Microsoft iSCSI initiator (which establish the session after Windows boots).

Various deployment configurations

If you are using hardware NICs or HBAs, please visit this TechNet page for a different approach for deployment.

Once you have a hardware or software solution selected, you need to think about whether you want to leverage DHCP and TFTP server for the deployment:

Using an iSCSI boot-capable NIC or HBA

Using a hardware solution is easy to deploy, you just configure the firmware with the iSCSI target IQN and IP address (see Figure 1 above), then power up the client. The basic iSCSI initiator built into the firmware will be able to connect to the target and mount the VHDs.

Since it doesn’t use a software boot loader, it doesn’t need additional infrastructure such as TFTP server in the deployment. The downside is that you need to acquire the hardware, and configure each one manually. To find out how to automate the configuration, you will need to contact the hardware vendor.

Required hardware:

• Client machine with iSCSI boot-capable NIC or HBA
• Server running the iSCSI Software Target

The boot process

Once the client NIC has been configured, the machine will boot as illustrated below:

Figure 4 - Boot process using iSCSI Boot capable hardware

1. When the client machine boots up, it reads the Target IP and IQN information, and the iSCSI NIC/HBA connects to the iSCSI target.
2. The iSCSI Target accepts the connection, and presents a VHD as a disk to the client. The disk is then mounted on the client.
3. The boot process proceeds as if the image resides on a local disk. Once Windows starts up, it starts the Microsoft iSCSI initiator which uses the parameters specified in the iBFT table to connect to the target. You can also find more about iBFT here.

Using a Software boot loader with DHCP and TFTP servers

A TFTP server hosts the boot loader in a central location. You can configure a DHCP server to provide each client machine a unique target. When a client boots up, it acquires the IP, TFTP server location, as well as information about the target to connect to from the DHCP server. Using DHCP in combination with the TFTP server will reduce the management operations. This is the approach used in Windows HPC diskless cluster deployments. The HPC management pack guides users to configure the DHCP and TFTP servers in a simplified manner. This configuration can scale higher with automation support. The hardest part is to configure the DHCP server. The configuration needs to adhere to the software loader specification. To get more information on how to configure the DHCP server, please visit here.

Required hardware:

• Client machine with a NIC that supports PXE boot
• Server with the iSCSI Software Target installed
• Server with the DHCP and Windows Deployment Services roles installed (WDS contains the TFTP server feature)

The boot process

Once the configuration of the DHCP server is done, and the boot loader has been copied to the TFTP server, the machine boot sequence follows this process:

Figure 5 - Boot process when using DHCP, TFTP server with software boot loader

1. The client machine is powered up with PXE boot enabled. It requests the IP address, the TFTP server location, as well as iSCSI target connection information, from the DHCP server. (Note, most computers today support PXE boot.)
2. The DHCP server responds with all the information.
3. The client machine contacts the TFTP service for the boot loader.
4. The boot loader is copied to the client machine which has a basic iSCSI boot initiator.
5. The client machine uses the basic iSCSI boot initiator to log on to the iSCSI target. Once the connection is established, the disk is mounted on the client machine.
6. The boot process proceeds as if the boot image is on a local disk. Once Windows starts up, it loads the Microsoft iSCSI initiator which uses the parameters specified in the iBFT table to connect to the target.
   

Using a Software Boot Loader without DHCP and TFTP

In a test environment, it may not be feasible to configure a DHCP and TFTP server. Without a TFTP server, you need to find another way to get the boot loader image presented to the client machines. You can put the boot loader on a USB drive or a CD in the client machine. This is the simplest way to try out and does not require any extra hardware investment. The step-by-step guide below focuses on this configuration.

Required hardware:

• A client machine with a NIC.
• A device (CD/USB, etc.) that can store the software boot loader
• A server with the iSCSI Software Target installed

The boot process

Once the software boot loader has been configured, the machine boot will follow this process:

Figure 6 – Boot process for local software boot loader

1. Set the client BIOS boot order to use the USB/or CD where the boot loader is stored.
2. When the client machine boots, the boot loader uses the iSCSI target IP and IQN information, and connects to the iSCSI target.
3. The iSCSI target accepts the connection and presents the VHD to the client. At this point the disk will be mounted on the client.
4. The boot process proceeds as if the boot image resides on a local disk. Once Windows starts up, it loads the Microsoft iSCSI initiator. The iSCSI initiator will use the parameters specified in the iBFT table to connect to the target.

Step-by-step deployment guide

Here is the recipe for using a software boot loader on a CD/USB.

There are two stages:

1. Create a golden image
2. Deploy diskless clients using a golden image

After stage 2, the client machine will be ready to run a workload. You can use the machine just like one that has an OS image stored locally.

A word of caution: While the iSCSI target is servicing diskless clients, rebooting the iSCSI target server will cause unexpected behavior for the diskless clients. This is similar to removing the local hard drive while the machine is running. If you require high availability in your environment, you should consider deploying a failover cluster with the iSCSI targets LUNs.

Create a Golden Image

To create a golden image, you need to do a normal OS and application install on the iSCSI VHD, to ensure the setup process can load the driver stacks in the correct order. This is important, because the Windows setup process goes through different path for installing the OS on local hard drive vs on a iSCSI disk. The image installed on a local hard drive can not be used for iSCSI boot, as the drives loading sequences will be different.

Once the installation is complete, run sysprep on the OS image.

Steps to create a golden image:

On the iSCSI target server:

• Create a 30GB fixed VHD (big enough to install the OS and applications).
• Create an iSCSI target.
• Assign the VHD to the target.
• Assign the target to the client iSCSI IQN.
  

On the client machine:

• Configure the boot loader:

• For iSCSI boot NICs or HBAs: configure the NIC or HBA to point to the correct target.

• For software boot loaders: Configure it if DHCP or TFTP is not available.
  You can get a copy of a software boot loader (gPXE or iPXE) from : http://etherboot.org/wiki/index.php or http://ipxe.org/download
  The image needs to be stored on CD/DVD or USB with the following custom script. You can use the Rom-o-Matic.net web site to generate custom boot images, and copy the images to the media. www.etherboot.com has a list of tools you can use to copy images onto various media.

• Set the boot order to boot from gPXE media.
• Set the system BIOS boot order to boot from OS installation media second.
• Boot the disk client.
• gPXE will fail because there is no OS to load, but the iSCSI Connection will be established. (See the scripts below for an explanation.)
• Choose the OS installation media to boot (this leads to a normal OS setup).
• Select the iSCSI LUN to install the OS and finish OS installation normally.
• Now you have an OS installation on an iSCSI LUN.
• Optional: Install additional applications which you want in the sysprep image.
• Sysprep the image of the first client machine. (Find more details on sysprep here.)
• Use the syspreped image as the parent VHD for future deployment

(Note, I used this page as reference when I did my setup. You may find it useful as well.)

Custom Script

dhcp net0
set initiator-iqn iqn.1991-05.com.microsoft:iscsiboot-${net0/mac}
set root-path iscsi:10.121.28.150::::iqn.1991-05.com.microsoft:testsvr -iscsiboot-${net0/mac}-target
set keep-san 1
sanboot ${root-path}

How to customize the boot loader script

The above script needs to be customized to your configuration:

• Line 1: Assume booting from first NIC
• Line 3:
• Replace “10.121.28.150” with the actual iSCSI Target server address.
• Replace “testsvr” with the actual iSCSI Target server host name.
• Use IPv4, since the current version of the gPXE doesn’t support IPv6.
• gPXE will use the standard iSCSI TCP port 3260 and LUN 0.
• Line 4 : The command “set keep-san 1” means keep iSCSI connection when failure occurs. By setting this, you will be able to install the OS image onto the iSCSI LUN when creating the golden image.

Deploy diskless clients using a Golden image

Once the golden image is prepared, you can use it to deploy more clients by following the steps below:

• On the target server:
• Create a differencing VHD using WMI scripts (see this sample), and specify the base VHD which contains the golden image which is the sysprep’d OS images created from the above section.
• Create an iSCSI target, and assign it to the client.
• Assign the differencing VHD to the target.
  
• On the client machine:
• Boot the client machine. Because the base OS contains the image, it will load Windows, and continue with the OS finalization phase, where all of the unique information is saved to the differencing VHD.
• P.S. Now you are done, enjoy your new diskless servers!