VIRTUALDESKTOPBLOG

Sometimes it's usefull to set a custum environment variable on a system (for example a XenApp silo server). You can use this variable in scripts and applications.
To flexibily set an environment variable at server startup you can use Windows Group Policy. In order to do this you need to create a custom ADM template.

This template looks something like this:

CLASS MACHINE

CATEGORY "Custom Settings"
CATEGORY "Systeem Variables"

       POLICY "Silo Variable"
             KEYNAME "SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
                  PART "Silo" DROPDOWNLIST
                  VALUENAME "Silo"
                         ITEMLIST
                         NAME "Desktop"
                         VALUE "Desktop"
                         NAME "Apps"
                         VALUE "Apps"
                         NAME "Support"
                         VALUE "Support"
                         END ITEMLIST
                 END PART
            END POLICY ; Silo Variable

       POLICY "Application UNC Path"
            KEYNAME "SYSTEM\CurrentControlSet\Control\Session Manager\Environment"
            PART "UNC Path"
                 EDITTEXT
                 VALUENAME "APP_Path"
            END PART
       END POLICY ; Application UNC Path
  
END CATEGORY ; Systeem Variables
END CATEGORY ; "Custom Settings"

The first GPO settings contains a dropdownlist in which the value for the silo variable can be selected. The second setting just contains a text string in which a UNC path can be inserted.

In the Group Policy Management Console this looks like this:

At a customer I ran into the following situation. Separate NIC’s are used on the PVS clients for provisioning traffic. And the provisioned clients are in a different subnet as the Provisioning Servers and the DHCP servers. Of course in DHCP the router address is configured so that clients are able to connect to the other subnet.

In between the two subnets the customer has two routers functioning as a HA pair. The default gateway address given to clients is a virtual IP configured on the HA pair. 

This looks something like this: 

So what went wrong? 

Let’s take a look at the bootprocess. The PXE client on the provisioned machine sends a broadcast over the network looking for a DHCP server. One of the routers in the HA pair (lets say 192.168.1.251) picks up the request and passes this on to the DHCP server (according to the helper). The DHCP server accepts the request and sends an ip address back to the provisioned machine, including the default gateway VIP as router address.  What happens next is something that I didn’t expect. Instead of using the default gateway VIP that was configured in DHCP, the PXE client uses the IP address of the router that answered to the DHCP request as the default gateway (192.168.1.251). And it hands this IP information over to the TFTP bootstrap program.

Next Windows boots without a problem. Within Windows a new DHCP request is being done resulting in the correct address (192.168.1.253) being configured as default gateway in Windows. 

This all looks ok.. right?

BUT…. a very interesting fact here is that the communication between the  provisioned client and the vDisk residing on the PVS server is NOT handled by Windows. Even when Windows is started the bootstrap program is still handling the communication from and to the vDisk. So because this bootstrap program uses the Router1 address as the route to the other subnet, there’s now no redundant situation for the routers. When Router1 fails the provisioned client just freezes, because it’s are unable to contact the vDisk that resides in the other subnet!

We went even further in testing this by configuring no default gateway at all in DHCP. In this scenario Windows is able to boot because PXE just uses the router that first answers the DHCP request. But within Windows there is now no gateway to the Provisioning Server. Doing a ping to the Provisioning Server results in the message “Destination host unreachable”. So Windows is not even communicating with the vDisk when booted. 

Ok.. back to the problem.

The problem is that the PXE client uses the physical IP of Router1 as the gateway and passes this through to the bootstrap program. From this moment on the bootstrap program communicates over this IP, even when in Windows. So when Router1 fails, contact with the vDisk is lost.

Luckily this problem can be easily fixed by configuring a specific default gateway in the bootstrap program. You can do this by clicking Configure Bootstrap in the Provisioning Services console, and specifying the subnet and gateway address.

Now the bootstrap program uses this gateway as the route to communicate with the vDisk, in stead of using the gateway it get’s passed through from the PXE client. Resulting in fully working HA situation!

Citrix is doing a real good job providing technotes. Technotes contain solutions for specific problems, describe best practices or just provide information about Citrix products that's not directly availible in product documentation. Working with Provisioning Server a lot, some of this technotes really saved my life! A few of these are in my opinion a must read for everyone who's working or going to work with Citrix Provisioning Services in the future.

I collected these technotes in this blog:

Best Practices for Configuring Provisioning Server on a Network
http://support.citrix.com/article/CTX117374

Registry Settings to Improve Failover Times for Citrix Provisioning
http://support.citrix.com/article/CTX119223

Understanding Write-Cache in Provisioning Server
http://support.citrix.com/article/CTX119469

Provisioning Server and Citrix License Server Interaction FAQ
http://support.citrix.com/article/CTX117378

Reference of PXE Error Codes
http://support.citrix.com/article/CTX120956

Hard Drive Local Cache and Boot Process for Provisioning Services
http://support.citrix.com/article/CTX123570

Provisioning Target and Daylight Savings Time
http://support.citrix.com/article/CTX123336

Today the guys from PQR and Login Consultants released the Phase II whitepaper of Project Virtual Reality Check. This phase is mainly focussed around the new Intel Nehalem Processor and has some interesting results!

From the Project VRC website: 

This new Project VRC phase 2 whitepaper focuses completely on analyzing Terminal Services (TS) workloads running on the latest generation hardware and hypervisors. While in phase 1 of project VRC VMware ESX3.5, Citrix XenServer 5.0 and Microsoft Hyper-V 1.0 were tested on a HP DL385G5 with AMD “Barcelona” processors, this whitepaper reviews vSphere 4.0, XenServer 5.5 and Hyper-V 2.0 running on HPDL380G6 with Intel “Nehalem” processor.

Project VRC made a deliberate choice not to directly compare Hypervisors in phase 1 (and published results in separate whitepapers), the community and the vendors, logically, did. Therefore, we decided that Project VRC will release this single whitepaper for the phase 2 results, including the three hypervisors.

You can download the new whitepaper on the website: http://www.projectvrc.com

When you have worked with Provisioning Services you know the process to create a vDisk. First you install your gold build server or workstation (the master target device) with the OS and all the software and configuration you want to have in your vDisk. You also install the Provisioning Services client on this machine. Then you create an empty vDisk on the Provisioning Server, add the master target device to the Provisioning Services database and link the empty vDisk to the master target device. After that you make sure boot from hard drive is selected on the master target device and then boot the device. After the device is started and the vDisk is connected you can start the image building process.

My opinion is that this is quite a time consuming procedure with lots of steps you have to take. Especially when you install your master target device fully automated, like I like to do, it sucks that you have to use this process afterwards to create the actual vDisk. So I was very happy to find another way to do this process of creating the vDisk. Now my goldbuild installation is literally 100% automated..! From installing the Operating System to creating the vDisk.

To do this, there are just 3 simple things you need to do.

First of all the Provisioning Services client software consists of two parts; 1: the target device drivers and optimization stuff and 2: XenConvert. The XenConvert contained in this package however is a customized version that only allows to convert to Provisioning Services vDisk. So the first thing I did is extract the two MSI’s from the Provisioning Services client setup. To do this just start the install and instead of finishing the installation, browse to the temp folder in your userprofile and locate the MSI’s. Actually the only MSI you really need from the installation package is the one that is called “Citrix Provisioning Services Target Device.msi”. The other MSI you need is the full XenConvert client (version 2.0.2 at the moment). The full XenConvert MSI can be downloaded from the Citrix website. So when you have both MSI’s separately you just install them (unattended if you want) on the master target device.

Second thing you need to do is make sure you have a separate disk partition installed on the master target device. Make sure this partition is big enough to contain the complete vDisk. We're going to use this partition to save the vDisk on.

Third and last action is building the disk via the full XenConvert client. You can do this by running the following command manually or via a script:

"%PROGRAMFILES%\Citrix\XenConvert\XenConvertCLI" P2VHD [Diskname] [Sourcedrive] [Destinationdrive]

Example:"%PROGRAMFILES%\Citrix\XenConvert\XenConvertCLI" P2VHD GoldBuild C:\ V:\

After the process is finished the destinationdrive contains the vDisk in VHD format. You can copy this VHD file directly to your Provisioning Server vDisk store and import in the database. Because the Target Device software is installed on the disk, the vDisk immediately works on your target devices!

Another very handy thing to do; when you have to change your vDisk, just apply your changes on the master target device and build a completely new vDisk using the process above. A big advantage is that this way, you can easily update stuff like network drivers, XenTools, VMware tools, PVS Target Device software.. without having to reverse image your vDisk!

Of course there are some limitations of using this procedure for creating your vDisks. As far as I found out one is that you only have the possibility to create a vDisk in dynamic format. Secondly you're only able to create a vDisk with one partition. But if these limitations don't matter to you I can assure you this is a very easy way to create your vDisks!

It’s been a while since I posted some content on my blog. But one of my goals for 2010 is to continue sharing my thoughts and knowledge about the products and technologies I love.

I’m using Citrix Provisioning Services for XenApp environments for a while now, and I must say I love the product! Using PVS you’re able to create an extremely flexible and efficient XenApp environment. I don’t think I need to point all advantages of using Provisioning Services in a XenApp environment here…

But after implementing the product for a while I also discovered that making your environment 100% High Available can be a real bitch. There’s a lot that needs to be done and you can easily forget a component. In this article I will try to explain how to implement a 100% highly available Provisioning Server environment. I know there’s already some documentation and blogs out there that describe Provisioning Server High Availability. But none of the seem to tell you the complete story.

Streaming Service

The streaming service is the core of your environment and is the service that provides the actual vDisk to the target devices. This is obviously an essential part of the environment. The streaming service is provides by the Provisioning Servers. Since version 5 Provisioning Server uses a farm model. So you can now easily add extra servers to your Provisioning Server farm to handle the load of your target devices. When one of your Provisioning Servers fail the target devices connected to that server will flawlessly failover to the other Provisioning Server in your farm.

vDisk Store

Having multiple Provisioning Servers in your farm, means that all servers must have access to the vDisk you want to provide to your target devices. This can be leveraged by pointing both Provisioning Servers to a file share containing the vDisk files. Watch out using this options in large environments as might not scale to way you expect it to. Another option is to use shared storage solution with in addition a clusterfilesystem that supports active/active mode disks. This means that you’re able to attach the same LUN to both your Provisioning Servers and they can both read and write at the same time. As this is probably the best solution, it’s definitely not the most affordable one.

A very good alternative is to use the local storage of your Provisioning Servers. Just create the same folder structure on both servers and create a vDisk store that points to the local folder. Of course you have to make sure the vDisk in both locations is always up-to-date. 

Database

The Provisioning Server database can be easily made high available using Microsoft clustering. And even when this is not a preferred option you can make use of the Offline database feature. When using this feature the database is being cashed locally on the Provisioning Server itself. So if the database becomes unavailable you have enough time to troubleshoot the problem and maybe restore the database from backup.

The Boot process

The real challenge in making Provisioning Services high available lies in the boot process. To fully understand this let’s take a look at the Provisioning Services boot process in detail. In our scenario we use 2 Provisioning Servers with the TFTP service installed. DHCP with options 66 and 67 is being used for delivering the TFTP location and bootfile name.

- The Provisioning Server target device is being booted
- The device is configured booting from network so a DHCP request is being broadcasted.
- The DHCP server receives the request and provides the target device with IP configuration.
- Together with this the target device receives the ip-address or DNS name of the TFTP server and the name of the TFTP bootstrap.
- The target device contacts the TFTP server and downloads the TFTP bootstrap.
- The bootstrap loads on the target device.
- The target device contacts the first available Provisioning Server configured in the bootstrap.
- The Provisioning Server being contacted matches the MAC address of the target device with the Provisioning Services database.
- When a match is being found the Provisioning Server checks if a vDisk is configured for the target device.
- If so, the target device is being assigned to the least loaded Provisioning Server configured for the vDisk using a load balancing algorithm.
- The target device connects to the chosen Provisioning Server and mounts the vDisk
- The vDisk is being booted and Windows starts. Looking at the above boot process we can easily point out the parts where we need redundancy.

DHCP

To be fully high available we first need a redundant DHCP solution. This is actually not really a pbig challenge. Because we use DHCP reservations for every target device, we can easily install a second DHCP server with the same scope and create the same set of reservations on this server. Because the ip-addresses are linked to the MAC addresses of the target devices, ip-addresses will only be used once.

TFTP

This is the tricky one, is the TFTP process. Provisioning Services provides it’s own TFTP service. This service can be installed as an option while installing the Provisioning Server. Using the Provisioning Services console we can configure the TFTP bootstrap to contain both Provisioning Servers. Now we can use DHCP option 66 to provide the TFTP server ip-address or DNS name to target devices. The problem we have here is that it’s only possible to add one ip-address or DNS name in option 66 in DHCP. This means only one of the TFTP servers can be provided to the target devices.

We have a few options here to provide redundancy in some form;

DNS roundrobin
We could create a DNS record which points two the ip-address of both/all TFTP servers. TFTP requests will then be equally load balanced between the TFTP servers. Problem here is that when one of the TFTP servers fail TFTP requests will then still be routed to that server. In this scenario half of your target devices will fail to boot.

Co-host TFTP with DHCP
This is actually a pretty nice solution. It’s not perfect, but gets close. In this setup we will have DHCP being hosted on the same servers as TFTP. In the DHCP configuration each server will point to itself for TFTP. So when one of the servers fail, the other server will answer all DHCP requests and thus automatically provides TFTP for all target devices. But as I said, this configuration is not perfect. When the TFTP server crashes on one of your servers, this server will still answer to DHCP request thereby providing itself as TFTP server. This means all target devices getting DHCP information from this server will fail.

Loadbalance TFTP using a hardware loadbalancer
This is the only option for providing 100% high availability for the Provisioning Services TFTP service. Using a hardware load balancer like Citrix NetScaler you can create a virtual TFTP server that’s intelligent enough to know when a node fails.

Boot using ISO
The last option is using an ISO to boot the target devices in stead of PXE. You can use the Provisioning Services Boot Device Manager to create a bootable ISO. In this ISO you configure the Provisioning Servers as you normally do in the Provisioning Services console at “Configure Bootstrap”. You then let your target devices boot from this ISO. In this case you can completely forget the DHCP / TFTP process.

Downside in this scenario is that this option is only valid in environments where XenApp servers are being virtualized. You don’t want to put bootable cdroms in all your physical XenApp servers;)

A very important thing to know is that ONLY the servers configured in the bootstrap will be used in loadbalancing and failover (This doesn’t only apply for the ISO method but also for using DHCP / TFTP). So lets say, if only one Provisioning Server is configured in the bootstrap, target devices booted from that Provisioning Server will never be able to failover to another Provisioning Server in case of an outage. Taken this into account means that whenever you add another Provisioning Server to your farm, you’ll have to update your ISO and reattach this ISO to ALL your target devices. Believe me… you don’t want to do this in an environment with more then 100 target devices.

Conclusion

While using Provisioning Services in your XenApp environment has lots of advantages, the complexity of the product doesn’t need to be underestimated. When not implemented correctly Provisioning Server can cause you a big headache. Especially when implementing a highly available environment you have to make sure all of above components are well configured and fully redundant.