Category: Hyper-V

Install VMWare ESXi 6 as a Nested VM in Hyper-V

Install VMWare ESXi 6 as a Nested VM in Hyper-V

Windows Server 2016 is out and has been for a while now, and as a lot of people start to delve into it they will notice some new features lurking in the background. One of the not so obvious new features is Nested VM’s; what this means is that you can run a Hypervisor inside a virtual machine.

Some of you may think why would anyone want to run a Hypervisor inside another Hypervisor of which a couple scenarios spring to mind quite quickly:

  1. You may provide hosting services and a customer requests a Hyper-V (or ESXi) server but rather then use another physical server you could allocate the resources on one of your servers and provide them access to the virtual hypervisor, thus giving them a secluded space to make and delete VMs freely without any risk of them affecting your Virtual Infrastructure.
  2. Another as was the case for me I needed to test something on a Mac to confirm compatibility, not having one to hand I decided a VM would be a simple solution but I needed an ESXi and so I decided to spin up an ESXi server as a VM inside one of my Hyper-V servers. This enabled me to quickly confirm my query without needing to find some spare hardware to spin up a physical ESXi box.

As you can see there could be some uses for Nested VMs although I think it will take time to be adopted… even for non-production systems. When I went to set-up this ESXi server I struggled initially to find a guide that was readily available and simple to follow as such I have decided to share what i’ve learnt below:


  1.  A Hyper-V Host:
    1. A Hyper-V host running Windows Server 2016 or Windows 10 Anniversary Update.
    2. A Hyper-V VM running Windows Server 2016 or Windows 10 Anniversary Update (Yes you can do Nested Virtualization inside Nested Virtualization…).
    3. A Hyper-V VM with configuration version 8.0 or greater.
    4. An Intel processor with VT-x and EPT technology.
  2. Enable-NestedVM.ps1 – A PowerShell script for enabling Nested Virtualization in a Hyper-V VM. Click here to get the file from the Microsoft team on GitHub.
  3. VMWare PowerShell CLI installed – I used 6.3 release 1 that I downloaded from here. (Note: you will need a VMWare account, if you don’t have one then you can sign up here.)
  4. ESXi-Customizer-PS.ps1 – A PowerShell script that will download the ESXi 6.0 iso and inject the required network drivers into it for you. I downloaded it from here.

To make matters simpler if you have all three items in the same working folder – mine is “C:\NestedESXi\” then it may be easier to follow my guide, (but of course that is totally up to you).

Phase 1 – Preparing ESXi ISO with Injected Drivers:

As a stock install the ESXi Kernel doesn’t support the “Microsoft Virtual Network Adapter” or indeed the “Legacy Network Adapter”, however as the Legacy Adapter emulates a DECchip 21140 (net-tulip) we can inject the drivers into the ISO thus giving ESXi the ability to use the Legacy Network Adapter.

  1. Install VMWare PowerCLI (simple Next, Next, Next).
  2. Open a PowerShell window
  3. Navigate to your working directory, in my case  CD C:\NestedESXi\
  4. Once in the working directory we can execute the following  .\ESXi-Customizer-PS-v2.5.ps1 -v60 -vft -load net-tulip (Note: the script version may have changed so double check the part as highlighted - ESXi-Customizer-PS-v2.5.ps1).
  5. After a few minutes the ISO will have been downloaded and the net-tulip drivers inserted into it making it ready to use, that ISO is also put straight into the folder that you ran the script from (in my case C:\NestedESXi\).

Phase 2 – Creating the VM

  1. In Hyper-V Manager go to create a new Virtual Machine.
  2. Choose what you want to Name it and make sure the location is where you want the ESXi VM to be stored, then click Next
  3. Select Generation 1 and click Next.
  4.  Set the Memory to at least 4096MB and Uncheck “Use Dynamic Memory for this Virtual Machine” then click Next.
  5. Don’t configure networking now, just click Next since we need to use the Legacy Adapter which can only be selected after the initial config wizard.
  6. Select Create a new virtual hard disk and set the Size to 10GB (this is just going to be the boot disk for the ESXi Hypervisor), click Next.
  7. Select “Install an Operating System from a bootable CD/DVD-ROM” and browse to the ESXi ISO that is in your working directory from Phase 1. 
  8.  Click Next and then Finish to create the VM.  
  9.  Right-Click the new Virtual Machine and select Settings.
  10. Navigate to the Processors tab and change the allocated number of Virtual Processors (it needs to be at least 2) 
  11.  Select the existing Network Adapter and click Remove
  12.  Select the Add Hardware tab and select Legacy Network Adapter then Add
  13.  Select the relevant Virtual Switch so that the ESXi has network access
  14.  Click OK

The majority of the VM configuration is now complete, we just need to enable the Nested Virtualization Extensions for the VM and then we can move onto configuring the ESXi itself.


Phase 3 – Enabling Nested Virtualization

  1. Open a PowerShell console.
  2. Enter the following commands (adjusting the vmName to match the name of your Virtual Machine in my case the name is “NestedESXi”)
    CD C:\NestedESXi\
    .\Enable-NestedVm.ps1 -vmName 'NestedESXi'


  3. Enter Y when asked to confirm any of the changes (Note : the Y is case sensitive).
  4. The Virtual Machine is now ready to have ESXi installed into it.


Phase 4 – Initial Boot of ESXi Virtual Machine

  1. Start up the ESXi Virtual Machine and make sure you’re connected to it so you can see the ESXi boot screen:
  2. Quickly press Tab.
  3. Add the ignoreHeadless=TRUE to the Boot Options
  4. Press Enter.
  5. The ESXi Installation process will now start.
  6. After a few minutes the VMWare ESXi 6 Installer will start up  
  7. You can now go through the ESXi installation process.
  8. You may receive this warning during the installation process but you can ignore it ss_vmwareinhv_esxiinstallerwaring
  9. The installation process will begin 
  10. Once the ESXi installation has completed you will see this message ss_vmwareinhv_esxiinstallercomplete
  11. Make sure the ESXi Installation ISO is ejected prior to rebooting the Virtual Machine
  12. Press Enter to reboot the VM.

Phase 5 – Configure the ESXi Boot Options to persistently use “ignoreHeadless=TRUE”

The last thing we need to do is to set the ESXi VM to persistently use the “ignoreHeadless=TRUE” so that you do not experience any boot issues.

  1. When the ESXi machine reboots, quickly press SHIFT-O to set the boot options.
  2. Add the ignoreHeadless=TRUE to the Boot Options
  3. Press Enter to boot up the ESXi host
  4. Once the ESXi has booted up, press F2.
  5. Enter the root login credentials that were set during the ESXi installation process.
  6. Select Troubleshooting Options and press Enter
  7. Select Enable ESXi Shell and press Enter
  8. Press ALT+F1 to bring up the console
  9. Enter your root credentials.
  10. Enter the following command:
    esxcfg-advcfg --set-kernel "TRUE" ignoreHeadless


  11. Press ALT+F2 to return to the main ESXi screen.

The ignoreHeadless=TRUE setting is now persistent and will be applied at ever boot without manual intervention required.

There you have it a fully functional ESXi Host running inside a Hyper-V Server. Of course Microsoft do support running Hyper-V inside another Hyper-V but neither Microsoft or VMWare formally support running an ESXi server in a nested environment (at least at present) and as such this should not be used for any production purposes but hopefully this has opened some eyes to a hidden feature which hasn’t received much publicity. Hyper-V may finally be catching up to the VMWare party after joining exceptionally late who knows what features Microsoft are looking at for future releases.