Virtual Network/Switch Troubleshooting

For the most part, virtual switches are pain-free to implement and maintain. When something does go wrong, there are tools available to help you get through.

If you’re not sure that you completely understand the Hyper-V virtual switch, I wrote a two-part series about it on Altaro’s blog.

If you’re having issues with implementation, you might be interested in some ideas about design.

2012/12/31 Edit: Added NETCFG command for Windows Server and/or Hyper-V Server 2012

NVSPBIND

This tool was designed by a member of Microsoft’s Hyper-V team specifically to assist with virtual switches, although it could be of use in other circumstances. It is available on the MSDN site. Execute the download file on a full GUI installation of Windows; it will ask you to accept the licensing agreement and ask you where to extract its contents. It will place three files; the main executable, a text document showing the command syntax and an example of how to change adapter order binding, and a .DOC file with more extensive instructions.

This tool interacts with the various NIC drivers to affect protocol binding. It can be used to re-order adapter binding (as outlined in the text file), but that will be unnecessary in most installations. If you run NCPA.CPL on a full GUI installation of Windows and then open the Properties dialog for an adapter, the protocols and checkboxes are the rough equivalent of what this tool works with. Unbinding a protocol in that window is as simple as unchecking a box and clicking OK. This tool mimics that functionality with the /u, /b, /d, and /e switches. The /u and /b respectively unbind and bind the virtual switch protocol (VSP) to the specified NIC. The /d and /e switches are similar, but are used to work with other protocols. The included .DOC file gives examples of this usage. When the virtual switch protocol is bound to a NIC, all other protocols are automatically unbound; this is why the NIC disappears from IPCONFIG after creating a virtual switch on it.

The most common use of this tool would be to remove the virtual switch protocol after a partially successful bind. A symptom of this condition is usually that, when creating the virtual network, Hyper-V Manager hangs for an extremely long time and finally reports a timeout. There are multiple potential causes. If none of the host’s other NICs are reachable and you create the virtual switch without selecting the “Allow management operating system to share this network adapter” box is one way. If you select the box and the management adapter needs a VLAN but you didn’t select it, you did select it but Hyper-V didn’t apply it (happens sometimes), or both you and Hyper-V did everything right but the NIC driver can’t handle the configuration, then you might need to run this tool to unbind the VSP. Another possible scenario that leads to this is when Hyper-V Manager is communicating with the host using the NIC you’re trying to create a virtual switch on even though you have a different NIC dedicated to management.

Another possible reason you’d need this tool is to correct a message indicating that the NIC is already bound to another virtual network. That can occur if you tried to use Hyper-V Manager to attempt to create the virtual switch after the above situation without first unbinding VSP. It can also occur after driver mishaps, such as problems derived from teaming.

NVSPBIND expects you to pass GUIDs to represent adapters. In order to determine the GUID to pass in, run “NVSPBIND /N”. It will show the GUID and names for each of the adapters in the system (Broadcoms in iSCSI HBA mode are not visible). If you don’t know the name of the NIC that you’re looking for, use “NVSPBIND /O * *”; you’re looking for items marked as “enabled” under the  “vms_pp” / ”Microsoft Virtual Switch Protocol” entry. You’ll then have to look under “NVSPBIND /N” for those names to get the GUID for the NIC to use with /U.

To try to prevent the above situations, after unbinding the VSP using NVSPBIND but before trying to create the virtual switch again, check these things: if you have a dedicated management NIC, ensure that it has the lowest metric using NETSH (the master installation document has a section covering this). You can also ensure the NIC to be used as a virtual switch has a static IP with no connectivity to the computer you’re running Hyper-V Manager from by using a static IP with no gateway and in a different subnet or by setting it as a DHCP client but not allowing it to contact a DHCP server. Once the VSP protocol is bound to it, it will no longer have any IP connectivity, so there is no risk to using completely bogus TCP/IP information beforehand to ensure you have no troubles.

If you do not have a dedicated management NIC and will be splitting that role with this particular NIC, the first thing to do is reconsider that approach. NICs really aren’t that expensive and it seems to already be causing you headaches. If you must set it up this way, then ensure that you check the “Allow…” box when creating the virtual switch and include any VLAN information that the management adapter needs. Remember that even if the physical port you are plugging the NIC into is a member of a particular VLAN, converting this physical NIC to a virtual switch means that any virtual NICs plugged into it exist only in VLAN 1 unless configured otherwise; that includes the management adapter.

Even though this tool has a /b switch, you should not use it to bind the VSP to an adapter. Only use Hyper-V Manager for that task.

I know of no one that has ever used the repair (/r) switch and the documentation only indicates that it exists. If you’re setting out to fix a problem with a protocol or virtual switch, this might be the place to start.

NVSPSCRUB

This is a script created by the same individual that gave us NVSPBIND, but its usage is for more drastic ends. It goes beyond the virtual switch protocol and targets the virtual NICs and virtual switch configurations. It is also available from the MSDN site.

Running this tool on native Hyper-V or Server Core will require you to call it from CSCRIPT: “CSCRIPT NVSPSCRUB.JS /?”. The typical usage will be the /P switch, which purges all virtual network settings. The /N switch will allow you to target specific NICs for cleaning. The /V switch only removes disabled virtual NICs, which is uncommon usage. After using this tool, you will generally need to reboot before attempting to create another virtual switch.

Because it’s a JavaScript file, you can examine it for an example of using WMI with Hyper-V.

Tool of Last Resort

If neither of the above tools fixed your problem, or if your host is isolated and you have no feasible method of getting the tools downloaded to it, there is a final option. This method completely removes the VSP from the server, taking all bindings, configurations, and just about everything else VSP-related along. It would be best to term this as the “nuclear option”. You do not need anything other than what came on the server to perform this procedure.

1. At the command prompt, run “NETCFG –U VMS_PP” and reboot the server.
2. The next command is extremely long and error-prone. I will use green text to indicate the optimal places to press [TAB] for autocomplete to help you out, and red text to indicate where auto-complete will stop and you must resume typing. Double-check the displayed command before pressing [Enter]. There are several similarly titled files in the folder you’ll be working in, and it is quite frustrating to skip around until you get the right one.
   The commands are entered on one line; I have broken them up for web display. If you have remote desktop connectivity to your host, use a text editor to reattach these into a single line before copy/pasting into the RDP session. There are no spaces where the line breaks are.
   If your host is Hyper-V 2008 R2 or Server Core 2008 R2 RTM (no service packs):
   NETCFG –L C:\Windows\winsxs\
   amd64_wvms_pp.inf_31bf3856ad364e35_6.1.7600.16385_none_beda85050b13680c
   \wvms_pp.inf -C P -I vms_pp
   If your host is Hyper-V 2008 R2 SP1 or Server Core 2008 R2 SP1:
   NETCFG –L C:\Windows\winsxs\
   amd64_wvms_pp.inf_31bf3856ad364e35_6.1.7601.17514_none_c10b98cd0801eba6
   \wvms_pp.inf -C P -I vms_pp
   If your host is Windows Server or Hyper-V Server 2012 RTM:
   NETCFG –L C:\Windows\winsxs\
   amd64_wvms_pp.inf_31bf3856ad364e35_6.2.9200.16384_none_bbaf3ac27b26975c
   \wvms_pp.inf -C P -I vms_pp
3. Reboot the server. The VSP will be available to all NICs but not bound to any.
4. You can attempt to fix whatever prevented the virtual switch from working the first time and attempt to recreate it.

I have had to use this tool to fix a host after attempting to enable VMQ on an Intel card and assign it to a virtual switch. Unfortunately, before I was even able to run this, I had to stop the server from blue-screening on each boot (IRQL_NOT_LESS_OR_EQUAL). To do that:

1. Boot to Safe Mode.
2. Run “REGEDIT”.
3. Navigate to HKLM\System\CurrentControlSet\Control\Class\{4D36E972-E325-11CE-BFC1-08002BE10318}.
4. These entries represent all NICs in the system. Expand each node in turn, looking for a “PROSetNdi” key; adapters with this key are Intel cards.
5. Highlight the numeric node(s) that represent the card(s). Find the entry named “*vmq” and set it to 0. Repeat for all Intel adapters.
6. Reboot the server into normal mode and follow the NETCFG instructions above.