Hyper-V and VMQ’s – Mythbusting

Updated in Mar 2023:
Ignore the May 2022 update, I was right all along. Microsoft actually change the default VMQ in Azure Stack HCI deployments, so that says all I need to know.

**** ALWAYS change base processor to core 2 (or 1 if HT is not enabled) ****

Updated in May 2022:
For about a year or so now since MS made some tweaks and a growing confidence in the out of the box settings, the optimal position has changed on WS2019. Essentially leave VMQ in default settings for Windows Server 2019.

A few months after this ranty post, Dan Quomo @ Microsoft published some blogs cleaning up some of the confusion around VMQ and its friends:
Synthetic Accelerations in a Nutshell – Windows Server 2012 – Microsoft Tech Community
Synthetic Accelerations in a Nutshell – Windows Server 2012 R2 – Microsoft Tech Community
Synthetic Accelerations in a Nutshell – Windows Server 2016 – Microsoft Tech Community
Synthetic Accelerations in a Nutshell – Windows Server 2019 – Microsoft Tech Community

Also since this rant, some blogs and posts have been taken down removing some of the confusion, so hopefully like is a little simpler in the Hyper-V world of networking.

Windows Server 2016 settings below still stand.

Cheers
Dan 🙂

End of May 2022 update.
Ok, so this has been something that’s been brewing for years now…

Having worked with many clients who have had Hyper-V issues related to VMQ, and spending a lot of my time debating incorrect/stale blog posts, I feel it’s time to set to story straight. So please excuse me if I come across a little too brash…

Yes, this is a bit of a “blowing of steam” post, but hopefully this can be helpful to those who have been misdirected or just need some assistance with VMQs on Hyper-V in scaled environments…

Ultimately, the main goal of this post is to help you succeed in your Hyper-V experience.

Target audience

Before I continue, let’s set the context of the post. This is for Hyper-V environments running at least 10GbE networks and using SUPPORTED hardware and drivers… What do I mean supported? Well… It’s on the Windows Server Catalog (WSC) and has “Virtual Machine Queue (VMQ)” listed as a certification. This is also targeted at those using legacy storage such as iSCSI or FC SANs (wow, I really do have a bee in my bonnet).

This post may also be helpful for those who are having random performance or stability issues in their new Hyper-V environment. This is often because VMQ is using core 0 and it is getting pinned to 100% starving hosts of being able to communicate with the cluster and or other critical low level operations. To be honest, those issues are often very hard to qualify and it’s just a matter of making some tweaks and seeing how things go.

Not for you…

If you’re running a single Hyper-V host with 1GbE NIC, then this post is not for you. If you are using HCI or your storage is SMB, then whilst this post would be helpful, there are considerations outside of what I will cover here.

Also, this post is not intended on teaching you what VMQ’s are or how they work (I may touch on it), this is a how to gather the information required and to set them correctly.

If you want more accurate info on what VMQ’s and other settings are, then these two blogs are your go-to’s:

Darryl van der Peijl 
Altaro Hyper-V Dojo

Note: There are more, but these two blogs are from others that I know have extensive experience deploying and managing Hyper-V in scaled production environments. Not labs.

 

Ok, the disclaimers are out of the way and we’re all on the same page? Great 🙂

Sections of this post:

1. Misconceptions of VMQs
2. Common VMQ rules
3. VMQ modes
4. Teaming Modes
5. VMQ Settings

 

1. Misconceptions of VMQs (and some of Hyper-V in general)

Firstly, let’s address a few common misconceptions:

• The out of the box settings are best – WRONG
• Both VMQ and RSS need to be configured on the same interface – WRONG
• Working out optimal VMQ is complicated – WRONG
• disable VMQ on Team interface – WRONG
• disable VMQ for any reason – WRONG
• NUMA = physical Proc – WRONG
• Dynamic load balancing algorithm is best – WRONG
• LACP gives more network perf to VMs – WRONG
• Switch Independent won’t work on my switches – WRONG

Note: I bet you’re a bit confused right now because you’ve read some blogs or support posts saying the opposite to much of the above. That’s fine, I’ll help you with that. They’re either outdated or wrong. 🙂 (no fence-sitting going on here…)

 

2. Common VMQ (and aligning with VMQ) rules

Some common rules for VMQ:

• Never use Core 0
• Do not overlap NUMA nodes
• Physical cores only (ignores HT)
• Max 64 physical CPUs
• In VMs with more than 1 vCPU, enable vRSS

When it comes to VMQ and RSS, think of it as RSS is a physical adapter setting but when you apply a virtual switch the setting changes to VMQ. Kinda correct kinda incorrect, but if you think about it that way, you’ll increase your likelihood of getting VMQ and RSS settings correct.

You mentioned core 0, what’s this all about?

Well, I won’t go in too deep here, but essentially some primary processing functions of the OS happen on core 0. And if we have some high network IO being processed this can essentially pin core 0 to 100% utilization and cause all sorts of performance and stability issues

In older environments with 1Gb networks, the amount of processing required to compute the network IO was minimal. But with 10GbE and higher networks, the amount of throughput utilized requires more processing. This is what VMQ helps with, distributing the compute requirements of network IO across multiple cores and removing this overhead from core 0.

Image: Incorrectly configured VMQ at work…
(Original source unknown. A client sent this to me but I am totally using it…)

3. Teaming Modes

Before we configure VMQ, we need to understand our teaming modes, and more importantly which is best?

To clarify what these are:

Switch Independent

This is the default recommendation for all Hyper-V virtual switches. Each interface is it’s own trunk and the physical switch sees these as separate independent interfaces.

This is your go-to config and will give a higher likelihood of success.

Summary: Switch Independent is the way to do Hyper-V switches properly

Switch Dependent

This is LACP or Static Teaming. Quite often I have customers who’s networking teams dictate this and try to mandate LACP as the methodology for fail-over. I STRONGLY recommend this be overruled and Switch Independent be the methodology used.

Whilst technically LACP works (I ran LACP in prod for a long time) too many time have I seen incorrect, misaligned or configs that just don’t work which give organisations a poor Hyper-V experience. Additionally, with VMQ we can get more networking performance out of our environments. Ok, networking people will argue this, but they thinking physical aggregation, not virtualization.

So which is best? I think you know my answer already 🙂

The long politically correct answer – it sometimes depends on the environment but ultimately Switch Independent is typically the optimal configuration.

Simple direct answer – Switch Independent is best.

 

4. VMQ modes

When configuring VMQ, there are two modes that we must be aware of: Min-queue mode and Sum-of-queue mode

Sum-of-queue mode:
All NICs in the team/switch must have separate processor assignments

Min-queue mode:
All NICs in the team/switch must have identical processor assignments

A good blog from Charbel Nemnom has a little bit more info on the two modes:

But which mode when? This common table should help with the overall selection…

Teaming mode	Hyper-V Port	Dynamic	Address Hash
Switch Independent	Sum-of-Queues	Sum-of-Queues	Min-Queues
Switch Dependent	Min-Queues	Min-Queues	Min-Queues

If you’ve been reading properly though, you’ll note I stated above that you should always be using Hyper-V Port as your teaming algorithm. Following that rule, your table really should look like this:

Teaming mode	Hyper-V Port
Switch Independent	Sum-of-Queues
Switch Dependent	Min-Queues

Much simpler right? 😀

 

5. VMQ Settings

This formula has seen successful implementations of Hyper-V for the last 6-7 years using varying manufacturers and configurations. The one thing that has been consistent, is the correct application of VMQs. Ok, I won’t be THAT jerk, this has taken many hours/days/weeks of trial and error and many late nights up with my environment or assisting customers to work out the best results… i.e. a lot of blood, sweat and tears

Firstly get your numbers right… (This example is of a dual proc Intel 5118 with Mellanox ConnectX-4 NICs)

Get-WmiObject -Class Win32_Processor | select Name,SocketDesignation,NumberOfCores,NumberOfLogicalProcessors

What we know so far: 2 Sockets. Each socket has 12 physical cores and 24 hyper-threaded cores.

How many NUMA nodes per socket?

(Get-VMHostNumaNode).count

We have 4 NUMA nodes in total, meaning 2 NUMA nodes per socket. Just to reiterate the point that NUMA is not equal to physical Proc i.e. we can have more than one NUMA node per Proc as shown here. Although the NUMA nodes is important to note when designing VM configs (i.e. don’t give more RAM or CPU than per NUMA etc), Hyper-V is NUMA aware so this is more of an FYI at this stage…

Taking into consideration that we ignore HT and avoid using core 0, we are left with 11 processors per socket we can use…

Now we have Sum-of-queue mode or Min-Queue mode.. I will give both examples below.

For Switch Independent configs

So if we use the magic Excel table that we see in many blogs (yes, I use this too…), we see based on my hardware the optimal settings become quite clear.

i.e. Switch Independent > Hyper-V Port > Sum-of-Queues mode

For WGE2-0-17, find the first ‘x’. This is out -BaseProcessorNumber. Now could the number of ‘x’ and this is your -MaxProcessors value.

Edit: Since Windows Server 2016, we no longer needed the maxprocesors setting as the hypervisor takes care of this for us. Technically the Core 0 is also handled automagically in 2019 as well but still proposes a risk if we have driver or firmware bugs… So I still strongly recommend to bypass core 0. Absolutely zero negative to doing this, whilst there is minimal risk to not doing this. Fairly simple decision for me 🙂

Windows Server 2016:

Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 -MaxProcessors 11
Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 26 -MaxProcessors 11

Windows Server 2019: **Refer to May 2022 update at top of this post**

Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 
Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 26

 

Here we see those settings applied…

 

For Switch Dependent configs (or those stuck in the dark ages)

The same logic applies for building your config, the difference is that because we’re using Min-Queues mode (i.e. Switch Dependent > Hyper-V Port > Min-Queues mode) then we need to pin both NICs to the same processor set.

Windows Server 2016:

Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 -MaxProcessors 8
Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 2 -MaxProcessors 8

Windows Server 2019:

Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 
Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 2

Note: for Switch Dependent adapters, we are limited to a maxprocessors value of 1, 2, 4, 8 & 16. (Just another reason why Switch Independent is better 🙂 )

Unfortunately I can’t show that setting applied in my environment as I’m not stuck in 2007 using LACP.

 

Much, much more to consider…

Some other variables to consider… If you are using LACP (you poor soul) you are limited to a Load-Balancing Failover Team (LBFO) for your config. If using Switch Independent, you can use LBFO or preferably use Switch Embedded Teaming (SET) for Windows Server 2016 and above.

If you are using Switch Independent, use SET unless there is a specific reason not to. If you think you have a reason not to use SET, then reassess as you’re likely just over-complicating your environment.

After configuring LBFO teams we are presented a new interface in the OS with interface description of ‘Microsoft Network Adapter Multiplexor Driver’. By default this is VMQ enabled and should remain this way. Do not modify the team interface VMQ settings though, leave them as default.

NUMA Distance is also relative when it comes to finding the absolute optimal config, but using the defaults will not affect stability, but will have a very minimal impact on performance. The rule here is to keep you NIC and NUMA distance at 0 to ensure the moist efficient operation. More on the in Darryl’s blog here… Go to the section on NUMA Node Assignment.

 

Some example configs

Here are some examples where we have 10 core procs with hyper-threading enabled

Below is a correctly configured LBFO switch in Switch Independent mode.

Below is a correctly configured LBFO switch in Switch Dependent (LACP) mode

Below is correctly configure SET switch adapters (i.e. only supports Switch Independent)

 

i.e. Don’t overthink this. Just optimize VMQ for the team members as shown in the above examples

 

Note: For HCI or SMB environments, we need to make some considerations for RSS on the physical adapters. If you stick to the same guidelines of VMQ (because VMQ and RSS are the same technology under the hood), you’re on the right track.

Also I’d really like to keep going and assist with other features and/or settings but the purpose of this post is to assist with VMQ. But I won’t go into all of that here.

 

Let’s wrap this up – If you’re still confused, have some feedback or simply disagree with some of the above then please reach out. Whilst I’ve been in many Hyper-V environments over the years, I haven’t been in them all, so there will be outliers that go against the rules I’ve stated above.

I’ll maybe add to this over time but hopefully this dispels some of the myths and misconceptions out there.

Happy Hyper-V’ing!
Dan