Ok, so this has been something that’s been brewing for years now…
Having worked with many clients who have had Hyper-V issues related to VMQ, and spending a lot of my time debating incorrect/stale blog posts, I feel it’s time to set to story straight. So please excuse me if I come across a little too brash…
Yes, this is a bit of a “blowing of steam” post, but hopefully this can be helpful to those who have been misdirected or just need some assistance with VMQs on Hyper-V in scaled environments…
Ultimately, the main goal of this post is to help you succeed in your Hyper-V experience.
Before I continue, let’s set the context of the post. This is for Hyper-V environments running at least 10GbE networks and using SUPPORTED hardware and drivers… What do I mean supported? Well… It’s on the Windows Server Catalog (WSC) and has “Virtual Machine Queue (VMQ)” listed as a certification. This is also targeted at those using legacy storage such as iSCSI or FC SANs (wow, I really do have a bee in my bonnet).
This post may also be helpful for those who are having random performance or stability issues in their new Hyper-V environment. This is often because VMQ is using core 0 and it is getting pinned to 100% starving hosts of being able to communicate with the cluster and or other critical low level operations. To be honest, those issues are often very hard to qualify and it’s just a matter of making some tweaks and seeing how things go.
Not for you…
If you’re running a single Hyper-V host with 1GbE NIC, then this post is not for you. If you are using HCI or your storage is SMB, then whilst this post would be helpful, there are considerations outside of what I will cover here.
Also, this post is not intended on teaching you what VMQ’s are or how they work (I may touch on it), this is a how to gather the information required and to set them correctly.
If you want more accurate info on what VMQ’s and other settings are, then these two blogs are your go-to’s:
Note: There are more, but these two blogs are from others that I know have extensive experience deploying and managing Hyper-V in scaled production environments. Not labs.
Ok, the disclaimers are out of the way and we’re all on the same page? Great 🙂
Sections of this post:
- Misconceptions of VMQs
- Common VMQ rules
- VMQ modes
- Teaming Modes
- VMQ Settings
1. Misconceptions of VMQs (and some of Hyper-V in general)
Firstly, let’s address a few common misconceptions:
- The out of the box settings are best – WRONG
- Both VMQ and RSS need to be configured on the same interface – WRONG
- Working out optimal VMQ is complicated – WRONG
- disable VMQ on Team interface – WRONG
- disable VMQ for any reason – WRONG
- NUMA = physical Proc – WRONG
- Dynamic load balancing algorithm is best – WRONG
- LACP gives more network perf to VMs – WRONG
- Switch Independent won’t work on my switches – WRONG
Note: I bet you’re a bit confused right now because you’ve read some blogs or support posts saying the opposite to much of the above. That’s fine, I’ll help you with that. They’re either outdated or wrong. 🙂 (no fence-sitting going on here…)
2. Common VMQ (and aligning with VMQ) rules
Some common rules for VMQ:
- Never use Core 0
- Do not overlap NUMA nodes
- Physical cores only (ignores HT)
- Max 64 physical CPUs
- In VMs with more than 1 vCPU, enable vRSS
When it comes to VMQ and RSS, think of it as RSS is a physical adapter setting but when you apply a virtual switch the setting changes to VMQ. Kinda correct kinda incorrect, but if you think about it that way, you’ll increase your likelihood of getting VMQ and RSS settings correct.
You mentioned core 0, what’s this all about?
Well, I won’t go in too deep here, but essentially some primary processing functions of the OS happen on core 0. And if we have some high network IO being processed this can essentially pin core 0 to 100% utilization and cause all sorts of performance and stability issues
In older environments with 1Gb networks, the amount of processing required to compute the network IO was minimal. But with 10GbE and higher networks, the amount of throughput utilized requires more processing. This is what VMQ helps with, distributing the compute requirements of network IO across multiple cores and removing this overhead from core 0.
Image: Incorrectly configured VMQ at work…
(Original source unknown. A client sent this to me but I am totally using it…)
3. Teaming Modes
Before we configure VMQ, we need to understand our teaming modes, and more importantly which is best?
To clarify what these are:
This is the default recommendation for all Hyper-V virtual switches. Each interface is it’s own trunk and the physical switch sees these as separate independent interfaces.
This is your go-to config and will give a higher likelihood of success.
Summary: Switch Independent is the way to do Hyper-V switches properly
This is LACP or Static Teaming. Quite often I have customers who’s networking teams dictate this and try to mandate LACP as the methodology for fail-over. I STRONGLY recommend this be overruled and Switch Independent be the methodology used.
Whilst technically LACP works (I ran LACP in prod for a long time) too many time have I seen incorrect, misaligned or configs that just don’t work which give organisations a poor Hyper-V experience. Additionally, with VMQ we can get more networking performance out of our environments. Ok, networking people will argue this, but they thinking physical aggregation, not virtualization.
So which is best? I think you know my answer already 🙂
The long politically correct answer – it sometimes depends on the environment but ultimately Switch Independent is typically the optimal configuration.
Simple direct answer – Switch Independent is best.
4. VMQ modes
When configuring VMQ, there are two modes that we must be aware of: Min-queue mode and Sum-of-queue mode
All NICs in the team/switch must have separate processor assignments
All NICs in the team/switch must have identical processor assignments
A good blog from Charbel Nemnom has a little bit more info on the two modes:
But which mode when? This common table should help with the overall selection…
|Teaming mode||Hyper-V Port||Dynamic||Address Hash|
If you’ve been reading properly though, you’ll note I stated above that you should always be using Hyper-V Port as your teaming algorithm. Following that rule, your table really should look like this:
|Teaming mode||Hyper-V Port|
Much simpler right? 😀
5. VMQ Settings
This formula has seen successful implementations of Hyper-V for the last 6-7 years using varying manufacturers and configurations. The one thing that has been consistent, is the correct application of VMQs. Ok, I won’t be THAT jerk, this has taken many hours/days/weeks of trial and error and many late nights up with my environment or assisting customers to work out the best results… i.e. a lot of blood, sweat and tears
Firstly get your numbers right… (This example is of a dual proc Intel 5118 with Mellanox ConnectX-4 NICs)
Get-WmiObject -Class Win32_Processor | select Name,SocketDesignation,NumberOfCores,NumberOfLogicalProcessors
What we know so far: 2 Sockets. Each socket has 12 physical cores and 24 hyper-threaded cores.
How many NUMA nodes per socket?
We have 4 NUMA nodes in total, meaning 2 NUMA nodes per socket. Just to reiterate the point that NUMA is not equal to physical Proc i.e. we can have more than one NUMA node per Proc as shown here. Although the NUMA nodes is important to note when designing VM configs (i.e. don’t give more RAM or CPU than per NUMA etc), Hyper-V is NUMA aware so this is more of an FYI at this stage…
Taking into consideration that we ignore HT and avoid using core 0, we are left with 11 processors per socket we can use…
Now we have Sum-of-queue mode or Min-Queue mode.. I will give both examples below.
For Switch Independent configs
So if we use the magic Excel table that we see in many blogs (yes, I use this too…), we see based on my hardware the optimal settings become quite clear.
i.e. Switch Independent > Hyper-V Port > Sum-of-Queues mode
For WGE2-0-17, find the first ‘x’. This is out -BaseProcessorNumber. Now could the number of ‘x’ and this is your -MaxProcessors value.
Edit: Since Windows Server 2016, we no longer needed the maxprocesors setting as the hypervisor takes care of this for us. Technically the Core 0 is also handled automagically in 2019 as well but still proposes a risk if we have driver or firmware bugs… So I still strongly recommend to bypass core 0. Absolutely zero negative to doing this, whilst there is minimal risk to not doing this. Fairly simple decision for me 🙂
Windows Server 2016:
Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 -MaxProcessors 11 Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 26 -MaxProcessors 11
Windows Server 2019:
Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 26
Here we see those settings applied…
For Switch Dependent configs (or those stuck in the dark ages)
The same logic applies for building your config, the difference is that because we’re using Min-Queues mode (i.e. Switch Dependent > Hyper-V Port > Min-Queues mode) then we need to pin both NICs to the same processor set.
Windows Server 2016:
Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 -MaxProcessors 8 Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 2 -MaxProcessors 8
Windows Server 2019:
Set-NetAdapterVMQ -Name WGE2-0-17 -BaseProcessorNumber 2 Set-NetAdapterVMQ -Name WGE1-0-17 -BaseProcessorNumber 2
Note: for Switch Dependent adapters, we are limited to a maxprocessors value of 1, 2, 4, 8 & 16. (Just another reason why Switch Independent is better 🙂 )
Unfortunately I can’t show that setting applied in my environment as I’m not stuck in 2007 using LACP.
Much, much more to consider…
Some other variables to consider… If you are using LACP (you poor soul) you are limited to a Load-Balancing Failover Team (LBFO) for your config. If using Switch Independent, you can use LBFO or preferably use Switch Embedded Teaming (SET) for Windows Server 2016 and above.
If you are using Switch Independent, use SET unless there is a specific reason not to. If you think you have a reason not to use SET, then reassess as you’re likely just over-complicating your environment.
After configuring LBFO teams we are presented a new interface in the OS with interface description of ‘Microsoft Network Adapter Multiplexor Driver’. By default this is VMQ enabled and should remain this way. Do not modify the team interface VMQ settings though, leave them as default.
NUMA Distance is also relative when it comes to finding the absolute optimal config, but using the defaults will not affect stability, but will have a very minimal impact on performance. The rule here is to keep you NIC and NUMA distance at 0 to ensure the moist efficient operation. More on the in Darryl’s blog here… Go to the section on NUMA Node Assignment.
Some example configs
Here are some examples where we have 10 core procs with hyper-threading enabled
Below is a correctly configured LBFO switch in Switch Independent mode.
Below is a correctly configured LBFO switch in Switch Dependent (LACP) mode
Below is correctly configure SET switch adapters (i.e. only supports Switch Independent)
i.e. Don’t overthink this. Just optimize VMQ for the team members as shown in the above examples
Note: For HCI or SMB environments, we need to make some considerations for RSS on the physical adapters. If you stick to the same guidelines of VMQ (because VMQ and RSS are the same technology under the hood), you’re on the right track.
Also I’d really like to keep going and assist with other features and/or settings but the purpose of this post is to assist with VMQ. But I won’t go into all of that here.
Let’s wrap this up – If you’re still confused, have some feedback or simply disagree with some of the above then please reach out. Whilst I’ve been in many Hyper-V environments over the years, I haven’t been in them all, so there will be outliers that go against the rules I’ve stated above.
I’ll maybe add to this over time but hopefully this dispels some of the myths and misconceptions out there.