All about Memory Tiering configuration in VCF 9 / vSphere 9

As the demand physical dynamic random access memory (DRAM) continues to grow, it is required to handle its usage as efficiently as possible. Memory Tiering is the perfect feature by storing cold memory pages on NVME based SSD to extent ESXi-hosts DRAM. This post covers the configuration of this feature including advanced parameters and guidance for usage in Omnissa Horizon VDI environments.

 

Agenda:

1.       Enable Memory Tiering NVME SSD Device

2.      Configure Memory Tiering parameterrs

3.      Advanced Memory Tiering parameters

4.      Verify Memory Tiering

5.      Memory Tiering for VDI (Omnissa Horizon)

 

Enable Memory Tiering NVME SSD Device

Use an SSD which typically is represented as Mixed Used to allow up to 3 Drive writes per Day. Random I/O performance is crucial. So use SSDs with modern SSD controllers of the PCI-E Gen4 or 5 generation.

For homelab builds use a TLC based SSD with an integrated DRAM cache. Modern PCI-E Gen5 SSDs like SanDisk WD SN8100 or Samsung 9100 pro will result in the best performance.

Use esxcli storage core device list to identify the SSD which should be used for Memory Tiering.

Then claim the SSD for Memory Tiering:

esxcli system tierdevice create -d /vmfs/devices/disks/"ID of SSD from step1"

esxcli system tierdevice list should now present a device.

Enable Memory Tiering

esxcli system settings kernel set -s MemoryTiering -v TRUE

A host reboot is required for this change.

 

Configure Memory Tiering parameters

The following configurations are host specific. They can also be configured at scale using vSphere Configuration profiles.

Configure the percentage of Tiered Memory on the SSD device. You can use the following examples as reference. The default value from VMware is 100% which doubles the memory. You can also use values like 50% in case you run RAM intensive workloads like SQL servers.

NVME SSD Disk Partition Size	DRAM-Size	NVMe Tier RAM Size	Mem.TierNvmePct (%)
512 GB	256 GB	256 GB	100
128 GB	256 GB	128 GB	50
512 GB	256 GB	512 GB	200
1,46TiB = 1,6TB	3 TB	0,75 TB	25
1,46TiB = 1,6TB	2,25 TB	0,5625 TB	25
1,46TiB = 1,6TB	1,5 TB	0,375 TB	25
1,46TiB = 1,6TB	3 TB	1,46 TB	50
1,46TiB = 1,6TB	2,25 TB	1,125 TB	50
1,46TiB = 1,6TB	1,5 TB	0,75 TB	50

 

Configure the desired percentage on the host:

esxcli system settings advanced set -o /Mem/TierNvmePct -i 100

Optionally configure encryption of the NVME Tier, which slightly decreases performance:

esxcli system settings advanced set -i 1 -o /Mem/EncryptTierNvme

 

Cluster Wide vSphere Configuration Profiles (Desired State)

vCenter GUI à Cluster à Configure à Desired State à Configuration à Settings à Draft

esx – vmkernel: memory tiering: true
esx – advanced options: tier_nvme_pct: 50
esx – advanced options: encrypt_tier_nvme: 1

 

Advanced Memory Tiering parameters

You can configure Memory Tiering overrides per VM to allow Memory being used or not used for specific VMs:

Disable Memory Tiering:

Advanced Attribute sched.mem.enableTiering: FALSE

Encrypt Memory Tiering for specific VM

sched.mem.EncryptTierNVMe: TRUE

 

Verify Memory Tiering

Configuration Verification can be either performed in the UI or via CLI

UI

vCenter -> ESXi Host à Configure à Hardware à Overview

CLI

esxcli hardware memory get

vsish -e get /memory/tiers/0/info

vsish -e get /memory/tiers/1/info

 

Memory Tiering Usage of each VMs can be checked with the memstats command via ESX CLI:

memstats -r hw-stats

Memstats -r vmtier-stats

Typacally Memory Tiering gets used at 60-70% host DRAM Tier0 usage. On lower levels of usage all VM memory will be stored in Tier0 to optimize for best performance and reduce SSD wear level.

Memory Tiering for VDI (Omnissa Horizon)

VDI is an ideal UseCase for Memory Tiering. There is often a low percentage of really active memory as Windows, Linux and the apps consume a certain amount of RAM but often do not use it all the time.

A Mem.TierNvmePct (%) of 100% is BestPractice as it’s double the memory. For low performance VDI 200% could also be an option.

Please note that Memory Tiering cannot be used with physical GPU activated VMs like NVIDIA vGPU.