sap_vm_provision/kubevirt_vm: Make usage of hugepages configurable

[newkit]

Right now, usage of hugepages is mandatory, make it configurable through a flag.

[marcelmamula]

@newkit Even if this is some kind of internal note for work you will do, then it still needs more information than just one sentence.

Please consider how you create these kind of issues in future as this is not productive. @rhmk

[newkit]

When creating a VM, currently the usage of hugepages is mandatory, see here.

Let's add a flag in to the VM definition to switch off the usage of hugepages:

sap_vm_provision_kubevirt_vm_host_specifications_dictionary:
  example_host_specification_plan:
    host1: # Hostname, must be 13 characters or less
      # SMT-2 (i.e. 2 CPU Threads per CPU Core) is default for Intel CPU Hyper-Threading, optionally can be altered to SMT-1
      kubevirt_vm_cpu_smt: 2
      kubevirt_vm_cpu_cores: 2
      kubevirt_vm_memory_gib: 24
      hugepages: false

When omitted / by default hugepages should be on to be backwards compatible.

[sean-freeman]

Please tread very lightly here, as HugePages is an OS capability and therefore in this Ansible Role - the configuration parameter would apply to all platforms and require testing as such (i.e. regression tests across each platform using Red Hat Product Demo System [RHPDS]).

However, HugePages configuration is always tricky. I would actually prefer the current static solution for KubeVirt VM specifications, hiding this from the end-user and keeping to the only official documentation for KVM-based Hypervisors.

From my memory, KubeVirt VM is the only API Spec that accepts a HugePages variable (not sure why, maybe something is being done at the Hypervisor level to align with NUMA Node Topology). All other platforms would anticipate the regular OS commands for this configuration, but advise against it.

Suggest speaking with the IBM Power representative in the SLL-OSI Governance meetings, as there is internal SAP-IBM documentation about performance tuning configurations with HugePages and NUMA.

---

Transparent HugePages (THP) is 100% no-go (see disabling in sap_install > https://github.com/search?q=repo%3Asap-linuxlab%2Fcommunity.sap_install+huge+pages&type=code).

HugePages is not well documented by anyone, particularly to tune that finely you need to start considering NUMA Node Topology of the CPU to DRAM..... which the user won't really know on a Cloud platform, and requires calculations for Hypervisors. See.....

• https://access.redhat.com/sites/default/files/attachments/deploying_sap_hana_on_red_hat_virtualization_3.0_0.pdf
• https://documentation.suse.com/sbp/sap-15/html/SBP-SLES4SAP-HANAonKVM-SLES15SP4/index.html#_sec_memory_backing
• https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/troubleshoot-performance-memory-linux
• https://learn.microsoft.com/en-us/troubleshoot/azure/virtual-machines/linux/linux-vm-no-boot-incorrect-hugepages-configuration
• https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Oracle.Concepts.HugePages.html
• https://repost.aws/knowledge-center/configure-hugepages-ec2-linux-instance

[marcelmamula]

Thank you @sean-freeman, we are already at the point where each platform has its own specific variables and quirks. Which is completely fine and still aligned with same structure across platforms.
This change is minor impact as it adds THP flow only for one platform which is maintained by @newkit and I do not see issue with this #147.

If you have any other concerns, lets us know before we LGTM it.

[newkit]

Hugepages (HPs) improve performance but make allocation / setup of VMs due to the before mentioned caveats less flexible. Nevertheless, for KVM and also for kubevirt we made sure that this works. In sap_hypervisor_node_preconfigure the worker nodes are setup with hugepages in order that VMs can be created with HPs in sap_vm_provision.

This was part of this roles from day 1 on. The enhancement described here wants to give the flexibility to the user whether to use HPs or not at all, because maybe the last bit of performance is not needed in favor of the flexibility.

Hugepages != Transparent Hugepages. While SAP does not support TPHs on the OS level, the usage of hugepages discussed here is on the hypervisor level.

[sean-freeman]

Indeed HP != THP, see https://mpolednik.github.io/2017/06/20/hugepages-and-virtualization

I'll leave you both to discuss with your performance teams about the HP configuration for SAP, but if I am more to the point specific to this Ansible Role:

1. I am concerned about the scope of configuring HugePages for the VM Guest OS, which would apply to all platforms and is generally not documented well because it is a very subjective area of performance tuning. In the current phrasing and parameter naming, it implies that Huge Pages would be configured inside the VM Guest OS and this would impact all platforms

2. I am concerned about clarity of what the parameter will do, as KubeVirt is a 'middleware' of sorts on behalf of the KVM Hypervisor running as a Type 2 Hypervisor atop an OS (e.g. RH CoreOS) therefore - from what I understand the objective here to be - this parameter is specific to the allocation of the Hypervisor Node's DRAM memory, and NOT the virtual memory inside the Guest OS Virtual Machine (as per point 1 above).

   1. Which is why I was referring to the URLs, where the only existing KVM hypervisor documentation does specifically state do not change HugePages from 1GiB. That's why it was statically defined within the Ansible Role

[newkit]

To be clear, the change proposed here does not affect the OS inside the guest. It does not change the default behavior which was, and still is, to used HPs on the hypervisor level for the VM. It gives the flexibility to the user to have the freedom of choice.

I wrote the RH guide for SAP HANA on RHV that you quoted @sean-freeman and benchmarked the RHV setup for SAP HANA. I am aware that using HPs is required for best performance. Hence it stays the default. But there are setups where not having to use HPs can be beneficial, e.g. when there is not plenty of RAM and the VM is coexisting with other VMs / workloads. Would this be supported by SAP? Certainly not, but that's the case for OCPv / all kubvirt VM solutions.

[sean-freeman]

Fine with all that, I'm just sanity checking for scope control. If an end-user sees hugepages: XGiB they could get confused easily.

Whereas the below would be verbose and explicitly clear....

sap_vm_provision_kubevirt_vm_host_specifications_dictionary:
  example_host_specification_plan:
    host1:
      kubevirt_vm_cpu_smt: 32
      kubevirt_vm_cpu_cores: 2
      kubevirt_vm_memory_gib: 256
      # kubevirt_vm_memory_hypervisor_hugepages: false
      # kubevirt_vm_memory_hypervisor_hugepages: 1GiB
      sap_system_type: project_prd
      sap_host_type: hana_primary

[newkit]

Okay, got your point @sean-freeman, thanks for your feedback. I renamed the variable in the PR accordingly.