Wednesday, March 2, 2022

ora_lms Consume Lots of Memory and May Lead to Memory Starvation in RAC 19c


ora_lms process is consuming too much memory in 19c RAC database leading to memory resource starvation and heavily swapping ends up with having Nodes to evict from the cluster.


- Don't try to kill the ora_lms process from OS (thinking it will restart itself), this will terminate the whole DB instance.

- The number of LMS processes is determined by the number of CPUs being used in the server. 

For example:

    n = "number of CPUs reported by the OS and used for CPU_COUNT by default"
         n < 4            => 1 LMS process will be started
         4 <= n < 16 => 2 LMS processes will be started
         n >=16        => 2 LMS + (1 LMS processes for every 32 CPU) will be started.

Reference: Doc ID 1392248.1


This issue is Bug.31969719 and currently being worked by Oracle's Development team.


Implement Huge pages on OS side on all RAC nodes.

How to implement HugePages? here you go:

1- Calculate the number of huge pages using this formula:

vm.nr_hugepages = [SGA_MAX of all instances(KB) Except ASM / HugePageSize_you_willing_to_set(KB)] + 6 

i.e. If the SGA_MAX is 10G then the number of huge pages will be calculated as follows:
vm.nr_hugepages = [(10 x 1024 x 1024)  /  2048]  +  6  =

2- Add
vm.nr_hugepages to /etc/sysctl.conf
# vi

3- Update memlock value for the database user owner in limits.conf: [in KB]

Make sure that memlock size in KB is equal or higher than the size of [HugePages Number x HugePage Size(KB)] =
5126 x 2048 = ‭10498048‬

# vi /etc/security/limits.conf
oracle soft memlock ‭10498048‬
oracle; hard memlock ‭

Verify the setting using this command after you log out and login back into oracle OS user:

# ulimit -l

3- Force the DB instance to use Huge Pages when instance startup:

SQL> alter system set use_large_pages=only scope=spfile sid='*';

4- Enable HugePages and Disable transparent_hugepage in the bootloader:

Edit the bootloader with the number of the HugePages along with the HugePage Size:
For Linux 7 and higher: Use grubby command to modify the bootloader:

[Get the default kernel boot path]
# grubby --default-kernel

[Use the path from the previous output to update the default kernel with HugePages settings]:

# grubby --args="transparent_hugepage=never hugepagesz=2M hugepages=
5126 default_hugepagesz=2M" --update-kernel /boot/vmlinuz-4.1.12-94.3.9.el7uek.x86_64

5- Restart all RAC nodes one by one and make sure the following message in the alertlog has no errors when the instance startup:

 Supported system pagesize(s):
        4K       Configured              11              11        NONE
     2048K            5126            5120            5120       NONE

No comments:

Post a Comment