Wednesday, September 11, 2019

RAC Database Fail To Start Due To "terminating the instance due to error 119" error

On a Linux 7, after system reboot I have an RAC Database failed to start due to this error in the alertlog:

USER (ospid: 13324): terminating the instance due to error 119
Instance terminated by USER, pid = 13324

While investigating I figured out that the SCAN name "rac1-scan" which is used in "remote_listener" initialization parameter is not pingable:
$ ping rac1-scan
ping: unknown host rac1-scan

When checked /etc/resolv.conf file I found the DNS server entry got removed:
$ cat /etc/resolv.conf
# Generated by NetworkManager

# No nameservers found; try putting DNS servers into your
# ifcfg files in /etc/sysconfig/network-scripts like so:

Luckily I always run a script called to keep a backup of the critical system files entries inside one log file, it helps to restore the entries later whenever any bad change happens to these files.

I restored the original entries of /etc/resolv.conf which holds the right DNS server name in my environment then managed to start the RAC DB successfully.

$ cat /etc/resolv.conf
# Generated by NetworkManager

terminating the instance due to error 119 error is mainly related to "remote_listener" initialization parameter setting, if this setting is messed up it can prevent the RAC instance from starting up.

Always keep a copy of /etc/resolv.conf or immune the file from getting reset by the system after system reboot by using the following command:
chattr +i /etc/resolv.conf

You can use this script to help you with keeping the entries of your Linux & Oracle critical files saved somewhere, in case you will have a need to restore them later:

1 comment:

  1. Fantastic post.

    Really enjoyed reading it and it held my attention all the way through! Keep it up.

    Read my Latest Post