Goldengate Process Crashed Due to ORA-00600:[kglLockGranted-illegal], [0x15AE0A410], [2], [3], [0x15AE0A578], [0x10BA3C018], [0x10BA3C0E8]

 Problem:

On a 19.5 RAC DB, Goldengate process crashed and went into ABENDED status due to the below internal error:

ORA-00600: internal error code, arguments: [kglLockGranted-illegal], [0x15AE0A410], [2], [3], [0x15AE0A578], [0x10BA3C018], [0x10BA3C0E8], [], [], [], [], []

Analysis:

"_kgl_cluster_lock_read_mostly" parameter was set years back to improve the cursor caching when the DB was at 11.2.0.3 version, after upgrading the DB to 19c we faced many ORA-00600:[kglLockGranted-illegal] --each time it comes with different arguments, and were always been traced to that parameter. I observed that this parameter stimulates the DB to hold more locks on dictionary tables; the thing that can randomly impacts any internal process/activity that requires Oracle to update those tables in the background and has a need to acquire a lock; I mean by processes/activities like processing AQs, granting/revoking permissions, gathering statistics, goldengate extract process, .... etc

Solution:

If "_kgl_cluster_lock_read_mostly" parameter is set to true, then reset it back to its default value "FALSE" to get rid of ORA-00600:[kglLockGranted-illegal]

SQL> alter system reset "_kgl_cluster_lock_read_mostly" sid='*';

 Database restart is required after resetting the parameter in order to take effect, if you have a RAC setup, then you can do it node by node smoothly.