Prerequisite check "CheckActiveFilesAndExecutables" failed

Problem:
While patching the GRID INFRASTRUCTURE home with July2019 RU patch I got this error:

[root@fzppon06vs1n 29708769]# $GRID_HOME/OPatch/opatchauto apply $PATCH_LOCATION -oh $GRID_HOME

OPatchauto session is initiated at Mon Sep  9 11:14:23 2019

System initialization log file is /u01/grid/12.2.0.3/cfgtoollogs/opatchautodb/systemconfig2019-09-09_11-14-29AM.log.

Session log file is /u01/grid/12.2.0.3/cfgtoollogs/opatchauto/opatchauto2019-09-09_11-15-43AM.log
The id for this session is TV9U

Executing OPatch prereq operations to verify patch applicability on home /u01/grid/12.2.0.3
Patch applicability verified successfully on home /u01/grid/12.2.0.3


Bringing down CRS service on home /u01/grid/12.2.0.3
CRS service brought down successfully on home /u01/grid/12.2.0.3


Start applying binary patch on home /u01/grid/12.2.0.3
Failed while applying binary patches on home /u01/grid/12.2.0.3

Execution of [OPatchAutoBinaryAction] patch action failed, check log for more details. Failures:
Patch Target : fzppon06vs1n->/u01/grid/12.2.0.3 Type[crs]
Details: [
---------------------------Patching Failed---------------------------------
Command execution failed during patching in home: /u01/grid/12.2.0.3, host: fzppon06vs1n.
Command failed:  /u01/grid/12.2.0.3/OPatch/opatchauto  apply /u01/oracle/RU_PATCHES/29708769 -oh /u01/grid/12.2.0.3 -target_type cluster -binary -invPtrLoc /u01/grid/12.2.0.3/oraInst.loc -jre /u01/grid/12.2.0.3/OPatch/jre -persistresult /u01/grid/12.2.0.3/OPatch/auto/dbsessioninfo/sessionresult_fzppon06vs1n_crs.ser -analyzedresult /u01/grid/12.2.0.3/OPatch/auto/dbsessioninfo/sessionresult_analyze_fzppon06vs1n_crs.ser
Command failure output: 
==Following patches FAILED in apply:

Patch: /u01/oracle/RU_PATCHES/29708769/29834717
Log: /u01/grid/12.2.0.3/cfgtoollogs/opatchauto/core/opatch/opatch2019-09-09_11-19-51AM_1.log
Reason: Failed during Patching: oracle.opatch.opatchsdk.OPatchException: Prerequisite check "CheckActiveFilesAndExecutables" failed. 

After fixing the cause of failure Run opatchauto resume

]
OPATCHAUTO-68061: The orchestration engine failed.
OPATCHAUTO-68061: The orchestration engine failed with return code 1
OPATCHAUTO-68061: Check the log for more details.
OPatchAuto failed.

OPatchauto session completed at Mon Sep  9 11:20:28 2019
Time taken to complete the session 6 minutes, 6 seconds

 opatchauto failed with error code 42

The environment is two nodes RAC 19c, while the patching worked fine on node1 it failed with above error on node2. 

Neither Restarting the Clusterware or killing the sessions IDs that come out of below commands did help as well:

fuser /u01/grid/12.2.0.3/lib/libsqlplus.so

fuser /u01/grid/12.2.0.3/lib/libasmclntsh19.so

Solution:

Apply the RU patch manually for each patch using opatch instead of opatchauto:

Manual patching of RU patch on GRID HOME:

Note the patch has been extracted under /u01/oracle/RU_PATCHES/ make sure that PATCH_LOCATION variables refers to the location where you extracted the RU binaries.

[As ROOT]

        Run Pre-patching script:

        # export GRID_HOME=/u01/grid/12.2.0.3

# $GRID_HOME/crs/install/rootcrs.sh -prepatch 

Stop clusterware on the local node:

        # crsctl stop cluster

[As GRID OWNER]

        Start patching:

# export PATCH_LOCATION=/u01/oracle/RU_PATCHES/29708769

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local $PATCH_LOCATION/29850993 

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local $PATCH_LOCATION/29851014 

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local $PATCH_LOCATION/29834717 

$GRID_HOME/OPatch/opatch apply -oh $GRID_HOME -local $PATCH_LOCATION/29401763 

[As ROOT]

        Run Post-patching script:

$GRID_HOME/crs/install/rootcrs.sh -postpatch 

Manual patching of RU patch on ORACLE HOME:

[As ORACLE OWNER]

        Run Pre-patching script:

# export PATCH_LOCATION=/u01/oracle/RU_PATCHES/29708769

$PATCH_LOCATION/29850993/custom/scripts/prepatch.sh -dbhome $ORACLE_HOME 

Stop all instances that runs from ORACLE_HOME on the local node:

# srvctl stop instance-d xxx -i xxx

        Start patching:

        # export ORACLE_HOME=/u01/oracle/12.2.0.3

# export PATCH_LOCATION=/u01/oracle/RU_PATCHES/29708769

$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local $PATCH_LOCATION/29850993 

$ORACLE_HOME/OPatch/opatch apply -oh $ORACLE_HOME -local $PATCH_LOCATION/29834717 

        Run Post-patching script:

$PATCH_LOCATION/29850993/custom/scripts/postpatch.sh -dbhome $ORACLE_HOME 

Start back all instances on the local node:

# srvctl start instance-d xxx -i xxx