Sunday, December 30, 2018

DBA Bundle v4.8 | December 2018 Release

DBA Bundle V4.8 download:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0


What is new in this version:

oracle_cleanup.sh script: - Verify the log/trace files location validity before the deletion of log files.

dbalarm.sh script: - Check the existence of Restore Points when FRA hits the threshold.
                       - Avoid running the script if there is another session already running it.
                       - Fix a bug in the Monitoring Services section.

configuration_baseline.sh script: - Included the Routing table in the OS configuration baseline report.

Fixed minor bugs in dbdailychk.sh & asmdisks.sh scripts.

In case you are new to the DBA Bundle, please visit this article which explains the tool in details:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Tuesday, October 23, 2018

DBA BUNDLE 4.7 released

New features of DBA Bundle 4.7

active_sessions.sh script:
- Show current running RMAN Backups if exist.
- Fixed divide by zero bug when TOTALWORK=0 in v$session_longops

dbalarm.sh script:
- Enclosing OS Network Failure Statistics (Dropped Packets) whenever TIMEOUT errors appear in the DB or Listener ALERTLOG.
- Added RMAN Backup Failure new Check.
- Added the Checking of Running RMAN Backup when CPU hit the threshold

configuration_baseline.sh script:
- Enabled the reporting of NOLOGGING objects in the report.

To download DBA Bundle V4.7:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

This article explains the major features of the DBA BUNDLE:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Tuesday, October 16, 2018

Monitoring Failed RMAN Backup Jobs in dbalarm Shell Script

dbalarm script is now capable of monitoring RMAN backup failures. Just make sure to adjust the interval of the following parameter (in minutes) to the same execution interval of dbalarm.sh in your crontab.

e.g. If you are scheduling dbalarm.sh script to run every 5 minutes in the crontab, the following parameter should be set to 5
LAST_MIN_BKP_CHK=5      # REPORT RMAN Backup FAILURE in the last N MINUTES. Should be similar to the crontab execution interval of dbalarm script. [DB]

By default, dbalarm.sh script will monitor RMAN backup failure, in case you want to change this behavior and stop this feature for any reason, just set the following parameter to N
CHKRMANBKP=N           # Enable/Disable Checking of RMAN Backup FAILURE.      [DB]

To download dbalarm script:

To read more about dbalarm script:

Thanks to Rahul Malode who recommended adding this feature.

Wednesday, September 19, 2018

Bundle V4.6 Is Released

The new features of Bundle V4.6 are:

db_locks.sh script: Replaced the column that checks the wait in seconds from SECONDS_IN_WAIT to use WAIT_TIME_MICRO (which is more accurate) for 11gr2 compatibility.

dbalarm.sh script:
- The BLOCKING SESSION time threshold has been changed so the script will monitor only the blocking sessions that are blocked for more than 1 min.
- For 11g version onwards, The BLOCKING SESSION time threshold will be compared against WAIT_TIME_MICRO column which is more accurate.

sql_id_details.sh script: If the statement was already been vetted by SQL Advisor, the script will automatically show the tuning recommendations.

To download DBA Bundle V4.6:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

To visit the main article that explains all the bundle's features:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Monday, August 27, 2018

DBA Bundle Version 4.5 Is Now Available

DBA Bundle version 4.5 is now available for download.

Enhancements:
- Enhanced the display of the Execution Plan when running sql_id_details.sh using sqlid alias.
- Added REPORT UNRECOVERABLE DATABASE FILES that don't have a valid backup to the Daily Health Check Report [dbdailychk.sh script]

To download DBA Bundle version 4.5:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

To read the full article:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Tuesday, July 24, 2018

DBA Bundle 4.4

DBA Bundle version 4.4 is released. It comes with the following:

Bug fixes:
- Fixed small bugs with the following scripts:
configuration_baseline.sh
schedule_rman_image_copy_bkp.sh
schedule_rman_full_bkp.sh
dbalarm.sh

Enhancements:
- Added new parameter to dbdailychk_html.sh & dbdailychk.sh to control the display of LAST MODIFIED OBJECTS in the Health Check report.

To download DBA Bundle v4.4:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

To read the full article about the bundle features and how to use it:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Saturday, July 7, 2018

Auditing Using LogMiner

Scenario: Let's suppose you want to investigate a recent case on a database whereas auditing is NOT enabled, or it's enabled but it didn't cover the activities you are looking for, in this case, you still have the ability to dig and find those activities using LogMiner feature.

LogMiner is a free feature (No license required) available in both Standard and Enterprise Editions, that can analyze the contents of archive log files and display (SYS, non-SYS, COMMITTED, NON-COMMITTED) transactions that happened during a specific period of time. It can also be used to recover a logical corruption of one or multiple transactions as it also contains the reverse (UNDO) statement of all transactions (similar to flashback transaction query).

I'll start with building the test scenario, a new user will be created and by that user, I'll execute some DDLs and DMLs along with wrong statements to figure out what are the statements that LogMiner can fetch for us.

-- Login as SYSDBA and Create a new user "AA" for testing: <let's see if this CREATE USER statement by SYSDBA will get captured>
06-Jul-2018 15:16:15 SYS@orcl> create user AA identified by "aa#12345";
06-Jul-2018 15:16:30 SYS@orcl> grant connect,resource to aa;

-- Connect to the new user:
06-Jul-2018 15:16:37 SYS@orcl> conn aa/aa#12345
Connected.

06-JUL-18 AA@orcl> create table a (a number);
Table created.

-- execute a wrong insert statement: <let's see if wrong statements are getting captured>
06-JUL-18 AA@orcl> insert into a('1');
insert into a('1')
              *
ERROR at line 1:
ORA-00928: missing SELECT keyword

-- execute a committed insert transaction:
06-JUL-18 AA@orcl> insert into a values (2);
1 row created.

06-JUL-18 AA@orcl> commit;
Commit complete.

-- execute a commited delete transaction:
06-JUL-18 AA@orcl> delete from a;
1 row deleted.

06-JUL-18 AA@orcl> commit;
Commit complete.

-- Execute a commited update tansaction that update nothing: 
06-JUL-18 AA@orcl> update a set a=5 where a=1;
0 rows updated.

-- Execute a truncate table statement:
06-JUL-18 AA@orcl> truncate table a;
Table truncated.

-- Create a procedure with compilation errors:
06-JUL-18 AA@orcl> create or replace procedure p as select * from a;
  2  /
Warning: Procedure created with compilation errors.

-- Execute a non-committed insert statement:
06-JUL-18 AA@orcl> insert into a values (3);
1 row created.

06-JUL-18 AA@orcl> rollback;
Rollback complete.

-- Now I'm done let's check the time at the end of this demo:
06-JUL-18 AA@orcl> !date
Fri Jul  6 15:34:51 GST 2018

And now let's use the LogMiner to reveal the transaction details that happened during that demo by user AA.

Step 1: Turn on the database level Supplemental Logging:

-- First check if the Supplemental Logging is already enabled, if supplemental_log_data_min shows "YES" this means it's ENABLED, then better do NOT run the last step of deactivating it:

SQL> SELECT supplemental_log_data_min FROM v$database;
SUPPLEME
--------
NO

-- supplemental_log_data_min shows "NO", so we have to enable the database level Supplemental Logging:
SQL> ALTER DATABASE ADD SUPPLEMENTAL LOG DATA;

Step 2: Provide the date & time range of the transactions you want to view, try to make the range as short as possible:

SQL> begin
dbms_logmnr.start_logmnr(
starttime => '06-Jul-2018 15:16:00',
endtime => '06-Jul-2018 15:36:00',
options => dbms_logmnr.dict_from_online_catalog + dbms_logmnr.continuous_mine + dbms_logmnr.print_pretty_sql + dbms_logmnr.committed_data_only
);
end;
/

Note: Specifying "dbms_logmnr.committed_data_onlyoption means to look for committed transactions only, if you really interested in fetching rolled back transaction, just remove "dbms_logmnr.committed_data_only" from the list of options.


Step 3: Query V$LOGMNR_CONTENTS to get the information you need:

SQL>    set linesize 170 pages 100
col TIMESTAMP for a20
col sql_redo for a40
col sql_undo for a40
col username for a20
col os_username for a20
col MACHINE_NAME for a20
select xid,to_char(timestamp,'DD-MON-YY HH24:MI:SS') TIMESTAMP,
username,OS_USERNAME,MACHINE_NAME,sql_redo,sql_undo,ROLLBACK from v$logmnr_contents
where SEG_OWNER='AA';

=> sql_redo: the statement it self.
=> sql_undo: the reverse statement that can be used to rollback the original statement.
-- You can also search by Segment Name "SEG_NAME='A'" or by operation "operation in ('INSERT','UPDATE','DELETE')"

XID TIMESTAMP       USERNAME  OS_USERNAME  MACHINE_NAME SQL_REDO
---------------- -------------------- --------- ------------ ------------ ----------------------------------------
SQL_UNDO
----------------------------------------
09001900E39D0000 06-JUL-18 15:16:30   UNKNOWN  UNKNOWN UNKNOWN create user aa identified by  VALUES 'S:
D4D17D1076EC7C6F5B1D68FBCB244E8789691A14
6F5209DF2A51AC579007;E3EE1448DE26764F' ;


08001E002E830000 06-JUL-18 15:16:57   UNKNOWN  UNKNOWN UNKNOWN create table a (a number);


0A002100A8510600 06-JUL-18 15:17:34   UNKNOWN  UNKNOWN UNKNOWN insert into "AA"."A"
  values
"A" = 2;
delete from "AA"."A"
 where
    "A" = 2 and
    ROWID = 'AAAjrvAAEAAAACvAAA';

09000600E09D0000 06-JUL-18 15:17:43   UNKNOWN  UNKNOWN UNKNOWN delete from "AA"."A"
  where
"A" = 2 and
ROWID = 'AAAjrvAAEAAAACvAAA';
insert into "AA"."A"
 values
    "A" = 2;

05000D00AC800000 06-JUL-18 15:18:21   UNKNOWN  UNKNOWN UNKNOWN truncate table a;


0A0007004D510600 06-JUL-18 15:18:45   UNKNOWN  UNKNOWN UNKNOWN create or replace procedure p as select
* from a;;
6 rows selected.

As you can see, LogMiner fetches all kind of statements that made changes to the database including DML & DDLs along with the statements executed with SYSDBA privilege.
Any other statements that don't make any change to the database like SELECT and failed SQL statements along with the DML statements that didn't make any actual change to the data will not be shown by the LogMiner.

Once you are done:

- End the Log Miner Session and flush the contents of v$logmnr_contents:
SQL> execute dbms_logmnr.end_logmnr();
SQL> select * from v$logmnr_contents;
No Rows should be returned.

-Deactivate the database level Supplemental Logging:
In case the SUPPLEMENTAL LOG DATA was already enabled in step 1, then don't execute this step, replication software like Oracle Streams & golden gate depends on the SUPPLEMENTAL LOG DATA in order to perform the replication.

SQL> ALTER DATABASE drop SUPPLEMENTAL LOG DATA;

SQL> SELECT supplemental_log_data_min,supplemental_log_data_pk,supplemental_log_data_ui,supplemental_log_data_fk,supplemental_log_data_all FROM v$database;

LogMiner cannot work as an alternative to Auditing, LogMiner can only fetch the changes that happened to the database, unlike Auditing that can capture details like SELECT statements and login details, such information is not available in the Archivelogs to be mined.
Moreover, LogMiner can analyze only the available archive logs, it cannot go far in the past where the archive logs are not available, unless you restore the archivelogs from a backup media and feed them into the LogMiner.

To sum up. LogMiner is mainly used by the DBAs to recover logical data corruption, but we can use it as a forensic tool.

References:
https://oracle-base.com/articles/8i/logminer
https://asktom.oracle.com/pls/asktom/f?p=100:11:0::::P11_QUESTION_ID:433619656232

Thursday, May 17, 2018

DBA Bundle v4.3 Is Now Released

Version 4.3 is now available. It's packed up with the following features:

New Scripts:
goldengate_lag_mon.sh For monitoring goldengate LAG, more details: http://dba-tips.blogspot.ae/2018/05/linux-shell-script-to-monitor-and.html

New Enhancements:
- gather_stats.sh script for gathering statistics can now run in the background (nohup mode).
- Paranoid mode feature has been added to dbalarm.sh script for reporting EXPORT/IMPORT, ALTER SYSTEM, ALTER DATABASE, instance STARTUP/SHUTDOWN and other DB Major activities, for more details on this feature: http://dba-tips.blogspot.ae/2018/05/dbalarm-script-for-monitoring-oracle-db.html
- oracle_cleanup.sh script for cleaning up database & listener logs is now giving the option to the user to archive Audit Log files as well.
- Added more parameters to dbdailychk.shdbdailychk_html.sh (HTML version) to let the user have more control over the report contents (e.g. including/excluding SQL Tuning Advisor, Memory Advisor, Segment Advisor, Job details and credentials changes results in/from the health check report).

To download DBA Bundle v4.3:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

To read the full article about the bundle features and how to use it:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Monday, May 14, 2018

Linux Shell Script To Monitor and Report GoldenGate Lag

photographer's Twenty20 Philipbrunner
This script can monitor GoldenGate lag whenever it happens based on the per-defined LAG threshold inside the script.
It's highly recommended to deploy this script on all (source & destination) replication servers in order to detect the lag on all processes (Extract, Pump, and Replicate).

This script is not designed to monitor the replicated data inside the tables it totally relies on the native GoldenGate GGSCI console.

This script should be executed/scheduled by the GoldenGate installation owner OS user.                                               

How it works:

First, Download the script:
https://www.dropbox.com/s/l4dqzicviuaawt6/goldengate_lag_mon.sh?dl=0

Second, Adjust the following parameters:

MAIL_LIST="youremail@yourcompany.com"
Replace "youremail@yourcompany.com" pattern with your e-mail.


# ###########################################
# Mandatory Parameters To Be Set By The User:
# ###########################################
ORACLE_HOME= # ORACLE_HOME path of the database where GoldenGate is running against.
GG_HOME=           # GoldenGate Installation Home path. e.g. GG_HOME=/goldengate/gghome

Please note that ORACLE_HOME & GG_HOME are mandatory to be adjusted by YOU, in case you missed setting them up, the script will automatically try to guess the right values, but this will not be accurate most of the times.


# ################
# Script Settings:
# ################
# LAG THRESHOLD in minutes: [If reached an e-mail alert will be sent. Default 10 minutes]
LAG_IN_MINUTES=10

Here you define the LAG threshold in minutes (it's 10 minutes by default). Whereas if the lag reached 10 minutes it will send you an email.


# Excluded Specific PROCESSES NAME:
# e.g. If you want to exclude two replicate processes with names REP_11 and REP_12 from being reported then add them to below parameter as shown:
# EXL_PROC_NAME="DONOTREMOVE|REP_11|REP_12"
EXL_PROC_NAME="DONOTREMOVE"

In case you want to exclude specific (Extract, Pump, or Replicat) processes, let's say you want to exclude a process you use it for testing the replication, you can add it to the above parameter as shown in the blue color example.

DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".

GitHub version:
# ####################################################################################################################
# This Script Detects GOLDENGATE LAG If The LAG Hits the Pre-defined Threshold
# [Ver 1.0]
#
# To get this script work you need to Define the following variables:
# ORACLE_HOME # Must be set to the ORACLE_HOME path of the database where GoldenGate is running against.
# GG_HOME # Should be set to the Goldengate installation home directory path.
# LAG=xxxx # The number of minutes of lag, if reached an email alert will be sent [10 minutes is the default].
# EXL_PROC_NAME="DONOTREMOVE|REP11|REP12" In case you want to exclude specific processes e.g. REP11 & REP12
# LOG_DIR # The location of script logs [/tmp by default].
#
# # # #
# Author: Mahmmoud ADEL # # # # ###
# Created: 26-02-17 # # # # #
# Modified:
#
#
#
#
#
#
#
#
#
#
#
#
#
# ####################################################################################################################
MAIL_LIST="youremail@yourcompany.com"
export MAIL_LIST
case ${MAIL_LIST} in "youremail@yourcompany.com")
echo
echo "******************************************************************"
echo "Buddy! You forgot to edit line# 30 in the script."
echo "Please replace youremail@yourcompany.com with your E-mail address."
echo "******************************************************************"
echo
esac
SERVER_NAME=`uname -n`
export SERVER_NAME
# ###########################################
# Mandatory Parameters To Be Set By The User:
# ###########################################
ORACLE_HOME= # ORACLE_HOME path of the database where GoldenGate is running against.
GG_HOME= # GoldenGate Installation Home path. e.g. GG_HOME=/goldengate/gghome
# ################
# Script Settings:
# ################
# LAG THRESHOLD in minutes: [If reached an e-mail alert will be sent. Default 10 minutes]
LAG_IN_MINUTES=10
# Excluded Specific PROCESSES NAME:
# e.g. If you want to exclude two replicate processes with names REP_11 and REP_12 from being reported then add them to below parameter as shown:
# EXL_PROC_NAME="DONOTREMOVE|REP_11|REP_12"
EXL_PROC_NAME="DONOTREMOVE"
# ###############
# VARIABLES:
# ###############
LOG_DIRECTORY=/tmp # Log Location
LAG=$((LAG_IN_MINUTES * 100))
export LAG
export EXL_PROC_NAME
# #############################################
# Finding GoldenGate Installation Home Location: [In case the user didn't set it up above]
# #############################################
if [ ! -f ${GG_HOME}/ggsci ]
then
GG_PID=`pgrep -lf mgr.prm|awk '{print $1}'`
export GG_PID
GG_HOME=`pwdx ${GG_PID}|awk '{print $NF}'`
export GG_HOME
fi
if [ ! -f ${GG_HOME}/ggsci ]
then
GG_HOME=`ps -ef|grep "./mgr"|grep -v grep|awk '{print $10}'|sed -e 's/\/dirprm\/mgr\.prm//g'|grep -v sed|grep -v "//g"|tail -1`
export GG_HOME
fi
if [ ! -f ${GG_HOME}/ggsci ]
then
echo "The script cannot find GoldenGate installation home path, please export it inside the script just before \"VARIABLES\" section"
echo "e.g."
echo "export GG_HOME=/u01/goldengate"
fi
# ###############
# Script Engine:
# ###############
# ###################
# Getting ORACLE_HOME: [In case the user didn't set it up above]
# ###################
if [ -z ${ORACLE_SID} ]
then
ORACLE_SID=`ps -ef|grep pmon|grep -v grep|awk '{print $NF}'|sed -e 's/ora_pmon_//g'|grep -v sed|grep -v "s///g"|tail -1`
export ORACLE_SID
fi
ORA_USER=`ps -ef|grep ${ORACLE_SID}|grep pmon|grep -v grep|awk '{print $1}'|tail -1`
USR_ORA_HOME=`grep ${ORA_USER} /etc/passwd| cut -f6 -d ':'|tail -1`
# SETTING ORATAB:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
export ORATAB
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
export ORATAB
fi
# ATTEMPT2: If ORACLE_HOME not found get it from oratab file:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
## If OS is Linux:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
ORACLE_HOME=`grep -v '^\#' $ORATAB | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
ORACLE_HOME=`grep -v '^\#' $ORATAB | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
fi
fi
# ATTEMPT3: If ORACLE_HOME is still not found, search for the environment variable: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`env|grep -i ORACLE_HOME|sed -e 's/ORACLE_HOME=//g'`
export ORACLE_HOME
#echo "ORACLE_HOME from environment is ${ORACLE_HOME}"
fi
# ATTEMPT4: If ORACLE_HOME is not found in the environment search user's profile: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`grep -h 'ORACLE_HOME=\/' $USR_ORA_HOME/.bash_profile $USR_ORA_HOME/.*profile | perl -lpe'$_ = reverse' |cut -f1 -d'=' | perl -lpe'$_ = reverse'|tail -1`
export ORACLE_HOME
fi
# ATTEMPT5: If ORACLE_HOME is still not found, search for orapipe: [Least accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`locate -i orapipe|head -1|sed -e 's/\/bin\/orapipe//g'`
export ORACLE_HOME
fi
# TERMINATE: If all above attempts failed to get ORACLE_HOME location, EXIT the script:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
echo "Please export ORACLE_HOME variable inside this script in order to get it run properly."
echo "e.g."
echo "export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1"
mail -s "goldengate_lag_mon script failed on Server [${SRV_NAME}] due to the failure of finding ORACLE_HOME, Please export ORACLE_HOME variable inside the script" ${MAIL_LIST} < /dev/null
exit
fi
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib
echo LD_LIBRARY_PATH is: $LD_LIBRARY_PATH
# ################################################
# Checking the LAG status from Goldengate Console:
# ################################################
for GREP_SERVICE in EXTRACT REPLICAT
do
export GREP_SERVICE
export LOG_DIR=${LOG_DIRECTORY}
export LOG_FILE=${LOG_DIR}/${GREP_SERVICE}_lag_mon.log
# Identify lagging operation name:
case ${GREP_SERVICE} in
"REPLICAT") LAST_COL_OPNAME="RECEIVING"
export LAST_COL_OPNAME
BFR_LAST_COL_OPNAME="APPLYING"
export BFR_LAST_COL_OPNAME
;;
"EXTRACT") LAST_COL_OPNAME="SENDING"
export LAST_COL_OPNAME
BFR_LAST_COL_OPNAME="EXTRACTING"
export BFR_LAST_COL_OPNAME
;;
esac
$GG_HOME/ggsci << EOF |grep "${GREP_SERVICE}" > ${LOG_FILE}
info all
exit
EOF
# ################################
# Email Notification if LAG Found:
# ################################
for i in `cat ${LOG_FILE}|egrep -v ${EXL_PROC_NAME}|awk '{print $NF}'|sed -e 's/://g'`
do
if [ $i -ge ${LAG} ]
then
mail -s "Goldengate LAG detected in ${LAST_COL_OPNAME} TRAIL FILES on Server [${SERVER_NAME}]" ${MAIL_LIST} < ${LOG_FILE}
echo "Goldengate LAG detected in ${LAST_COL_OPNAME} TRAIL FILES on Server [${SERVER_NAME}]"
fi
done
for i in `cat ${LOG_FILE}|egrep -v ${EXL_PROC_NAME}|awk '{print $(NF-1)}'|sed -e 's/://g'`
do
if [ $i -ge ${LAG} ]
then
mail -s "Goldengate LAG detected in ${BFR_LAST_COL_OPNAME} TRAIL FILES on Server [${SERVER_NAME}]" ${MAIL_LIST} < ${LOG_FILE}
echo "Goldengate LAG detected in ${BFR_LAST_COL_OPNAME} TRAIL FILES on Server [${SERVER_NAME}]"
fi
done
done
# #############
# END OF SCRIPT
# #############
# REPORT BUGS to: mahmmoudadel@hotmail.com
# DOWNLOAD THE LATEST VERSION OF DATABASE ADMINISTRATION BUNDLE FROM:
# http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html
# DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".

Monday, May 7, 2018

dbalarm Script for Monitoring Oracle DB Server is Now Packed With New Features

I've introduced a new mode called "Paranoid" mode to the dbalarm script (which is used to monitor the Oracle database and listener logs along with monitoring locks, Space, and other events). When the paranoid  mode gets enabled, the script will monitor the following additional activities/events on the Oracle database:

- EXPORT/IMPORT activities whenever they start.
- ALTER SYSTEM commands.
- ALTER DATABASE commands.
- Instance STARTUP/SHUTDOWN events.
- Other Major DB activities.

This mode is disabled by default to let the notifications focused on reporting errors. In order to enable this mode, you have to set PARANOIDMODE parameter from N to Y:

# #########################
# THRESHOLDS:
# #########################
# Modify the THRESHOLDS to the value you prefer:
...

PARANOIDMODE=Y    # Paranoid mode will report more events like export/import, instance shutdown/startup. [Default Disabled]

To download and read the full list of features of dbalarm script, please visit:
http://dba-tips.blogspot.ae/2014/02/database-monitoring-script-for-ora-and.html

Thursday, May 3, 2018

Gather Statistics Shell Script Can Now Run In nohup Mode | Oracle On Linux

I've re-coded the gather statistics shell script for Oracle to run in nohup mode by default, thereby allowing the DBA to close the session without interrupting the gather statistics operation.

Script download link:
https://www.dropbox.com/s/fku4hf082xah1bm/gather_stats.sh?dl=0

For more details on how to use the script, please visit the original post:
http://dba-tips.blogspot.com/2014/09/script-to-ease-gathering-statistics-on.html

And here is the GitHub version:
# #################################################################################################################
# V. 2.2
# Backup & Gather Statistics On SCHEMA|TABLE.
# To be run by ORACLE user
# # # #
# Author: Mahmmoud ADEL # # # # ###
# Created: 02-02-2014 # # # # #
# Modified: 02-05-2018 Re-coded the script to run the gather stats commands in the background [nohup mode].
# 14-01-2021 Checking the statistics LOCK status before gathering statistics.
# 31-05-2021 Added the option of gathering STALE statistics only instead of gathering all statistics.
# #################################################################################################################
# ###########
# Description:
# ###########
echo
echo "======================================================="
echo "This script Gather & Backup Statistics on SCHEMA|TABLE."
echo "======================================================="
echo
sleep 1
# #######################################
# Excluded INSTANCES:
# #######################################
# Here you can mention the instances the script will IGNORE and will NOT run against:
# Use pipe "|" as a separator between each instance name.
# e.g. Excluding: -MGMTDB, ASM instances:
EXL_DB="\-MGMTDB|ASM|APX" #Excluded INSTANCES [Will not get reported offline].
# ##############################
# SCRIPT ENGINE STARTS FROM HERE ............................................
# ##############################
# ###########################
# Listing Available Databases:
# ###########################
# Count Instance Numbers:
INS_COUNT=$( ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|wc -l )
# Exit if No DBs are running:
if [ $INS_COUNT -eq 0 ]
then
echo No Database Running !
exit
fi
# If there is ONLY one DB set it as default without prompt for selection:
if [ $INS_COUNT -eq 1 ]
then
export ORACLE_SID=$( ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|awk '{print $NF}'|sed -e 's/ora_pmon_//g'|grep -v sed|grep -v "s///g" )
# If there is more than one DB ASK the user to select:
elif [ $INS_COUNT -gt 1 ]
then
echo
echo "Select the ORACLE_SID:[Enter the number]"
echo ---------------------
select DB_ID in $( ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|awk '{print $NF}'|sed -e 's/ora_pmon_//g'|grep -v sed|grep -v "s///g" )
do
integ='^[1-9]+$'
if ! [[ ${REPLY} =~ ${integ} ]] || [ ${REPLY} -gt ${INS_COUNT} ]
then
echo
echo "Error: Not a valid number!"
echo
echo "Enter a valid NUMBER from the displayed list !: i.e. Enter a number from [1 to ${INS_COUNT}]"
echo "-----------------------------------------------"
else
export ORACLE_SID=$DB_ID
echo
printf "`echo "Selected Instance: ["` `echo -e "\033[33;5m${DB_ID}\033[0m"` `echo "]"`\n"
echo
break
fi
done
fi
# Exit if the user selected a Non Listed Number:
if [ -z "${ORACLE_SID}" ]
then
echo "You've Entered An INVALID ORACLE_SID"
exit
fi
# #########################
# Getting ORACLE_HOME
# #########################
ORA_USER=`ps -ef|grep ${ORACLE_SID}|grep pmon|grep -v grep|egrep -v ${EXL_DB}|grep -v "\-MGMTDB"|awk '{print $1}'|tail -1`
USR_ORA_HOME=`grep ${ORA_USER} /etc/passwd| cut -f6 -d ':'|tail -1`
# SETTING ORATAB:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
export ORATAB
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
export ORATAB
fi
# ATTEMPT1: Get ORACLE_HOME using pwdx command:
export PGREP=`which pgrep`
export PWDX=`which pwdx`
if [[ -x ${PGREP} ]] && [[ -x ${PWDX} ]]
then
PMON_PID=`pgrep -lf _pmon_${ORACLE_SID}|awk '{print $1}'`
export PMON_PID
ORACLE_HOME=`pwdx ${PMON_PID}|awk '{print $NF}'|sed -e 's/\/dbs//g'`
export ORACLE_HOME
fi
# ATTEMPT2: If ORACLE_HOME not found get it from oratab file:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
## If OS is Linux:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
ORACLE_HOME=`grep -v '^\#' $ORATAB | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
ORACLE_HOME=`grep -v '^\#' $ORATAB | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
fi
#echo "ORACLE_HOME from oratab is ${ORACLE_HOME}"
fi
# ATTEMPT3: If ORACLE_HOME is still not found, search for the environment variable: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`env|grep -i ORACLE_HOME|sed -e 's/ORACLE_HOME=//g'`
export ORACLE_HOME
#echo "ORACLE_HOME from environment is ${ORACLE_HOME}"
fi
# ATTEMPT4: If ORACLE_HOME is not found in the environment search user's profile: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`grep -h 'ORACLE_HOME=\/' $USR_ORA_HOME/.bash_profile $USR_ORA_HOME/.*profile | perl -lpe'$_ = reverse' |cut -f1 -d'=' | perl -lpe'$_ = reverse'|tail -1`
export ORACLE_HOME
#echo "ORACLE_HOME from User Profile is ${ORACLE_HOME}"
fi
# ATTEMPT5: If ORACLE_HOME is still not found, search for orapipe: [Least accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`locate -i orapipe|head -1|sed -e 's/\/bin\/orapipe//g'`
export ORACLE_HOME
#echo "ORACLE_HOME from orapipe search is ${ORACLE_HOME}"
fi
# TERMINATE: If all above attempts failed to get ORACLE_HOME location, EXIT the script:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
echo "Please export ORACLE_HOME variable in your .bash_profile file under oracle user home directory in order to get this script to run properly"
echo "e.g."
echo "export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1"
exit
fi
export LD_LIBRARY_PATH=${ORACLE_HOME}/lib
# ########################################
# Exit if the user is not the Oracle Owner:
# ########################################
CURR_USER=`whoami`
if [ ${ORA_USER} != ${CURR_USER} ]; then
echo ""
echo "You're Running This Sctipt with User: \"${CURR_USER}\" !!!"
echo "Please Run This Script With The Right OS User: \"${ORA_USER}\""
echo "Script Terminated!"
exit
fi
# ########################
# Getting ORACLE_BASE:
# ########################
# Get ORACLE_BASE from user's profile if not set:
if [ -z "${ORACLE_BASE}" ]
then
ORACLE_BASE=`grep 'ORACLE_BASE=\/' $USR_ORA_HOME/.*profile | perl -lpe'$_ = reverse' |cut -f1 -d'=' | perl -lpe'$_ = reverse'|tail -1`
export ORACLE_BASE
fi
# #########################
# SQLPLUS Section:
# #########################
# PROMPT FOR VARIABLES:
# ####################
if [ ! -d ${USR_ORA_HOME} ]
then
export USR_ORA_HOME=/tmp
fi
GATHERSTATSSCRIPT=${USR_ORA_HOME}/gather_stats_script_DBA_BUNDLE.sql
GATHERSTATSSCRIPTRUNNER=${USR_ORA_HOME}/gather_stats_script_DBA_BUNDLE.sh
GATHERSTATSSPOOL=${USR_ORA_HOME}/gather_stats_script_DBA_BUNDLE.log
STATS_TABLE=BACKUP_STATS
STATS_OWNER=SYS
STATS_TBS=SYSTEM
VAL33=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT STATUS FROM V\$INSTANCE;
EOF
)
VAL44=`echo $VAL33| awk '{print $NF}'`
case ${VAL44} in
"OPEN") echo ;;
*) echo;echo "ERROR: INSTANCE [${ORACLE_SID}] IS IN STATUS: ${VAL44} !"
echo; echo "PLEASE OPEN THE INSTANCE [${ORACLE_SID}] AND RE-RUN THIS SCRIPT.";echo; exit ;;
esac
echo "Enter the SCHEMA NAME/TABLE OWNER:"
echo "=================================="
while read SCHEMA_NAME
do
if [ -z ${SCHEMA_NAME} ]
then
echo
echo "Enter the SCHEMA NAME/TABLE OWNER:"
echo "=================================="
else
VAL11=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT COUNT(*) FROM DBA_USERS WHERE USERNAME=upper('${SCHEMA_NAME}');
EOF
)
VAL22=`echo ${VAL11}| awk '{print $NF}'`
if [ ${VAL22} -eq 0 ]
then
echo
echo "ERROR: USER [${SCHEMA_NAME}] IS NOT EXIST ON DATABASE [${ORACLE_SID}] !"
echo
echo "Enter the SCHEMA NAME:"
echo "====================="
else
break
fi
fi
done
echo
echo "Enter the TABLE NAME: [BLANK VALUE MEANS GATHER THE WHOLE [${SCHEMA_NAME}] SCHEMA STATISTICS]"
echo "===================="
while read TABLE_NAME
do
if [ -z ${TABLE_NAME} ]
then
echo
echo "Confirm GATHERING STATISTICS ON WHOLE [${SCHEMA_NAME}] SCHEMA? [Y|N] [Y]"
echo "===================================================="
while read ANS
do
case $ANS in
""|y|Y|yes|YES|Yes) echo "GATHERING STATISTICS ON SCHEMA [${SCHEMA_NAME}] ..."
echo
echo "GATHER HISTOGRAMS ALONG WITH STATISTICS? [Y|N] [N]"
echo "======================================="
while read ANS1
do
case $ANS1 in
y|Y|yes|YES|Yes) HISTO="FOR ALL COLUMNS SIZE SKEWONLY";HISTOMSG="(+HISTOGRAMS)"; break ;;
""|n|N|no|NO|No) HISTO="FOR ALL COLUMNS SIZE 1"; break ;;
*) echo "Please enter a VALID answer [Y|N]" ;;
esac
done
echo
printf "`echo "GATHER STATISTICS FOR"` `echo -e "\033[33;5mSTALE\033[0m"` `echo "STATISTCS ONLY? [Y|N] [N]"`\n"
echo "==========================================="
while read ANS2
do
case $ANS2 in
y|Y|yes|YES|Yes) STALE_OPTION="OPTIONS => 'GATHER STALE',"; export STALE_OPTION; break ;;
""|n|N|no|NO|No) STALE_OPTION=""; export STALE_OPTION; break ;;
*) echo "Please enter a VALID answer [Y|N]" ;;
esac
done
# Check The Existence of BACKUP STATS TABLE:
VAL1=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT COUNT(*) FROM DBA_TABLES WHERE OWNER=upper('${STATS_OWNER}') AND TABLE_NAME=upper('${STATS_TABLE}');
EOF
)
VAL2=`echo $VAL1| awk '{print $NF}'`
if [ ${VAL2} -gt 0 ]
then
echo
echo "STATISTICS BACKUP TABLE [${STATS_OWNER}.${STATS_TABLE}] IS ALREADY EXISTS."
else
echo
${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
SET LINESIZE 157
SET PAGESIZE 5000
SET HEADING OFF
SET VERIFY OFF
PROMPT CREATING STATS TABLE [Holds a backup of original statistics, if you want to restore later]...
BEGIN
dbms_stats.create_stat_table (
ownname => upper('${STATS_OWNER}'),
tblspace => upper('${STATS_TBS}'),
stattab => upper('${STATS_TABLE}'));
END;
/
PROMPT
EOF
fi
echo "spool ${GATHERSTATSSPOOL}" > ${GATHERSTATSSCRIPT}
echo "PROMPT BACKING UP CURRENT STATISTICS OF SCHEMA [${SCHEMA_NAME}] ..." >>${GATHERSTATSSCRIPT}
echo "BEGIN" >>${GATHERSTATSSCRIPT}
echo "DBMS_STATS.EXPORT_SCHEMA_STATS (" >>${GATHERSTATSSCRIPT}
echo "ownname => upper('${SCHEMA_NAME}')," >>${GATHERSTATSSCRIPT}
echo "statown => upper('${STATS_OWNER}')," >>${GATHERSTATSSCRIPT}
echo "stattab => upper('${STATS_TABLE}'));" >>${GATHERSTATSSCRIPT}
echo "END;" >>${GATHERSTATSSCRIPT}
echo "/" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT GATHERING STATISTICS ${HISTOMSG} ON SCHEMA [${SCHEMA_NAME}] ..." >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT Feel free to exist this session any time as this script is running in the background :-)" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT To check the progress:" >>${GATHERSTATSSCRIPT}
echo "PROMPT tail -f ${GATHERSTATSSPOOL}" >>${GATHERSTATSSCRIPT}
echo "BEGIN" >>${GATHERSTATSSCRIPT}
echo "DBMS_STATS.GATHER_SCHEMA_STATS (" >>${GATHERSTATSSCRIPT}
echo "ownname => upper('${SCHEMA_NAME}')," >>${GATHERSTATSSCRIPT}
echo "METHOD_OPT => '${HISTO}'," >>${GATHERSTATSSCRIPT}
echo "DEGREE => DBMS_STATS.AUTO_DEGREE," >>${GATHERSTATSSCRIPT}
echo "${STALE_OPTION}" >>${GATHERSTATSSCRIPT}
echo "estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE);" >>${GATHERSTATSSCRIPT}
echo "END;" >>${GATHERSTATSSCRIPT}
echo "/" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT (IN CASE THE NEW STATISTICS ARE PERFORMING BAD, RESTORE BACK THE ORIGINAL STATISTICS USING THE FOLLOWING SQL COMMAND):" >>${GATHERSTATSSCRIPT}
echo "PROMPT >>>>" >>${GATHERSTATSSCRIPT}
echo "PROMPT EXEC DBMS_STATS.IMPORT_SCHEMA_STATS (ownname => upper('${SCHEMA_NAME}'), statown => upper('${STATS_OWNER}'), stattab => upper('${STATS_TABLE}'));;">>${GATHERSTATSSCRIPT}
echo "PROMPT >>>>" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT SCRIPT COMPLETED!" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "spool off" >>${GATHERSTATSSCRIPT}
echo "exit" >>${GATHERSTATSSCRIPT}
echo "#!/bin/bash" > ${GATHERSTATSSCRIPTRUNNER}
echo "nohup ${ORACLE_HOME}/bin/sqlplus \"/ as sysdba\" @${GATHERSTATSSCRIPT} | tee ${GATHERSTATSSPOOL} 2>&1 &" >>${GATHERSTATSSCRIPTRUNNER}
chmod 740 ${GATHERSTATSSCRIPTRUNNER}
echo ""
source ${GATHERSTATSSCRIPTRUNNER}
exit 1 ;;
n|N|no|NO|No) echo; echo "Enter the TABLE NAME:";echo "====================";break ;;
*) echo "Please enter a VALID answer [Y|N]" ;;
esac
done
else
# Check The Existence of ENTERED TABLE:
VAL1=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT COUNT(*) FROM DBA_TABLES WHERE OWNER=upper('${SCHEMA_NAME}') AND TABLE_NAME=upper('${TABLE_NAME}');
EOF
)
VAL2=`echo $VAL1| awk '{print $NF}'`
if [ ${VAL2} -eq 0 ]
then
echo
echo "ERROR: TABLE [${SCHEMA_NAME}.${TABLE_NAME}] IS NOT EXIST !"
echo;echo "Enter the TABLE NAME: [BLANK VALUE MEANS GATHER THE WHOLE SCHEMA [${SCHEMA_NAME}] STATISTICS]"
echo "===================="
else
${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
SET LINESIZE 157
SET PAGESIZE 5000
SELECT TABLE_NAME,to_char(LAST_ANALYZED, 'DD-MON-YYYY HH24:MI:SS')LAST_STATISTICS_DATE FROM DBA_TABLES WHERE TABLE_NAME=upper('${TABLE_NAME}');
EOF
break
fi
fi
done
# Check Lock status on Statistics:
LOCK_STATS_RAW=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT COUNT(*) FROM DBA_TAB_STATISTICS WHERE OWNER=upper('${SCHEMA_NAME}') AND TABLE_NAME=upper('${TABLE_NAME}') and STATTYPE_LOCKED='ALL';
EOF
)
LOCK_STATS=`echo ${LOCK_STATS_RAW} | perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'`
if [ "${LOCK_STATS}" -eq 1 ]
then
echo
printf "`echo "Statistics are "` `echo -e "\033[33;5mLOCKED\033[0m"` `echo " on TABLE [${SCHEMA_NAME}.${TABLE_NAME}]"`\n"
echo
echo "NOTE: Statistics will be UNLOCKED temporarily during the statistics gather and will be LOCKED after."
export UNLOCK_STATS_MSG="PROMPT UNLOCKING STATISTICS ON TABLE [${SCHEMA_NAME}.${TABLE_NAME}]"
export UNLOCK_STATS_SQL="exec dbms_stats.unlock_table_stats('${SCHEMA_NAME}','${TABLE_NAME}');"
export LOCK_STATS_MSG="PROMPT LOCKING BACK STATISTICS ON TABLE [${SCHEMA_NAME}.${TABLE_NAME}]"
export LOCK_STATS_SQL="exec dbms_stats.lock_table_stats('${SCHEMA_NAME}','${TABLE_NAME}');"
break
fi
echo
echo "GATHER HISTOGRAMS ALONG WITH STATISTICS? [Y|N] [N]"
echo "======================================="
while read ANS1
do
case $ANS1 in
y|Y|yes|YES|Yes) HISTO="FOR ALL COLUMNS SIZE SKEWONLY"; HISTOMSG="(+HISTOGRAMS)";break ;;
""|n|N|no|NO|No) HISTO="FOR ALL COLUMNS SIZE 1"; break ;;
*) echo "Please enter a VALID answer [Y|N]" ;;
esac
done
echo
echo "GATHER STATISTICS ON ALL TABLE's INDEXES? [Y|N] [Y]"
echo "========================================="
while read ANS2
do
case $ANS2 in
""|y|Y|yes|YES|Yes) CASCD="TRUE";CASCMSG="AND ITS INDEXES"; break ;;
n|N|no|NO|No) CASCD="FALSE"; break ;;
*) echo "Please enter a VALID answer [Y|N]" ;;
esac
done
# Execution of SQL Statement:
# ##########################
VAL1=$(${ORACLE_HOME}/bin/sqlplus -s '/ as sysdba' << EOF
set pages 0 feedback off;
SELECT COUNT(*) FROM DBA_TABLES WHERE OWNER=upper('${STATS_OWNER}') AND TABLE_NAME=upper('${STATS_TABLE}');
EOF
)
VAL2=`echo $VAL1| awk '{print $NF}'`
if [ ${VAL2} -gt 0 ]
then
echo
echo "BACKUP STATS TABLE [${STATS_OWNER}.${STATS_TABLE}] IS ALREADY EXISTS."
else
${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
SET LINESIZE 157
SET PAGESIZE 5000
SET HEADING OFF
SET VERIFY OFF
PROMPT CREATING BACKUP STATS TABLE ...
BEGIN
dbms_stats.create_stat_table (
ownname => upper('${STATS_OWNER}'),
tblspace => upper('${STATS_TBS}'),
stattab => upper('${STATS_TABLE}'));
END;
/
PROMPT
EOF
fi
echo "SET LINESIZE 157" > ${GATHERSTATSSCRIPT}
echo "SET PAGESIZE 5000" >>${GATHERSTATSSCRIPT}
echo "SET HEADING OFF" >>${GATHERSTATSSCRIPT}
echo "spool ${GATHERSTATSSPOOL}" >>${GATHERSTATSSCRIPT}
echo "${UNLOCK_STATS_MSG}" >>${GATHERSTATSSCRIPT}
echo "${UNLOCK_STATS_SQL}" >>${GATHERSTATSSCRIPT}
echo "PROMPT BACKING UP CURRENT STATISTICS OF TABLE [${SCHEMA_NAME}.${TABLE_NAME}] ..." >>${GATHERSTATSSCRIPT}
echo "BEGIN" >>${GATHERSTATSSCRIPT}
echo "DBMS_STATS.EXPORT_TABLE_STATS (" >>${GATHERSTATSSCRIPT}
echo "ownname => upper('${SCHEMA_NAME}')," >>${GATHERSTATSSCRIPT}
echo "tabname => upper('${TABLE_NAME}')," >>${GATHERSTATSSCRIPT}
echo "statown => upper('${STATS_OWNER}')," >>${GATHERSTATSSCRIPT}
echo "stattab => upper('${STATS_TABLE}'));" >>${GATHERSTATSSCRIPT}
echo "END;" >>${GATHERSTATSSCRIPT}
echo "/" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT GATHERING STATISTICS ${HISTOMSG} FOR TABLE [${SCHEMA_NAME}.${TABLE_NAME}] ${CASCMSG} ..." >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT Feel free to exist this session any time as this script is running in the background :-)" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT To check the progress:" >>${GATHERSTATSSCRIPT}
echo "PROMPT tail -f ${GATHERSTATSSPOOL}" >>${GATHERSTATSSCRIPT}
echo "BEGIN" >>${GATHERSTATSSCRIPT}
echo "DBMS_STATS.GATHER_TABLE_STATS (" >>${GATHERSTATSSCRIPT}
echo "ownname => upper('${SCHEMA_NAME}')," >>${GATHERSTATSSCRIPT}
echo "tabname => upper('${TABLE_NAME}')," >>${GATHERSTATSSCRIPT}
echo "cascade => ${CASCD}," >>${GATHERSTATSSCRIPT}
echo "METHOD_OPT => '${HISTO}'," >>${GATHERSTATSSCRIPT}
echo "DEGREE => DBMS_STATS.AUTO_DEGREE," >>${GATHERSTATSSCRIPT}
echo "estimate_percent => DBMS_STATS.AUTO_SAMPLE_SIZE);" >>${GATHERSTATSSCRIPT}
echo "END;" >>${GATHERSTATSSCRIPT}
echo "/" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT => IN CASE THE NEW STATISTICS ARE PERFORMING BAD, RESTORE BACK THE ORIGINAL STATISTICS USING THE FOLLOWING SQL COMMAND:" >>${GATHERSTATSSCRIPT}
echo "PROMPT >>>>" >>${GATHERSTATSSCRIPT}
echo "PROMPT EXEC DBMS_STATS.IMPORT_TABLE_STATS (ownname => upper('${SCHEMA_NAME}'), tabname => upper('${TABLE_NAME}'), statown => upper('${STATS_OWNER}'), stattab => upper('${STATS_TABLE}'));;" >>${GATHERSTATSSCRIPT}
echo "PROMPT >>>>" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "${LOCK_STATS_MSG}" >>${GATHERSTATSSCRIPT}
echo "${LOCK_STATS_SQL}" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "PROMPT SCRIPT COMPLETED!" >>${GATHERSTATSSCRIPT}
echo "PROMPT" >>${GATHERSTATSSCRIPT}
echo "spool off" >>${GATHERSTATSSCRIPT}
echo "exit" >>${GATHERSTATSSCRIPT}
echo "#!/bin/bash" > ${GATHERSTATSSCRIPTRUNNER}
echo "nohup ${ORACLE_HOME}/bin/sqlplus \"/ as sysdba\" @${GATHERSTATSSCRIPT} | tee ${GATHERSTATSSPOOL} 2>&1 &" >>${GATHERSTATSSCRIPTRUNNER}
chmod 740 ${GATHERSTATSSCRIPTRUNNER}
echo ""
source ${GATHERSTATSSCRIPTRUNNER}
# #############
# END OF SCRIPT
# #############
# REPORT BUGS to: mahmmoudadel@hotmail.com
# DOWNLOAD THE LATEST VERSION OF DATABASE ADMINISTRATION BUNDLE FROM: http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html
# DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".
view raw gather_stats.sh hosted with ❤ by GitHub

Saturday, April 21, 2018

Script To Automatic Kill of Long Running Sessions Connecting From Specific Program | Oracle On Linux

In the previous post, I've shared a script reports long running sessions/queries. In this post I'll explain another simple script that kills long running sessions connecting from specific programs e.g. SQL Developer

The script can be tuned to target any specific programs, but I'm mainly defaulting the script to target SQL Developer, as it's widely used by the developers, moreover, it doesn't maintain disconnecting the running session properly when the user close the session's window, keeping the "supposed to be closed" session query running in the database!

Download link:
https://www.dropbox.com/s/ke01v10bu0bat84/kill_long_running_queries.sh?dl=0

How to use:

The script is designed to be scheduled in the crontab to run every let's say 5 minutes to kill specific sessions based on the killing criteria which you can identify by editing the following parameters:

export ORACLE_SID=orcl
You have to provide the name of the instance you want to run the script against e.g. orcl

MAIL_LIST="youremail@yourcompany.com"
Provide your email to get a detailed report after each successful session kill.

export MODULE_NAME="'SQL Developer'"
You have to provide the module name that appears in v$session which you are targeting its sessions, in order to provide more than one module you have to enclose each module name between single quote putting comma "," in between.
e.g. To include SQL Developer and Toad sessions in the killing criteria you have to provide the values to MODULE_NAME parameter like the following:
export MODULE_NAME="'SQL Developer','Toad'"

To get an idea of module names that are connecting to your database run this statement:
select distinct(module_name) from gv$active_session_history;

export DURATION="2.5"
Duration parameter is the session's elapsed active time in hours which if reached the session will be a candidate for kill. It's set to 2 hours and half as a default value. of course you have to change it to whatever fits your requirements.

export REPORT_ONLY="N"
This parameter if set to "N" to enable the real kill of the candidate session, if it's set to "Y" it will not do the actual kill but will ONLY report the candidate session to your email. This parameter is helpful during testing phase, you've to test this script first by setting this parameter to "Y", it will keep reporting to you the candidate sessions (to be killed) without killing them, once you're confident with the script accuracy, set it to"N" to start killing the candidate sessions.

Ultimately, although I'm not keen to share such scripts that may cause problems if they are not accurately configured , I'm sharing this script in the hope that you will use it wisely, and in order to do so, you have to test and test and test such scripts before you deploy them on production.
By your use of this script you acknowledge that you use it at your own risk.

Github version:



# ################################################################################################################
# Kill QUERIES/SESSIONS running for more than N minutes based on specific criteria.
# ################################################################################################################
VER="[2.5]"
# # # #
# Author: Mahmmoud ADEL # # # # ###
# Created: 11-01-18 # # # # #
#
# Modified: 17-08-22 Included the following filters:
# INCLUDE_USERS & EXCLUDE_USERS They are MUTUALLY EXCLUSIVE, only one to be set at a time.
# 25-08-22 Added TEST_MODE option, to help the user test the script without killing any sessions.
# 25-08-22 Set DURATION variable to minutes instead of hours.
# 30-08-22 Added KILL_QUERY_ONLY option if set to Y will kill only the query instead of the session.
# 30-08-22 Added SESSION_STATUS to allow the user to kill INACTIVE sessions.
# be cautious when killing INACTIVE session if the application is using connection pooling.
# 12-09-22 Added the option of excluding SQLIDs through EXCLUDESQLID parameter.
# ################################################################################################################
# ########## ####################################
# VARIABLES: [Here you TWEAK the script Settings]
# ########## ####################################
MAIL_KILL_LIST="youremail@yourcompany.com" # Email Recipients: To be notified when a session kill occurs.
MAIL_REPORT_LIST=${MAIL_KILL_LIST} # Email Recipients: To be notified when TESTING mode is enabled. TEST_MODE="Y"
ORACLE_SID= # Define the INSTANCE NAME the script will run against e.g. ORACLE_SID="ORCL"
##### INCLUDE_USERS and EXCLUDE_USERS are MUTUALY EXCLUSIVE; define only one of them at a time, if both defined; the script will consider only INCLUDE_USERS.
INCLUDE_USERS="'TEST1'" # DB USERNAMES TO BE KILLED. e.g. INCLUDE_USERS="'TEST'" OR INCLUDE_USERS="'TEST','ELDAMARANI'"
EXCLUDE_USERS="'SYS'" # DB USERNAMES TO NOT BE KILLED. e.g. EXCLUDE_USERS="'SYS'" OR EXCLUDE_USERS="'SYS','APPUSER'"
MODULE="'SQL Developer','Toad'" # Candidate Modules to be KILLED. e.g. MODULE="'Toad'", or: MODULE="'SQL Developer','Toad'"
DURATION="60" # Duration: [In Minutes]. e.g. to kill defined sessions running for more than 30minutes: DURATION="30"
KILL_QUERY_ONLY=Y # KILL ONLY THE QUERY NOT THE SESSION [Works on 12c and above]
TEST_MODE="Y" # This is TEST MODE. e.g. to report the long running sessions WITHOUT kill: TEST_MODE="Y" , to report & KILL TEST_MODE="N"
SESSION_STATUS="ACTIVE" # Define the user's session status either ACTIVE or INACTIVE [Default is ACTIVE]
EXCLUDESQLID="'ckajr8v22554b'" # EXCLUDED SQL_IDs. e.g. EXCLUDESQLID="'ckajr8v22554b','59p1yadp2g6mb'"
SCRIPT_NAME="kill_long_queries.sh" # SCRIPT NAME
SRV_NAME=`uname -n` # MACHINE NAME
LOGDIR=/tmp # LOG DIRECTORY
LOGFILE=${LOGDIR}/kill_long_queries.log # LOGFILE PATH
TERMINATOR_SCRIPT=${LOGDIR}/KILL_LONG_QUERIES.sql # SQLPlus KILL SCRIPT PATH
#export smtp="mailrelay.mycompany.com:25" # Set it to the SMTP NAME:PORT your company is using. [If exist]
# ##################
# Checking Variables: [Do NOT modify the lines below]
# ##################
# Check if the user provided a valid Email address:
case ${EMAIL} in "youremail@yourcompany.com")
echo
echo "******************************************************************"
echo "Buddy! You forgot to provide your EMail to MAIL_KILL_LIST variable."
echo "Please replace youremail@yourcompany.com with your E-mail address,"
echo "to be able to receive a Notification when a session kill happens."
echo "******************************************************************"
echo
esac
# Check INCLUDE_USERS & EXCLUDE_USERS parameters and make sure to not collide:
# If BOTH INCLUDE_USERS & EXCLUDE_USERS variables are DEFINED, then notify the user:
if ! [ -z ${INCLUDE_USERS} ] && ! [ -z ${EXCLUDE_USERS} ]
then
echo ""
echo "WARNING: You have defined both variables INCLUDE_USERS & EXCLUDE_USERS."
echo "Note: The script will ONLY consider INCLUDE_USERS=${INCLUDE_USERS} and will ignore EXCLUDE_USERS list."
echo
export EXCLUDE_USERS=""
export DBUSERNAMES=${INCLUDE_USERS}
export PREDICATE="IN"
fi
# If INCLUDE_USERS is already set then UN-DEFINE EXCLUDE_USERS and set SQLPlus script variables accordingly:
if ! [ -z ${INCLUDE_USERS} ]
then
echo "Note: INCLUDE_USERS variable is set."
echo
export EXCLUDE_USERS=""
export DBUSERNAMES=${INCLUDE_USERS}
export PREDICATE="IN"
fi
# If EXCLUDE_USERS is set then set SQLPlus script variables accordingly:
if ! [ -z ${EXCLUDE_USERS} ]
then
echo
echo "Note: EXCLUDE_USERS variable is set."
echo
export DBUSERNAMES=${EXCLUDE_USERS}
export PREDICATE="NOT IN"
fi
# If BOTH INCLUDE_USERS & EXCLUDE_USERS variables are not set then hash the entire USERNAME filter line in SQLPlus script: [kill all sessions connecting through defined modules]
if [ -z ${INCLUDE_USERS} ] && [ -z ${EXCLUDE_USERS} ]
then
echo
echo "Note: USERNAME criteria is not been set."
echo
export HASHUSERNAME="--"
fi
# IF EXCLUDESQLID variable is empty:
if [ -z ${EXCLUDESQLID} ]
then
export HASHSQLID="--"
export SQLIDMSG=""
else
export HASHSQLID=""
export SQLIDMSG="> SQLID NOT IN: ${EXCLUDESQLID}"
fi
# Check TEST_MODE Flag:
case ${TEST_MODE} in
Y|y|yes|Yes|YES) export HASH_SCRIPT="--";export TEST_MODE_MESSAGE="PROMPT TEST_MODE value is set to Y, No Kill will happen"
echo
echo -e "\033[33;5mTEST MODE is Activated, NO KILL will happen.\033[0m"
echo
export MAIL_LIST="${MAIL_REPORT_LIST}"
export REPORT_ONLY_MSG="CANDIDATE TO BE"
;;
*) export HASH_SCRIPT="";export TEST_MODE_MESSAGE=""
export MAIL_LIST="${MAIL_KILL_LIST}"
;;
esac
# #######################################
# Excluded INSTANCES:
# #######################################
# Here you can mention the instances dbalarm will IGNORE and will NOT run against:
# Use pipe "|" as a separator between each instance name.
# e.g. Excluding: -MGMTDB, ASM instances:
EXL_DB="\-MGMTDB|ASM|APX" #Excluded INSTANCES [Will not get reported offline].
# #######################################
# Exporting Variables for child sessions:
# #######################################
export MAIL_KILL_LIST
export MAIL_REPORT_LIST
export MODULE
export DBUSERNAMES
export EXCLUDE_USERS
export DURATION
export TEST_MODE
export SESSION_STATUS
export ORACLE_SID
export SCRIPT_NAME
export SRV_NAME
export LNXVER
export LOGFILE
export TERMINATOR_SCRIPT
export EXL_DB
export HASH_KILL_QUERY="--"
export HASH_KILL_SESSION=""
export OBJECT="SESSIONS"
# #########################
# Setting ORACLE_SID:
# #########################
# Check if the user has set the ORACLE_SID:
if [ -z ${ORACLE_SID} ]
then
echo "WARNING: You didn't set the ORACLE_SID inside the script!"
echo "******* In order to accurately run this script you have to set the ORACLE_SID,"
echo " Otherwise the script will run against the only running DB if applicable."
echo ""
fi
# If no Databases are up show a warning message:
INS_COUNT=$( ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|wc -l )
if [ ${INS_COUNT} -eq 0 ]
then
ps -ef|grep -v grep|grep pmon
echo "WARNING: No Databases Are Running on the Server!"
echo
ps -ef|grep -v grep|grep pmon
echo "Script Terminated!"
exit
fi
# Set ORACLE_SID to the only running Instance: [If the user didn't define a specific ORACLE_SID]
if [ -z ${ORACLE_SID} ] && [ ${INS_COUNT} -eq 1 ]
then
export ORACLE_SID=`ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|awk '{print $NF}'|sed -e 's/ora_pmon_//g'|grep -v sed|grep -v "s///g"`
fi
if [ ${INS_COUNT} -gt 1 ] && [ -z ${ORACLE_SID} ]
then
echo "Error: Too many instances running!"
echo "***** Please set the ORACLE_SID inside the script to accurately define the DB the script will run against."
echo
echo "Script Terminatd!"
echo ""
exit
fi
# #########################
# Getting ORACLE_HOME
# #########################
ORA_USER=`ps -ef|grep ${ORACLE_SID}|grep pmon|grep -v grep|egrep -v ${EXL_DB}|awk '{print $1}'|tail -1`
USR_ORA_HOME=`grep ${ORA_USER} /etc/passwd| cut -f6 -d ':'|tail -1`
# SETTING ORATAB:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
export ORATAB
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
export ORATAB
fi
# ATTEMPT1: Get ORACLE_HOME using pwdx command:
PMON_PID=`pgrep -lf _pmon_${ORACLE_SID}|awk '{print $1}'`
export PMON_PID
ORACLE_HOME=`pwdx ${PMON_PID} 2>/dev/null|awk '{print $NF}'|sed -e 's/\/dbs//g'`
export ORACLE_HOME
# ATTEMPT2: If ORACLE_HOME not found get it from oratab file:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
## If OS is Linux:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
ORACLE_HOME=`grep -v '^\#' ${ORATAB} | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
ORACLE_HOME=`grep -v '^\#' ${ORATAB} | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
fi
fi
# ATTEMPT3: If ORACLE_HOME is in /etc/oratab, use dbhome command:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`dbhome "${ORACLE_SID}"`
export ORACLE_HOME
fi
# ATTEMPT4: If ORACLE_HOME is still not found, search for the environment variable: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`env|grep -i ORACLE_HOME|sed -e 's/ORACLE_HOME=//g'`
export ORACLE_HOME
fi
# ATTEMPT5: If ORACLE_HOME is not found in the environment search user's profile: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`grep -h 'ORACLE_HOME=\/' ${USR_ORA_HOME}/.bash_profile ${USR_ORA_HOME}/.*profile | perl -lpe'$_ = reverse' |cut -f1 -d'=' | perl -lpe'$_ = reverse'|tail -1`
export ORACLE_HOME
fi
# ATTEMPT6: If ORACLE_HOME is still not found, search for orapipe: [Least accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
if [ -x /usr/bin/locate ]
then
ORACLE_HOME=`locate -i orapipe|head -1|sed -e 's/\/bin\/orapipe//g'`
export ORACLE_HOME
fi
fi
# ###################
# Getting DB Version:
# ###################
#echo "Checking DB Version"
DB_VER_RAW=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set pages 0 feedback off lines 1000;
prompt
select version from v\$instance;
exit;
EOF
)
export DB_VER=`echo ${DB_VER_RAW}|perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'|cut -f1 -d '.'`
# Is it KILL SESSION or DISCONNECT SESSION in the alter system command: [If DB version is <11g use KILL if >=11g then use DISCONNECT keyword]
if [ ${DB_VER} -gt 10 ]
then
export KILLARG="DISCONNECT"
else
export KILLARG="KILL"
fi
# If the KILL_QUERY_ONLY is set to Y and Database version is 12c and above disable the KILL SESSION command and enable the CANCEL QUERY command:
if [ ${DB_VER} -gt 11 ] && [ ${KILL_QUERY_ONLY} = "Y" ]
then
echo "Note: KILL QUERY MODE IS ACTIVATED. [DB_VERSION=${DB_VER} and KILL_QUERY_ONLY=${KILL_QUERY_ONLY}]"
echo
export HASH_KILL_SESSION="--"
export HASH_KILL_QUERY=""
export OBJECT="QUERY"
fi
# ###########
# SCRIPT BODY:
# ###########
# Flush the logfile:
cat /dev/null > ${LOGFILE}
# CHECKING RUNNING SESSIONS:
SESSIONS_COUNT_RAW=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set pages 0 feedback off;
prompt
select count(*) from GV\$SESSION where
MODULE in (${MODULE})
${HASHUSERNAME} and username ${PREDICATE} (${DBUSERNAMES})
${HASHSQLID} and SQL_ID not in (${EXCLUDESQLID})
and last_call_et > 60*${DURATION}
and status = '${SESSION_STATUS}'
and wait_class <>'Idle'
;
exit;
EOF
)
SESSIONS_COUNT=`echo ${SESSIONS_COUNT_RAW}|perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'|cut -f1 -d '.'`
#echo SESSIONS_COUNT_RAW is: $SESSIONS_COUNT_RAW
#echo SESSIONS_COUNT is: $SESSIONS_COUNT_RAW
# KILLING LONG RUNNING SESSIONS IF EXIST:
# ######################################
if [ ${SESSIONS_COUNT} -gt 0 ]
then
echo "Found ${SESSIONS_COUNT} Candidate ${OBJECT} to be killed!"
KILL_SESSION_RAW=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
spool ${LOGFILE} APPEND
set pages 0 feedback off
prompt
PROMPT *****
select 'TIME: '||to_char(systimestamp, 'dd-Mon-yy HH24:MI:SS') from dual;
PROMPT *****
set linesize 170 pages 1000;
prompt
prompt Session Details: [${OBJECT} To be killed]
prompt ***************
col inst for 99
col module for a27
col event for a28
col MACHINE for a27
col "ST|ACT_SINC|LOG_TIME" for a51
col "USER|OS|SID,SER|MACHIN|MODUL" for a75
select substr(USERNAME||'|'||OSUSER||'|'||sid||','||serial#||'|'||substr(MACHINE,1,25)||' | '||substr(MODULE,1,25),1,75)"USER|OS|SID,SER|MACHIN|MODUL"
,substr(status||'|'||LAST_CALL_ET||'|'||LOGON_TIME,1,50) "ST|ACT_SINC|LOG_TIME"
,SQL_ID CURR_SQL_ID
from gv\$session
where
MODULE in (${MODULE})
${HASHUSERNAME} and username ${PREDICATE} (${DBUSERNAMES})
and last_call_et > 60*${DURATION}
and status = '${SESSION_STATUS}'
${HASHSQLID} and SQL_ID not in (${EXCLUDESQLID})
and wait_class <>'Idle'
;
spool off
-- Kill SQL Script creation:
set pages 0 feedback off echo off
spool ${TERMINATOR_SCRIPT}
${HASH_KILL_SESSION} select /*+RULE*/ 'ALTER SYSTEM ${KILLARG} SESSION '''||sid||','||serial#||',@'||inst_id||''' IMMEDIATE;'
${HASH_KILL_QUERY} select /*+RULE*/ 'ALTER SYSTEM CANCEL SQL '''||sid||','||serial#||',@'||inst_id||','||SQL_ID||''';'
from GV\$SESSION
where
MODULE in (${MODULE})
${HASHUSERNAME} and username ${PREDICATE} (${DBUSERNAMES})
and last_call_et > 60*${DURATION}
and status = '${SESSION_STATUS}'
${HASHSQLID} and SQL_ID not in (${EXCLUDESQLID})
and wait_class <>'Idle'
;
spool off
-- Run the Terminator Script to kill the sessions:
set pages 1000 feedback on echo on
spool ${LOGFILE} APPEND
PROMPT
PROMPT Running The Terminator Script:
PROMPT *****************************
${TEST_MODE_MESSAGE}
${HASH_SCRIPT}START ${TERMINATOR_SCRIPT}
spool off
exit;
EOF
)
sleep 10
CURRENT_LONG_SESS_RAW=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set linesize 170 pages 1000;
spool ${LOGFILE} APPEND
prompt
prompt CHECK For other long queries fulfill the killing criteria: [Still Running]
prompt *********************************************************
col inst for 99
col module for a27
col event for a28
col MACHINE for a27
col "ST|ACT_SINC|LOG_TIME" for a51
col "USER|SID,SER|MACHIN|MODUL" for a72
select substr(USERNAME||'|'||sid||','||serial#||'|'||substr(MACHINE,1,25)||' | '||substr(MODULE,1,25),1,72)"USER|SID,SER|MACHIN|MODUL"
,substr(status||'|'||LAST_CALL_ET||'|'||LOGON_TIME,1,50) "ST|ACT_SINC|LOG_TIME"
,SQL_ID CURR_SQL_ID
from gv\$session
where
MODULE in (${MODULE})
${HASHUSERNAME} and username ${PREDICATE} (${DBUSERNAMES})
and last_call_et > 60*${DURATION}
and status = '${SESSION_STATUS}'
${HASHSQLID} and SQL_ID not in (${EXCLUDESQLID})
and wait_class <>'Idle'
;
spool off
exit;
EOF
)
# EMAIL Notification with the killed session:
cat ${LOGFILE}
echo
echo "Sending Email Notification to: ${MAIL_LIST}"
echo
/bin/mail -s "Info: Long Running ${OBJECT} ${REPORT_ONLY_MSG} KILLED on [${ORACLE_SID}]" ${MAIL_LIST} < ${LOGFILE}
else
echo ""
echo "On Instance: ${ORACLE_SID}"
echo -e "\033[33;5m No Candidate ${OBJECT} to be KILLED are found as per the given Criteria:\033[0m"
echo " > MODULE is: ${MODULE}"
echo " > USERNAME ${PREDICATE} (${DBUSERNAMES})"
echo " ${SQLIDMSG}"
fi
# #############
# END OF SCRIPT
# #############
# REPORT BUGS to: mahmmoudadel@hotmail.com
# DOWNLOAD THE LATEST VERSION OF DATABASE ADMINISTRATION BUNDLE FROM:
# http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html
# DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".

Shell Script To Monitor Long Running Queries In Oracle

Monitoring long-running queries is part of the early warning mechanism for the performance
issues on the database before it gets exaggerated


If you're running Oracle on Linux, then this script is for you.
Download link:
https://www.dropbox.com/s/yh6k3r7q3p876h0/report_long_runing_queries.sh?dl=0

The script is easy to use, it will run against all instances on the server reporting any active sessions/query for more than 60 minutes (the default), you can adjust this duration by changing the following parameter under THRESHOLDS section to whatever number of minutes you want:

EXEC_TIME_IN_MINUTES=60        # Report Sessions running longer than N minutes (default is 60 min).

Also, you can control the report generation if the number of long active sessions have reached a specific count. By adjusting the following parameter:

LONG_RUN_SESS_COUNT=0        # The count of long-running sessions, report will not appear unless reached (0 report all long running sessions).

Lastly, you must provide your email address in order to receive a detailed Email report of long-running queries/active sessions, by editing below template in navy color:

MAIL_LIST="youremail@yourcompany.com"

You can schedule the script in the crontab to run every 5/10 minutes.

In the upcoming post, I'll share another script to automatically kill long active sessions connected by specific programs like SQL Developer or Toad. Let's make the life of our beloved developers easy as they always make our life easy :-)

Monitor Long Running Queries | Long Active Sessions Script (GitHub version):

# ##################################################################################
# Checking long running queries run by specific user
# [Ver 1.2]
#
# # # #
# Author: Mahmmoud ADEL # # # # ###
# Created: 09-03-17 # # # # #
# Modified: 21-01-19 Enhanced the fetch for ORACLE_HOME.
# 19-02-19 Enabled the HTML report version.
# 06-07-20 New calculation method for 12c+ updated.
#
#
#
#
#
#
#
#
#
#
# ##################################################################################
EMAIL="youremail@yourcompany.com"
SCRIPT_NAME="report_long_runing_queries"
SRV_NAME=`uname -n`
case ${EMAIL} in "youremail@yourcompany.com")
echo
echo "****************************************************************************************************"
echo "Buddy! You will not receive an E-mail with the result, because you didn't set EMAIL variable yet"
echo "Just replace youremail@yourcompany.com with your right email."
echo "****************************************************************************************************"
echo
esac
export EMAIL
# #########################
# THRESHOLDS:
# #########################
# Modify the THRESHOLDS to the value you prefer:
EXEC_TIME_IN_MINUTES=60 # Report Sessions running longer than N minutes [Default is 60 minutes].
LONG_RUN_SESS_COUNT=0 # CONTROL the number of long running sessions if reached, the report will tirgger. [Default 0 which means report all long running sessions].
HTMLENABLE=Y # Enable HTML Email Format
export EXEC_TIME_IN_MINUTES
export LONG_RUN_SESS_COUNT
# #######################################
# Excluded INSTANCES:
# #######################################
# Here you can mention the instances the script will IGNORE and will NOT run against:
# Use pipe "|" as a separator between each instance name.
# e.g. Excluding: -MGMTDB, ASM instances:
EXL_DB="\-MGMTDB|ASM|APX" #Excluded INSTANCES [Will not get reported offline].
# #########################
# SQLPLUS Output Format:
# #########################
SQLLINESIZE=160
SQLPAGES=1000
SQLLONG=999999999
export SQLLINESIZE
export SQLPAGES
export SQLLONG
# ##########################
# Neutralize login.sql file: [Bug Fix]
# ##########################
# Existance of login.sql file under Oracle user Linux home directory eliminates many functions during the execution of this script from crontab:
if [ -f ${USR_ORA_HOME}/login.sql ]
then
mv ${USR_ORA_HOME}/login.sql ${USR_ORA_HOME}/login.sql_NeutralizedBy${SCRIPT_NAME}
fi
# #########################
# Setting ORACLE_SID:
# #########################
for ORACLE_SID in $( ps -ef|grep pmon|grep -v grep|egrep -v ${EXL_DB}|awk '{print $NF}'|sed -e 's/ora_pmon_//g'|grep -v sed|grep -v "s///g" )
do
export ORACLE_SID
# #########################
# Getting ORACLE_HOME
# #########################
ORA_USER=`ps -ef|grep ${ORACLE_SID}|grep pmon|grep -v grep|egrep -v ${EXL_DB}|grep -v "\-MGMTDB"|awk '{print $1}'|tail -1`
USR_ORA_HOME=`grep -i "^${ORA_USER}:" /etc/passwd| cut -f6 -d ':'|tail -1`
# SETTING ORATAB:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
export ORATAB
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
export ORATAB
fi
# ATTEMPT1: Get ORACLE_HOME using pwdx command:
PMON_PID=`pgrep -lf _pmon_${ORACLE_SID}|awk '{print $1}'`
export PMON_PID
ORACLE_HOME=`pwdx ${PMON_PID}|awk '{print $NF}'|sed -e 's/\/dbs//g'`
export ORACLE_HOME
# ATTEMPT2: If ORACLE_HOME not found get it from oratab file:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
## If OS is Linux:
if [ -f /etc/oratab ]
then
ORATAB=/etc/oratab
ORACLE_HOME=`grep -v '^\#' ${ORATAB} | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
## If OS is Solaris:
elif [ -f /var/opt/oracle/oratab ]
then
ORATAB=/var/opt/oracle/oratab
ORACLE_HOME=`grep -v '^\#' ${ORATAB} | grep -v '^$'| grep -i "^${ORACLE_SID}:" | perl -lpe'$_ = reverse' | cut -f3 | perl -lpe'$_ = reverse' |cut -f2 -d':'`
export ORACLE_HOME
fi
fi
# ATTEMPT3: If ORACLE_HOME is in /etc/oratab, use dbhome command:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`dbhome "${ORACLE_SID}"`
export ORACLE_HOME
fi
# ATTEMPT4: If ORACLE_HOME is still not found, search for the environment variable: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`env|grep -i ORACLE_HOME|sed -e 's/ORACLE_HOME=//g'`
export ORACLE_HOME
fi
# ATTEMPT5: If ORACLE_HOME is not found in the environment search user's profile: [Less accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
ORACLE_HOME=`grep -h 'ORACLE_HOME=\/' ${USR_ORA_HOME}/.bash_profile ${USR_ORA_HOME}/.*profile | perl -lpe'$_ = reverse' |cut -f1 -d'=' | perl -lpe'$_ = reverse'|tail -1`
export ORACLE_HOME
fi
# ATTEMPT6: If ORACLE_HOME is still not found, search for orapipe: [Least accurate]
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
if [ -f /usr/bin/locate ]
then
ORACLE_HOME=`locate -i orapipe|head -1|sed -e 's/\/bin\/orapipe//g'`
export ORACLE_HOME
fi
fi
# TERMINATE: If all above attempts failed to get ORACLE_HOME location, EXIT the script:
if [ ! -f ${ORACLE_HOME}/bin/sqlplus ]
then
echo "Please export ORACLE_HOME variable in your .bash_profile file under oracle user home directory in order to get this script to run properly"
echo "e.g."
echo "export ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1"
exit
fi
# ###################
# Getting DB Version:
# ###################
echo "Checking DB Version"
VAL311=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set pages 0 feedback off lines 1000;
prompt
select version from v\$instance;
exit;
EOF
)
DB_VER=`echo ${VAL311}|perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'|cut -f1 -d '.'`
# #########################
# Variables:
# #########################
export PATH=$PATH:${ORACLE_HOME}/bin
export LOG_DIR=${USR_ORA_HOME}/BUNDLE_Logs
mkdir -p ${LOG_DIR}
chown -R ${ORA_USER} ${LOG_DIR}
chmod -R go-rwx ${LOG_DIR}
if [ ! -d ${LOG_DIR} ]
then
mkdir -p /tmp/BUNDLE_Logs
export LOG_DIR=/tmp/BUNDLE_Logs
chown -R ${ORA_USER} ${LOG_DIR}
chmod -R go-rwx ${LOG_DIR}
fi
export LOGFILE=${LOG_DIR}/reported_long_running_sessions.log
# #########################
# HTML Preparation:
# #########################
case ${HTMLENABLE} in
y|Y|yes|YES|Yes|ON|On|on)
if [ -x /usr/sbin/sendmail ]
then
export SENDMAIL="/usr/sbin/sendmail -t"
export MAILEXEC="echo #"
export HASHHTML=""
export HASHHTMLOS=""
export ENDHASHHTMLOS=""
export HASHNONHTML="--"
SENDMAILARGS=$(
echo "To: ${EMAIL};"
echo "Subject: ${MSGSUBJECT} ;"
echo "Content-Type: text/html;"
echo "MIME-Version: 1.0;"
cat ${LOGFILE}
)
export SENDMAILARGS
else
export SENDMAIL="echo #"
export MAILEXEC="mail -s"
export HASHHTML="--"
export HASHHTMLOS="echo #"
export ENDHASHHTMLOS=""
export HASHNONHTML=""
fi
;;
*)
export SENDMAIL="echo #"
export HASHHTML="--"
export HASHHTMLOS="echo #"
export ENDHASHHTMLOS=""
export HASHNONHTML=""
export MAILEXEC="mail -s"
;;
esac
# ##########################
# Neutralize login.sql file: [Bug Fix]
# ##########################
# Existance of login.sql file under Oracle user Linux home directory eliminates many functions during the execution of this script from crontab:
if [ -f ${USR_ORA_HOME}/login.sql ]
then
mv ${USR_ORA_HOME}/login.sql ${USR_ORA_HOME}/login.sql_NeutralizedBy${SCRIPT_NAME}
fi
# ####################################
# SCRIPT ENGINE:
# ####################################
# Check if the database version is 11g or higher then get the info from v$session_longops view:
if [ ${DB_VER} -gt 10 ]
then
LONG_RUN_COUNT_RAW2=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set pages 0 feedback off;
prompt
select count(*) from v\$session_longops
where ELAPSED_SECONDS > 60*${EXEC_TIME_IN_MINUTES}
and TOTALWORK <> '0'
and round(SOFAR/TOTALWORK*100,2) <> '100';
exit;
EOF
)
LONG_RUN_COUNT2=`echo ${LONG_RUN_COUNT_RAW2}|perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'|cut -f1 -d '.'`
if [ ${LONG_RUN_COUNT2} -gt ${LONG_RUN_SESS_COUNT} ]
then
# Long running query output:
LONG_QUERY_DETAIL2=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set linesize ${SQLLINESIZE} pages ${SQLPAGES}
-- Enable HTML color format:
${HASHHTML} SET WRAP OFF ECHO OFF FEEDBACK OFF MARKUP HTML ON SPOOL ON HEAD '<title></title> <style type="text/css"> table { font-size: 90%; } th { background: #AF601A; } </style>' TABLE "border='2' bordercolor='#E67E22'" ENTMAP OFF
col OPERATION for a21
col "%DONE" for 999.999
col "STARTED|MIN_ELAPSED|MIN_REMAIN" for a26
col MESSAGE for a77
col "USERNAME| SID,SERIAL#" for a28
spool ${LOGFILE}
select USERNAME||'| '||SID||','||SERIAL# "USERNAME| SID,SERIAL#",SQL_ID
--,OPNAME OPERATION
--,substr(SOFAR/TOTALWORK*100,1,5) "%DONE"
,round(SOFAR/TOTALWORK*100,2) "%DONE"
,to_char(START_TIME,'DD-MON HH24:MI')||'| '||trunc(ELAPSED_SECONDS/60)||'|'||trunc(TIME_REMAINING/60) "STARTED|MIN_ELAPSED|MIN_REMAIN" ,MESSAGE
from v\$session_longops
where ELAPSED_SECONDS > 60
and TOTALWORK <> '0'
and round(SOFAR/TOTALWORK*100,2) <> '100'
order by "STARTED|MIN_ELAPSED|MIN_REMAIN" desc, "USERNAME| SID,SERIAL#";
spool off
exit;
EOF
)
cat ${LOGFILE}
export MSGSUBJECT="Info: Long Running Queries on DB [${ORACLE_SID}] on Server [${SRV_NAME}]"
${MAILEXEC} "${MSGSUBJECT}" ${EMAIL} < ${LOGFILE}
(
echo "To: ${EMAIL};"
echo "MIME-Version: 1.0"
echo "Content-Type: text/html;"
echo "Subject: ${MSGSUBJECT}"
cat ${LOGFILE}
) | ${SENDMAIL}
fi
else
# If database version is 10g backwards:
# Check the Long Running Session Count:
LONG_RUN_COUNT_RAW2=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set pages 0 feedback off;
prompt
select count(*) from v\$session
where
-- To capture active session for more than defined EXEC_TIME_IN_MINUTES variable in minutes:
last_call_et > 60*${EXEC_TIME_IN_MINUTES}
and username is not null
and module is not null
and module not like 'backup%'
and module not like 'OGG-%'
and status = 'ACTIVE';
exit;
EOF
)
LONG_RUN_COUNT2=`echo ${LONG_RUN_COUNT_RAW2}|perl -lpe'$_ = reverse' |awk '{print $1}'|perl -lpe'$_ = reverse'|cut -f1 -d '.'`
if [ ${LONG_RUN_COUNT2} -gt ${LONG_RUN_SESS_COUNT} ]
then
# Long running query output:
LONG_QUERY_DETAIL2=$(${ORACLE_HOME}/bin/sqlplus -S "/ as sysdba" <<EOF
set linesize ${SQLLINESIZE} pages ${SQLPAGES}
-- Enable HTML color format:
${HASHHTML} SET WRAP OFF ECHO OFF FEEDBACK OFF MARKUP HTML ON SPOOL ON HEAD '<title></title> <style type="text/css"> table { font-size: 90%; } th { background: #AF601A; } </style>' TABLE "border='2' bordercolor='#E67E22'" ENTMAP OFF
set long ${SQLLONG}
col module for a30
col DURATION_HOURS for 99999.9
col STARTED_AT for a13
col "USERNAME| SID,SERIAL#" for a30
col "SQL_ID | SQL_TEXT" for a${SQLLINESIZE}
spool ${LOGFILE}
select username||'| '||sid ||','|| serial# "USERNAME| SID,SERIAL#",substr(MODULE,1,30) "MODULE", to_char(sysdate-last_call_et/24/60/60,'DD-MON HH24:MI') STARTED_AT,
last_call_et/60/60 "DURATION_HOURS"
,SQL_ID ||' | '|| (select SQL_FULLTEXT from v\$sql where address=sql_address and CHILD_NUMBER=SQL_CHILD_NUMBER) "SQL_ID | SQL_TEXT"
--,SQL_ID ||' | '|| (select SQL_FULLTEXT from v\$sql where address=sql_address) "SQL_ID | SQL_TEXT"
--,SQL_ID
from v\$session
where
-- To capture active session for more than defined EXEC_TIME_IN_MINUTES variable in minutes:
last_call_et > 60*${EXEC_TIME_IN_MINUTES}
and username is not null
and module is not null
and module not like 'backup%'
and status = 'ACTIVE'
order by "DURATION_HOURS" desc;
spool off
exit;
EOF
)
cat ${LOGFILE}
export MSGSUBJECT="Info: Long Running Queries on DB [${ORACLE_SID}] on Server [${SRV_NAME}]"
${MAILEXEC} "${MSGSUBJECT}" ${EMAIL} < ${LOGFILE}
(
echo "To: ${EMAIL};"
echo "MIME-Version: 1.0"
echo "Content-Type: text/html;"
echo "Subject: ${MSGSUBJECT}"
cat ${LOGFILE}
) | ${SENDMAIL}
fi
fi
done
# #############################
# De-Neutralize login.sql file:
# #############################
# If login.sql was renamed during the execution of the script revert it back to its original name:
if [ -f ${USR_ORA_HOME}/login.sql_NeutralizedBy${SCRIPT_NAME} ]
then
mv ${USR_ORA_HOME}/login.sql_NeutralizedBy${SCRIPT_NAME} ${USR_ORA_HOME}/login.sql
fi
# #############
# END OF SCRIPT
# #############
# REPORT BUGS to: mahmmoudadel@hotmail.com
# DOWNLOAD THE LATEST VERSION OF DATABASE ADMINISTRATION BUNDLE FROM:
# http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html
# DISCLAIMER: THIS SCRIPT IS DISTRIBUTED IN THE HOPE THAT IT WILL BE USEFUL, BUT WITHOUT ANY WARRANTY. IT IS PROVIDED "AS IS".

Thursday, April 5, 2018

DBA Bundle V4.2 Is Now Available

Fixed reported bugs inside (dbdailychk.sh, dbalarm.sh, export_data.sh, report_long_runing_queries.sh) scripts.

New Features:
Added Goldengate log monitoring feature to dbalarm.sh script [Disabled by default].
Added new aliases:
 - raclog to open the alert logfile for Clusterware/Oracle Restart.
 - asmlert to open the alert logfile for ASM instance.

To download DBA Bundle V4.2:
https://www.dropbox.com/s/xn0tf2pfeq04koi/DBA_BUNDLE4.tar?dl=0

To visit the main article that explains in details all bundle features:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html

Wednesday, February 28, 2018

DBA Bundle V4.1 Released

DBA Bundle V4.1 is now released with the following new scripts:

backup_ctrl_spf_AWR.sh Backup Controlfile as (Trace/RMAN BKP), Backup SPFILE and Generate AWR for full day This script can be scheduled in the crontab to run once a day. Script options/variables must be modified to match your environment. New Starting from V4.1
kill_long_running_queries.sh Kill all queries running longer than 2.5 Hours(can be customized inside the script) by specific modules (to be specified inside the script)
This script can be scheduled in the crontab. Script options/variables MUST be modified to get the killing criteria match your requirements. New Starting from V4.1
check_standby_lag.sh If you have a standby DB then you can use this script on Primary DB site to report to you any LAG happens between the primary and the standby DB.
The variables section at the top of the script must be populated by you to match your environment or the script will not be able to run.
This script can be scheduled in the crontab to run every 5 minutes. New Starting from V4.1
This link will show you how to use this script:
http://dba-tips.blogspot.ae/2017/11/shell-script-to-check-lag-sync-status.html

Many scripts are packed with new features like:

RMAN_full.sh
dbalarm.sh
dbdailychk.sh
schedule_rman_full_bkp.sh
active_sessions.sh
sql_id_details.sh



SQLPLUS is now more friendly for DBAs when invoked using "sql" alias:
It will display the following info when login:
 - Fundamental info about the DB you've connected to.
 - Your current session details (OSPID, SID, SERIAL#).
 - The prompt will show the time, instance_name you're connecting to and the connected user.
This will help you track your session log accurately plus minimize human mistakes.
 - The format of common columns in DBA_* & V$ views will be well fitted in your display.

To download DBA Bundle V4.1: [Feb 2018]
https://www.dropbox.com/s/k96rl0f4g39ukih/DBA_BUNDLE5.tar?dl=0

To visit the main article that explains all bundle features in details, and download the latest updated DBA Bundle version:
http://dba-tips.blogspot.com/2014/02/oracle-database-administration-scripts.html