ALERTHAWK / Oriole Corporation

Download ALERTHAWK_*.tar.Z

ALERTHAWK

Oracle, SQL*Net, SQL*Plus and SQL*Loader are registered trademarks of Oracle Corporation. All other products or company names are used for identification purposes only, and may be trademarks of their respective owners.

Purpose

When checking how an Oracle database runs, some things really need real-time (or almost real-time) monitoring: extent allocation errors, databases which are stuck because of an inability to archive redo log files, I/O errors... The quicker the DBA is aware of the problem, the sooner the problem is fixed.

Unfortunately, in most cases DBAs are alerted to the problem by angry calls from users, who are usually unaware of what is really going on and precious time is lost just trying to track down the problem.

The ALERTHAWK program first locates all the databases on a machine, then finds their alert files and regularly monitors the traces generated by Oracle. When errors are detected, the error, the relevant context and information ("Cause : ...,Action : ...") about the error and how to fix it are reported by mail to a list of users.

Installing ALERTHAWK on your machine

What you download from the Oriole Corporation web-site is a compressed tar file, named:

ALERTHAWK_for_<operating system>.tar.Z

You first have to uncompress it on your machine :

uncompress ALERTHAWK_for_<operating system>.tar.Z

then extract the files from the archive (it will create an ALERTHAWK directory under the current directory) :

tar xvf ALERTHAWK_for_<operating system>.tar

You must then change directory to ALERTHAWK where you must have 6 files :

READ.ME

alerthawk.o

alerthawkid.o

libalerthawk.a

mkalerthawk

alerthawk.mk

These files are all that's required to run, first a time limited free trial version of alerthawk or, once you have bought the license from Oriole Corporation, the unrestricted product.

Alerthawk is protected by a software key (a string of digits and letters) which will be forwarded to you on request for the free trial, or on receipt of payment for the unrestricted product.

As the key depends on the machine on which you will run alerthawk, you must provide Oriole Corporation with two things:

1. your identifier.

This identifier is a hexadecimal number returned by alerthawkid, which is built at the same time as alerthawk.

2. the license class you want for alerthawk.

Three license classes are available :

class 1 allows you to monitor a single database on your machine,

class 2 allows you to monitor two databases on your machine,

class 3 allows you to monitor any number of databases.

Once you have received your software key, you just have to set the environment variable ALERTHAWK_KEY to this value, by a command such as :

ALERTHAWK_KEY=<software key>; export ALERTHAWK_KEY

(Bourne shell)

setenv ALERTHAWK_KEY <software key>

(C shell)

This is best done in a command file which is run when you log on, such as .profile, .login or .cshrc, depending on your environment. Once the environment variable is set, you can enjoy alerthawk in all its glory.

ALERTHAWK operating mode

ALERTHAWK can be used in two modes :

either in 'interactive' mode.

Alert files are checked (optionally from a given date) and all errors encountered are written to the standard output,

or in 'daemon' mode.

The current contents of alert files is ignored but ALERTHAWK enters an infinite loop and checks for any new 'entry' in the alert files. When errors are encountered, they are either written to the standard output or mailed to database administrators. In this mode, ALERTHAWK spawns a child process the only function of which is to check that its father process is still alive. If the child process dies, the father process is signalled and spawns a new child process. If the father process dies, the child process spawns a new child process and assumes the role of the 'deceased' father process. Besides just looking for errors, ALERTHAWK also monitors database status : initially, it identifies databases and their status (up, down, crashed). For 'up' databases, the lgwr process is checked. If it disappears, the status is updated and the status change is reported. In the same way, alert file size is also checked and if the alert file starts growing again the status is updated and the change is reported.

Running ALERTHAWK

You can control how ALERTHAWK runs by entering the alerthawk command followed by various arguments; the program has been designed to give it an Oracle utility look-and-feel, which should give it a flavor familiar to any Oracle DBA.

To specify parameters, you use keywords:

Format: alerthawk KEYWORD=value or KEYWORD=(value1,value2,...,valueN)

Example:

alerthawk SYSTEM/MANAGER LOOP=Y SID=*

Keyword Description (Default) Keyword Description (Default) USERID username/password (INTERNAL) INTERVAL s between probes (60) PARFILE parameter filename SINCE errors since DDMMYYYY HELP display this screen (N) LOOP scan forever (N) SID SID list (*) SEARCHTRC scan .trc files (Y) MAILTO mailing list for output REDOCHECK check log allocation (Y)

ALERTHAWK parameters

USERID

this parameter is not normally used (as in most cases there is no connection to Oracle). It is used only in one case, which is the existence of an active Oracle instance for which we were unable to locate the alert file by other means (normally, init.ora files are searched). In this case, we connect to Oracle and query V$PARAMETER.

PARFILE

same as with exp/imp

SID

the default parameter (*) means that all the databases located on the machine are checked. This parameter can be used to restrict the monitoring to, for instance, production databases only. With an unlimited license you normally don't have to specify the default but if you do, it must of course be escaped to avoid being interpreted by the shell.

MAILTO

list of database administrators to be warned by mail. When a list of users is given, it is usually enclosed between parentheses which must be escaped, to avoid being interpreted by the shell.

INTERVAL

self-explanatory (daemon mode only)

SINCE

for the interactive mode only. By default, all errors found in alert files are reported. If a date is given, only what happened since this date is reported.

LOOP

when set to 'Y'. ALERTHAWK works in daemon mode (it must however explicitly be started as a background process, although 'nohup' is not necessary).

SEARCHTRC

in some cases, the information found in alert files can be complemented by information in .trc files (usually, traces generated by Oracle background or server processes). These files can be pretty big and reporting their contents can be disabled.

REDOCHECK

if set to 'Y', messages such as 'Cannot allocate new log' (which do not come with an ORA- error message) are also reported, otherwise they are ignored.

Notes

When ALERTHAWK is asked to mail its reports, it first writes to a file in /tmp. Although 'temporary', the same file is used throughout the life of the ALERTHAWK process. In order to prevent a cron job removing the file after a long quiet period, the file is 'touched' even if there is nothing to report.
It should normally be possible to delete a number of lines from an alert file while ALERTHAWK checks it. If the size of the alert file decreases, the modification is ignored and ALERTHAWK just 'remembers' the new file size for future checks.
ALERTHAWK lowers its priority by itself.
When ALERTHAWK is started by root and without a parameter, it always looks for a parameter file named alerthawk.prm located in the same directory as the alerthawk executable. This behavior is specific to root.
In daemon mode, ALERTHAWK should be started in the script which starts the database, typically by a command such as :

alerthawk loop=y mailto=$oraadmin@hq,oracle$ &