Using EELS

Processing an external file

Most log files have a fixed format that makes it relatively simple to segment the information into useful chunks. To import an external log file into the EELS database, you must segment your file into chunks of data that can then be inserted into a predefined database table. This predefined table is described in ``Database table overview''. For example, the following lines are from a Netscape Server access log: - - [07/Aug/1998:07:23:41 +0100] "GET / HTTP/1.0" 200 901 - - [07/Aug/1998:07:23:42 +0100] "GET /gif/uw7small.gif HTTP/1.0" 200 4458 - - [07/Aug/1998:07:23:42 +0100] "GET /info.shtml HTTP/1.0" 200 2931 - - [07/Aug/1998:07:23:43 +0100] "GET /gif/scologo.gif HTTP/1.0" 200 1520 - - [07/Aug/1998:07:23:43 +0100] "GET /cgi-bin/motd.cgi HTTP/1.0" 200 49
To import this file, you must first identify which fields of the access file you want to export, and to which database columns you want to map them.

Each line of the Netscape access file contains the following fields:

Decide which fields you want to use and to which columns you want to map them. For example:

Mapping fields to column names

Netscape Access field EELS Database column name
Client IP address OriginatorServiceName
Sysdate TimeOffset
Action EventSpecificInformation
File EventSpecificInformation
Protocol EventSpecificInformation
Status EventNumber
Content length Length
There are a number of considerations to take into account when splitting the log file into its component parts:

After you have decided which fields to export, and converted any fields that need converting, print them delimiting each field and record with field and record delimiters. For example, using ``#'' as a field and ``^'' as a record delimiter, the first line of the output shown above may be converted to: / HTTP/1.0#200#901^

NOTE: The format of ``TimeOffset'' (``902471021'') is described in detail in ``Converting the data type of a field''.

This output must then be piped into eels_log_import(1Meels).

Automatically updated columns

The eels_log_import(1Meels) utility automatically inserts some standard column values whenever you use it to import a row of data. You can override any of these values by specifying new values for them as part of your conversion script. The following columns are automatically updated by eels_log_import(1Meels):

A unique sequence number that identifies each record within the EELS database. Avoid altering this value as it may cause problems for you or other applications that use this database.

A sequence number for this record if there is more information than can be fitted in a single record. As with ``UniqEventID'', avoid altering this value as it may cause problems for you or other applications that use this database.

The process ID of eels_log_import(1Meels) at the time it imported the data into the database. If you specify a value for ``ProcessID'' from within your conversion script, that value is used instead.

The length of the data written to the database by eels_log_import(1Meels).

The version number of the EELS daemon. Avoid altering this value.

The time eels_log_import(1Meels) was used to import the data into the EELS database. This is expressed as the number of milliseconds from the beginning of the EPOCH. If you specify a value for ``TimeOffset'' from within your conversion script, that value is used instead. For more information see ``Converting the data type of a field''.

Converting the data type of a field

Before you insert any data into the EELS database, you must ensure that the data has the correct type and length. To determine the type and length of a particular field, see ``Database table overview''.

If any of the ``varchar'' fields exceed the stated length, eels_log_import(1Meels) creates additional rows to contain the remaining part of the field and increments the counter in the ``SequenceNumber'' column.

One of the most common problems with conversion to the correct data type is the conversion of the value for ``TimeOffset''. The ``TimeOffset'' is specified as being the number of milliseconds since the EPOCH. However, as in the example in ``Processing an external file'', the time offset is specified as a string. For example: [07/Aug/1998:07:23:43 +0100] If you are using Perl to convert you log files you can use the timelocal command of the ``Time::Local'' Perl module to convert the string version of a date into the number of seconds since the EPOCH. For more information, see the source example shown in ``A simple example using Perl''.

Field and record delimiters

When outputting the fields ready for processing by eels_log_import(1Meels), you must always separate fields with a field delimiter and records with a record delimiter. When choosing a delimiter, make sure that the delimiter does not occur naturally within any of your data.

If you are familiar with the construction of the log file you are exporting, you can probably pick a delimiter you know will not occur naturally. If you are unsure, you may want to make sure by first processing your datafile and escaping any potential delimiters. For example, in Perl, if you wanted to use ``#'' as a field delimiter, you could escape any potential hashes out of a data string by inserting a ``\'' before the hash, as shown in the following command:

   $ADataString_1 =~ s/#/\\#/g;
   $ADataString_2 =~ s/#/\\#/g;
   print $ADataString_1."#".$ADataString_2;

© 2004 The SCO Group, Inc. All rights reserved.
UnixWare 7 Release 7.1.4 - 22 April 2004