|
|
Most log files have a fixed format that makes it relatively simple to segment the information into useful chunks. To import an external log file into the EELS database, you must segment your file into chunks of data that can then be inserted into a predefined database table. This predefined table is described in ``Database table overview''. For example, the following lines are from a Netscape Server access log:
192.168.24.33 - - [07/Aug/1998:07:23:41 +0100] "GET / HTTP/1.0" 200 901 192.168.24.33 - - [07/Aug/1998:07:23:42 +0100] "GET /gif/uw7small.gif HTTP/1.0" 200 4458 192.168.24.33 - - [07/Aug/1998:07:23:42 +0100] "GET /info.shtml HTTP/1.0" 200 2931 192.168.24.33 - - [07/Aug/1998:07:23:43 +0100] "GET /gif/scologo.gif HTTP/1.0" 200 1520 192.168.24.33 - - [07/Aug/1998:07:23:43 +0100] "GET /cgi-bin/motd.cgi HTTP/1.0" 200 49To import this file, you must first identify which fields of the access file you want to export, and to which database columns you want to map them.
Each line of the Netscape access file contains the following fields:
Mapping fields to column names
| Netscape Access field | EELS Database column name |
|---|---|
| Client IP address | OriginatorServiceName |
| Sysdate | TimeOffset |
| Action | EventSpecificInformation |
| File | EventSpecificInformation |
| Protocol | EventSpecificInformation |
| Status | EventNumber |
| Content length | Length |
192.168.24.33#902471021#GET / HTTP/1.0#200#901^
This output must then be piped into eels_log_import(1Meels).
The eels_log_import(1Meels) utility automatically inserts some standard column values whenever you use it to import a row of data. You can override any of these values by specifying new values for them as part of your conversion script. The following columns are automatically updated by eels_log_import(1Meels):
Before you insert any data into the EELS database, you must ensure that the data has the correct type and length. To determine the type and length of a particular field, see ``Database table overview''.
If any of the ``varchar'' fields exceed the stated length, eels_log_import(1Meels) creates additional rows to contain the remaining part of the field and increments the counter in the ``SequenceNumber'' column.
One of the most common problems with conversion to the
correct data type is the conversion of the value
for ``TimeOffset''. The ``TimeOffset'' is
specified as being the number of milliseconds since the
EPOCH. However, as in the example in
``Processing an external file'',
the time offset is specified as a string. For example:
[07/Aug/1998:07:23:43 +0100]
If you are using Perl to convert you log files you can
use the timelocal command of the ``Time::Local''
Perl module to convert the string version of a date into
the number of seconds since the EPOCH. For more
information, see the source example shown in
``A simple example using Perl''.
When outputting the fields ready for processing by eels_log_import(1Meels), you must always separate fields with a field delimiter and records with a record delimiter. When choosing a delimiter, make sure that the delimiter does not occur naturally within any of your data.
If you are familiar with the construction of the log file you are exporting, you can probably pick a delimiter you know will not occur naturally. If you are unsure, you may want to make sure by first processing your datafile and escaping any potential delimiters. For example, in Perl, if you wanted to use ``#'' as a field delimiter, you could escape any potential hashes out of a data string by inserting a ``\'' before the hash, as shown in the following command:
$ADataString_1 =~ s/#/\\#/g; $ADataString_2 =~ s/#/\\#/g; print $ADataString_1."#".$ADataString_2;