Using Custom Functions in USM and OSSIM for Additional Parsing of Log Data

August 25, 2016  |  Dmitry Shulinin

Good day everybody.

Today I’m going to examine and explain the functionality of “custom functions”, used in OSSIM/USM parsers. Those are the functions meant to modify the data after the agent finishes parsing. There are several built-in functions such as:

  • “resolv()”, which resolves the IP by hostname
  • “normalize_date()” which normalizes the timestamp

In the following example I will explain how to use “custom functions” for additional parsing of log data.

What’s it for?

Let’s say we have a data source (which is an application) that writes its logs to a database (mysql). Using OSSIM parser we can map the fields from the database to event fields in OSSIM. One field of DB maps to one event field in OSSIM. No additional parsing is possible. And that’s okay if we have exactly the data we need in the fields of our database. But what if we have a field in our database that contains not a single value, but a whole string, for example: “issued command: ls /root; result: ACCEPT”. And we need to get the issued command “ls /root” and the result – “ACCEPT” and put this data into certain OSSIM fields? Obviously we can’t do that using only the functionality of the parser (because we can only put the whole string into an event field). This is where custom functions come into play. With the help of custom functions we will parse the string and get the exact data we want and put it into desired OSSIM event fields.

Prerequisites

We have a test database (fig.1) which contains application logs.

application logs for ossim parser customization
Figure 1 – Test database

I developed a parser which parses the database and puts the data from the DB fields into the event fields of OSSIM so I will not describe the parser itself (thankfully, there is enough info about that in official AlienVault documents).

What I will do is write two custom functions. The first one (parse_command) will parse the fields from the column “message” of the DB and get the issued command value. The other one (parse_result) will parse the fields from the column “message” of the DB and get the result of the command.

Solution

The solution consists of:

  • Writing the function “parse_command”;
  • Writing the function “parse_result”;
  • Adding the functionality of our new functions to OSSIM parser.

Writing the function “parse_command”

First of all I have to remind you that OSSIM custom functions should be written in Python. So if you are not familiar with Python, please explore tutorials about programming in python. To use a custom function in OSSIM you should first add it to the file:

/usr/share/alienvault/ossim-agent/ParserUtil.py

You can add your function anywhere in the file (just don’t add it inside the text of another function). I added mine after “dummy function”.

The following is the text of “parse_command” function:

def parse_command(input):
   res = re.search(r'command:.*;', input)
​   return (res.group(0).split(": ")[1].strip(";"))

For those familiar with Python it’s easy to understand that this function gets the value passed to it and applies a regular expression to it. The regular expression extracts the data that matches the regular expression “command:.*;”. After that it cuts away the word “command:” as well as the trailing semicolon. Then it returns what’s left, which is the text of the issued command.

Writing the function “parse_result”

This function looks pretty similar to the “parse_command” function.

def parse_result(input):
   res = re.search(r'result:s+S+', input)
   return (res.group(0).split(": ")[1])

Custom parser tuning

After we have added both of our functions to the “ParserUtil.py” file we need to teach our custom parser to use them. The main section of the “db_logs.cfg” file (which is the parser config) looks like this:

[query]
query="select event_id, date, event_type, username, message from data_table where event_id > $1;"
regexp=
ref=0
date={normalize_date($1)}
plugin_sid={translate($2)}
username={$3}
userdata1={$4}
userdata2={$2}

Let’s say we want to put the data retrieved by the “parse_command” function into the “userdata3” field and the data, retrieved by “parse_result” function into “userdata4” field of the OSSIM schema.

To accomplish this we would need to add two lines to the config:

userdata3={parse_command($4)}
​userdata4={parse_result($4)}

The first line tells the parser to get $4 value (that is, for example “issued command: ls –la /home; result: ACCEPT”) and apply the “parse_command” function to it. The output is “ls –la /home”.

The second line tells the parser to get $4 value and apply the “parse_result” function to it. The output is “ACCEPT”.

After doing all these manipulations we should restart the ossim-agent:

/etc/init.d/ossim-agent restart

After the agent restarts it is advised to check the agent.log for errors (in case there is some mistake with the parser or custom functions):

tail -f /var/log/alienvault/agent/agent.log|grep ERROR

If everything was done right we should see a similar output in the OSSIM GUI (Fig. 2).

ossim gui for parsing log data
Figure 2 – Parsed events in the OSSIM GUI

Improvements

As you may have already noticed, the two functions which were developed for this blog look quite similar. So I wondered if I could combine them and address certain parts of the parsed message by a key.

For example, I have a function “parser_general()” which accepts two variables – the text of the log message and the key. The key is used to parse certain parts of the message. In this case I would address the issued command like this:

parse_general(“issued command: ls –la /home; result: ACCEPT”, issued_command)

Output – “ls –la /home; result

And I would address the result of the command like this:

parse_general(“issued command: ls –la /home; result: ACCEPT”, result)
​Output – “ACCEPT”

I wrote this function and successfully tested it in Python but OSSIM seems to be able to take only ONE argument for its “custom functions” at this time. Maybe sometime in the future AlienVault will expand this functionality, but even now the abilities of “custom functions” are vast and very useful.

Thanks for reading. Hope the info was useful.

And have a good day!

 

Post-publication: Thanks for pointing out an issue @pktinspector

To prevent possible deletion of custom functions added to parser.py during an OSSIM update they should be put in a separate file, for example “/usr/share/alienvault/ossim-agent/plugin/db_logs_func.cfg” in the following manner:

Start Function  parse_general
      def parse_command(input):
             res = re.search(r'command:.*;', input)
             return (res.group(0).split(": ")[1].strip(";"))
​End Function

 Start Function  parse_general
      def parse_result(input):
             res = re.search(r'result:s+S+', input)
             return (res.group(0).split(": ")[1])
​End Function

The following modification should be applied to “config” section of plugin config file “db_logs.cfg”:

[config]
​custom_functions_file=/etc/ossim/agent/plugin/db_logs_func.cfg

 

About the Author

Dmitry Shulinin has completed over 8 years in the specialist field of information security and system administration. He gained experience implementing different information security solutions like firewalls, IPS, antivirus systems, DLP and SIEM. He also developed skills in detecting and mitigating attacks and conducting computer forensics. His interest is in automation of processes and creating something new, which led him learning scripting languages (bash) and python. The latter was also found to be very suitable in data analysis, which in conjunction with SIEM technologies is his main interest at present.
LinkedIn
Facebook

Share this with others

Tags: ossim, logging, usm

Get price Free trial