DK 'Log : /ossim/tutorials/tut4_correlation

Tutorial 4: Correlation engine primer
Mon, 10 Dec 2007

Introduction

In order to answer to a recent forum post I had to do a quick research since it had been some time since I last tested this.
The exact question was:

Hello,

Is there a document talking about how the directives are processed?  One question
that I have is if you have multiple directives created and an event comes in
that matches the initial states of more than a single directive will both actually
process the event, or only the first match (which I think is the case)?

Thanks for any clarification you can provide.

Stephen

This post gives a bit of insight to how the correlation engine works and features some simple, custom made directives that help me answer that question.

The test environment features two events belonging to the ssh plugin (plugin_id 4003):

SSH password failed (plugin_sid 1)
SSH password accepted (plugin_sid 7)

In order to test this I've created three directives (plugin_id 1505)

Test directive 21, grouping one login failure and one success
Test directive 22, grouping one login failure and one success
Test directive 23, used in the second case, grouping those two

So, with all of this in place it was easy to simulate this failing a login and succeeding afterwards.

::start here

Case 1: Two concurrent correlation rules

Directives

Put these into generic.xml for example

<directive id="21" name="Testing concurrency 1" priority="5">
  <rule type="detector" name="Level 1:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" >
        <rules>
                <rule type="detector" name="Level 1:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" />
        </rules>
  </rule>
</directive>

<directive id="22" name="Testing concurrency 2" priority="5">
  <rule type="detector" name="Level 2:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" >
        <rules>
                <rule type="detector" name="Level 2:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" />
        </rules>
  </rule>
</directive>

Logs

2007-12-10 04:19:56 OSSIM-Message: Event received: event id="0" alarm="0" type="detector" 
date="2007-12-10 04:19:53" plugin_id="4003" plugin_sid="1" src_ip="10.0.1.3" src_port="64270" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1" 
log="Dec 10 04:19:53 ossim sshd[11366]: Failed password for root from 10.0.1.3 port 64270 ssh2" 
username="root" 

2007-12-10 04:20:06 OSSIM-Message: Event received: event id="0" alarm="0" type="detector" 
date="2007-12-10 04:20:05" plugin_id="4003" plugin_sid="7" src_ip="10.0.1.3" src_port="64270" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1" 
log="Dec 10 04:20:05 ossim sshd[11366]: Accepted password for root from 10.0.1.3 port 64270 ssh2" 
username="root" 

2007-12-10 04:20:06 OSSIM-Message: Event received: event id="3" alarm="0" type="detector" 
date="2007-12-10 04:20:06" plugin_id="1505" plugin_sid="21" src_ip="10.0.1.3" src_port="64270" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10" 
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 1, Priority: 5
Rule 1 [2007-12-10 04:20:00] [4003:1] [Rel: 7] 10.0.1.3:64270 -> 0.0.0.0:0
Rule 2 [2007-12-10 04:20:06] [4003:7] [Rel: 10] 10.0.1.3:64270 -> 0.0.0.0:0
" 

2007-12-10 04:20:07 OSSIM-Message: Event received: event id="4" alarm="0" type="detector" 
date="2007-12-10 04:20:06" plugin_id="1505" plugin_sid="22" src_ip="10.0.1.3" src_port="64270" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10" 
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 2, Priority: 5
Rule 1 [2007-12-10 04:20:01] [4003:1] [Rel: 7] 10.0.1.3:64270 -> 0.0.0.0:0
Rule 2 [2007-12-10 04:20:06] [4003:7] [Rel: 10] 10.0.1.3:64270 -> 0.0.0.0:0

Explanation

Here we wrote two directives with exactly the same events matching, naming it differently. That way we'll be able to answer the concurrency question since I honestly don't remember how that part worked.
As you can see in the logs, two events arrived, and two directives got matched.

Result: two alarms

Case 2: Two concurrent correlation rules, a third grouping rule

Directives

<directive id="21" name="Testing concurrency 1" priority="5">
  <rule type="detector" name="Level 1:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" >
        <rules>
                <rule type="detector" name="Level 1:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" />
        </rules>
  </rule>
</directive>

<directive id="22" name="Testing concurrency 2" priority="5">
  <rule type="detector" name="Level 2:1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="4003" plugin_sid="1" >
        <rules>
                <rule type="detector" name="Level 2:2" reliability="10"
                occurrence="1" from="ANY" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="4003" plugin_sid="7" />
        </rules>
  </rule>
</directive>

<directive id="23" name="Aggregated test directive" priority="5">
  <rule type="detector" name="Directive 1" reliability="7"
    from="ANY" to="ANY" port_from="ANY" port_to="ANY"
    time_out="160" plugin_id="1505" plugin_sid="21,22" >
        <rules>
                <rule type="detector" name="Directive 2" reliability="10"
                occurrence="1" from="ANY" time_out="300" to="ANY" port_from="ANY" port_to="ANY"
                plugin_id="1505" plugin_sid="21,22" sticky="true" sticky_different="SRC_IP" />
        </rules>
  </rule>
</directive>

Logs

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="0" alarm="0" type="detector" 
date="2007-12-10 06:24:55" plugin_id="4003" plugin_sid="7" src_ip="10.0.1.3" src_port="64620" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" asset_src="1" asset_dst="1" 
log="Dec 10 06:24:55 ossim sshd[19967]: Accepted password for root from 10.0.1.3 port 64620 ssh2" 
username="root" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="35" alarm="0" type="detector" 
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="21" src_ip="10.0.1.3" src_port="64620" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10" 
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 1, Priority: 5
Rule 1 [2007-12-10 06:24:51] [4003:1] [Rel: 7] 10.0.1.3:64620 -> 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [4003:7] [Rel: 10] 10.0.1.3:64620 -> 0.0.0.0:0
" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="36" alarm="0" type="detector" 
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="22" src_ip="10.0.1.3" src_port="64620" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10" 
asset_src="1" asset_dst="1" data="directive_event: Testing concurrency 2, Priority: 5
Rule 1 [2007-12-10 06:24:51] [4003:1] [Rel: 7] 10.0.1.3:64620 -> 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [4003:7] [Rel: 10] 10.0.1.3:64620 -> 0.0.0.0:0
" 

2007-12-10 06:24:56 OSSIM-Message: Event received: event id="37" alarm="0" type="detector" 
date="2007-12-10 06:24:56" plugin_id="1505" plugin_sid="23" src_ip="10.0.1.3" src_port="64620" 
dst_ip="0.0.0.0" sensor="10.0.1.50" interface="eth2" protocol="OTHER" priority="5" reliability="10" 
asset_src="1" asset_dst="1" data="directive_event: Aggregated test directive, Priority: 5
Rule 1 [2007-12-10 06:24:56] [1505:21] [Rel: 3] 10.0.1.3:64620 -> 0.0.0.0:0
Rule 2 [2007-12-10 06:24:56] [1505:22] [Rel: 10] 10.0.1.3:64620 -> 0.0.0.0:0

Explanation

This is exactly the same event as before with two added features: a grouping directive with our two recently created directives and a couple of keywords that prevent our grouping directive to fire twice, since the same event is used for level 1 and level 2. Using sticky="true" and sticky_different="SRC_IP" we are telling the engine to not spawn a second instance of the directive if it comes from the same source IP.
I think everything is quite self-explanatory (having read the other correlation docs avaiable at the ossim.net docs section).

Result: one alarm, grouping two others

Foot note

I've detected some glitches during these tests, mainly in the alarm panel grouping code. As you can see on the screenshots event count isn't correctly interpreted and separate alarms are being spawned, besides the grouped one. I'll check that asap.

posted at: 12:47 | path: /ossim/tutorials | permanent link to this entry | 3 comments |
Tags: correlation, directives, tutorial, ossim

* Posted by Alberto Roman at Mon Dec 10 14:23:50 2007

And remember that you have more information about the correlation process in http://www.ossim.net/dokuwiki/doku.php?id=documentation:serverd#correlation

* Posted by Nitin at Thu Jun 19 14:36:31 2008

Hello Sir

Actually I wanted to write my own directive. I read this tutorial its very helpful for me to write new directives. Can you please tell me apart from editing generic.xml should I have to edit some other files too?

Thanks in advance.

* Posted by Nitin at Thu Jun 19 14:41:03 2008

Can you please refer any documet to get better understandability of correlation.