ALPS: Advanced Logfile Processing System

General Description

We started to develop an advanced logfile processing system with

a simple, intuitive, but sufficiently expressive query language to describe patterns of interest in log message sequences,

a layer mechanism such that events (described by queries) can build new logical and modular structures (e.g., a series of micro events triggers a macro event which again triggers some actions),

where on top a machine learning component can be applied (e.g., the system can learn new structures based on reoccurring log patterns).

Agenda

Define formal syntax and semantics of the event query language

Visual editor to generate patterns from log message sequences

Implement a prototype for this query language

Build an efficient and fast implementation (in C)

Design APIs, interfaces, and layout of the machine learning component

Think about integration with existing tools in a UNIX system (e.g., how to trigger system calls in a generic way for corresponding modular structures created by the machine learning component)

Consider the export of generated macro events and logical structures into standard formats (e.g., a first investigation showed that there is an interoperable XML format for communicating intrusion detection messages)

Examples

Example (Security, Intrusion Detection)
Assume you process log files from a distributed net of desktop computers. Then report a warning for more than three failed (SSH) login attempts (but no successful login inbetween): [fail,user=X]{3,} ![success,user=X] ==> [login=fail,user=X]

Example (Database Management System, Performance Alert)
Consider a DBMS that supports multi granularity locking and you want to trigger a performance alert (e.g., send an e-mail to the database admin) if more than 100 records are written in the same page (and hence the DBMS should switch the granularity, i.e., lock whole pages instead of tuples): [lockX,page=Y,record=_]{100,*} ==> sendMail()

Example (Machine Learning Component, dmesg)
Take the messages you typically find in a syslog of a UNIX system (e.g., as shown by dmesg) and start learning clusters. E.g., when a USB stick is attached, you get about 10 messages but in the end the sysadmin would just like to see the single event usb attached.

As you see the log system should be able to cover a broad range of possible applications; whenever a bunch of log messages is generated there is a need for (semi-)automatic analysis. The need for such a system also arises when we want to perform actions not supported by the individual system (e.g., the DBMS might adapt the lock granularity but might not be able to send an e-mail in this situation).

ALPS: Advanced Logfile Processing System

General Description

Agenda

Examples

Contact Persons