IBM Support

Log Analysis - Custom GROK Pattern

Technical Blog Post


Abstract

Log Analysis - Custom GROK Pattern

Body

There are times when you might have a need to filter some events in Logstash but could not figure out how. Readily available GROK patterns do not suit your needs.

 

First of all, for those that do not have an idea what GROK is, let's look at an example:

 

input {
        tcp {
                port => 5588
                type  => 'Win32-EventLog'
                codec =>plain {
                        charset => "ASCII-8BIT"
                }
        }
}

filter{
    if [type] == "Win32-EventLog"{
                grok {
                        match => ["message","%{WINLFAMESSAGE}"]
                        patterns_dir => ["/home/danielyeap/LogAnalysis/Logstash/logstash-2.2.1/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns"]
                }

                mutate {
                        replace =>["message","%{text}"]
                        add_tag => [ "lfaproc" ]
                }

 

Now, let's examine what those few lines of codes really do:

 

[danielyeap@rec1 patterns]$ grep WINLFA grok-patterns
WINLFAMESSAGE (?m)<START>.*TimeGenerated='%{DATA:TimeGenerated}';EventLog='%{DATA:EventLog}';text='%{DATA:text}';RemoteHost='%{DATA:RemoteHost}';EventID='%{DATA:EventID}';Level='%{DATA:Level}';User='%{DATA:User}';hostname='%{DATA:hostname}';logpath='%{DATA:logpath}';EventSource='%{DATA:EventSource}';Keywords='%{DATA:Keywords}';END

 

You can see that the pattern definition is a bunch of regular expression and some pre-defined patterns (in this case, DATA pattern).

 

So, what does this regular expression do to help you filter or match your Logstash events? Check this out...

 

{
         "@version" => "1",
       "@timestamp" => "2017-10-18T08:21:11.629Z",
             "host" => "lahost.ibmtest.com",
             "port" => 49315,
             "type" => "Win32-EventLog",
    "TimeGenerated" => "Oct 18 16:21:16 2017",
         "EventLog" => "The Multimedia Class Scheduler service entered the stopped state.",
             "text" => "EventLog:System,Oct 18 16:21:16 2017,Information,N/A,Service_Control_Manager,Classic,7036,The Multimedia Class Scheduler service entered the stopped state.",
          "EventID" => "7036",
            "Level" => "Information",
             "User" => "N/A",
         "hostname" => "lahost.ibmtest.com",
          "logpath" => "WindowsOSEventsLFA",
      "EventSource" => "Service_Control_Manager",
         "Keywords" => "Classic",
             "tags" => [
        [0] "lfaproc"
    ],
      "Description" => "EventLog:System,Oct 18 16:21:16 2017,Information,N/A,Service_Control_Manager,Classic,7036,The Multimedia Class Scheduler service entered the stopped state.",
         "Hostname" => "lahost.ibmtest.com",
             "path" => "winevt.log",
        "timestamp" => "Oct 18 16:21:16 2017"
}

 

** Some fields do not exist because they have undergone further processing like rename or delete.

 

In short, a pattern is able to further breakdown an event into fields that are useful to you based on regular expression.

 

Ok, it looks useful, but how do you define your own Logstash pattern? 

 

Easy!! You just need to declare the "patterns_dir" (see above), create a file and put the pattern in the file. 

 

[danielyeap@rec1 ~]$ cd /home/danielyeap/LogAnalysis/Logstash/logstash-2.2.1/vendor/bundle/jruby/1.9/gems/logstash-patterns-core-2.0.2/patterns
[danielyeap@rec1 patterns]$ ls -l
total 100
-rwxr-xr-x 1 danielyeap danielyeap 1197 Feb 12  2016 aws
-rwxr-xr-x 1 danielyeap danielyeap 4831 Feb 12  2016 bacula
-rwxr-xr-x 1 danielyeap danielyeap 2154 Feb 12  2016 bro
-rwxr-xr-x 1 danielyeap danielyeap  879 Feb 12  2016 exim
-rwxr-xr-x 1 danielyeap danielyeap 9544 Feb 12  2016 firewalls
-rwxr-xr-x 1 danielyeap danielyeap 8854 Dec 12 12:43 grok-patterns
-rwxr-xr-x 1 danielyeap danielyeap 3251 Feb 12  2016 haproxy
-rwxr-xr-x 1 danielyeap danielyeap 1339 Feb 12  2016 java
-rwxr-xr-x 1 danielyeap danielyeap 1087 Feb 12  2016 junos
-rwxr-xr-x 1 danielyeap danielyeap 1037 Feb 12  2016 linux-syslog
-rwxr-xr-x 1 danielyeap danielyeap   49 Feb 12  2016 mcollective
-rwxr-xr-x 1 danielyeap danielyeap  190 Feb 12  2016 mcollective-patterns
-rwxr-xr-x 1 danielyeap danielyeap  614 Feb 12  2016 mongodb
-rwxr-xr-x 1 danielyeap danielyeap 9597 Feb 12  2016 nagios
-rwxr-xr-x 1 danielyeap danielyeap  142 Feb 12  2016 postgresql
-rwxr-xr-x 1 danielyeap danielyeap  845 Feb 12  2016 rails
-rwxr-xr-x 1 danielyeap danielyeap  104 Feb 12  2016 redis
-rwxr-xr-x 1 danielyeap danielyeap  188 Feb 12  2016 ruby
[danielyeap@rec1 patterns]$ grep WINLFA grok-patterns
WINLFAMESSAGE (?m)<START>.*TimeGenerated='%{DATA:TimeGenerated}';EventLog='%{DATA:EventLog}';text='%{DATA:text}';RemoteHost='%{DATA:RemoteHost}';EventID='%{DATA:EventID}';Level='%{DATA:Level}';User='%{DATA:User}';hostname='%{DATA:hostname}';logpath='%{DATA:logpath}';EventSource='%{DATA:EventSource}';Keywords='%{DATA:Keywords}';END

 

** NOTE: In the example above, I chose to append the pattern into the default "grok-patterns" file. It is best if you use your own file.

 

Hope that helps!

 

[{"Business Unit":{"code":"BU053","label":"Cloud & Data Platform"},"Product":{"code":"","label":""},"Component":"","Platform":[{"code":"","label":""}],"Version":"","Edition":"","Line of Business":{"code":"","label":""}}]

UID

ibm11081659

Page Feedback