Posted on

using filebeat to watch a log

In filebeat.yml,  add the file you are watching to the ‘paths’ section as follows:

paths:
- /var/log/my.log
### Logstash as output
logstash:
# The Logstash hosts
hosts: ["192.168.0.1:5044"]

This will forward the contents of my.log to logstash.

If, like me, you like to keep traffic and  load to a minimum,  then you can add this to filebeat.yml:

scan_frequency: 180s

And if, in your logstash config, you want to be able to know which input you are handing, then define the ‘type’ as follows:

document_type:  mylog

 

Posted on

watching logs with perl

When you have a log file with lots of unpredictably formatted entries, it can be difficult to come up with a nice grok filter to parse it. This is frustrating if you only want a small amount of data from a very big log.
Below is a quickly written Perl script to watch a log file and print out a summary of the number of times a GET request, indicated with ‘q=’, was made for each 10 minute interval.
This gave me a nice summary.log which looked like this:

hits-per-10m=35,server=myservername,logdate=201701221032,year=2017,month=01,date=22,hour=10,minute=32,second=45


#!/usr/bin/perl
# Watch a log and write key-value pairs to a file

use File::Tail;
use URI::Escape ;

$file = File::Tail->new("/var/log/myfile.log");
%months = qw( Jan 1 Feb 2 Mar 3 Apr 4 May 5 Jun 6 Jul 7 Aug 8 Sep 9 Oct 10 Nov 11 Dec 12);
($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime();
$year=$year+1900;
$hold = int($min / 10);
$count=0;
$lasthour = 99;

while (defined(my $line= $file->read)) {
$line=~s/\s+/ /g;
@parts= split /[ ]/,$line;
# in my log, I could parse the values from the data line as follows
$server=$parts[3];
$month=$months{$parts[0]};
$date=$parts[1];
($hour,$min,$sec) = split /\:/, $parts[2];
($ipaddr,$port) = split /\:/,$parts[5];
# here I want to count the number of values for each 10 minute interval
# the problem with logs is that there may not be a log line for every minute
## so I use $hold to indicate whether to hold the data or write it to a file
## And in case the hour has changed since the last log entry, this is also checked
if ($lasthour != $hour) {
$hold = int($min / 10);
}
$lasthour = $hour;
if ( ($min >= 0) && ($min <=9) ){
if ($hold == 0 ) {
&printCount ;
$hold = 1;
}
}
if ( ($min >= 10) && ($min <=19) ){
if ($hold <= 1 ) {
&printCount ;
$hold = 2;
}
}
if ( ($min >= 20) && ($min <=29) ){
if ($hold <= 2 ) {
&printCount ;
$hold = 3;
}
}
if ( ($min >= 30) && ($min <=39) ){
if ($hold <= 3 ) {
&printCount ;
$hold = 4;
}
}
if ( ($min >= 40) && ($min <=49) ){
if ($hold <= 4 ) {
&printCount ;
$hold = 5;
}
}
if ( ($min >= 50) && ($min <=59) ){
if ($hold <= 5 ) {
&printCount ;
$hold = 6;
}
}
$count++;
$lastmin = $min;
# check if the log entry has the value you are counting
# in this case I was looking for the search term which followed 'q='
if ($line =~ /GET/) {
($junk,$q) = split /[[\&\?]q=/, $line;
($q, $junk) = split /[\& ]/, $q;
$q = uri_unescape($q);
# remove any commas because I will be using them as separators
$q =~s/\,/ /g;
}
} # end while

sub printCount {
# print the time and the count of searches for the 10 minute period
open (OUT,">>/var/log/summary.log");
print OUT "hits-per-10m=$count,server=$server,logdate=$year$month$date$hour$min,year=$year,month=$month,date=$date,hour=$hour,minute=$min,second=$sec\n";
close OUT;
$count = 0;
}