Parse LOG files engine

Parse LOG files engine, enabled in ipsetd-ng and sensord-ng. LOG engine is a powerful, versatile tool that provides universal query access to text-based data such as log files.

Obtaining lines of the log file occurs on possible scenarios:

  • through programs supporting logging FIFO/Pipe exchange mode.
  • using system functions SYSTEMD daemon.

For this methods, you can use any program who can write logs in FIFO/Pipe/SYSTEMD.
Examples settings program supports FIFO/Pipe/SYSTEMD exchange mode.

Examples settings SYSLOG program supports FIFO/Pipe exchange mode:

    # cat /etc/syslog.conf

    kern.error      /dev/console

    # Redirect to ipsetd-ng/sensord.ng log stream
    # xinetd: local2
    # apache: local3
    # system auth: authpriv
    authpriv.*,local3.*,local2.*    |/var/log/syslog.fifo

    *.bootps            /var/log/boot.bootps
    *.info              /var/log/boot.all
    *.error,*.emerg         /var/log/boot.error
    *.debug,local7.*        /var/log/boot.debug
    local0.*            /var/log/boot.other
    authpriv.*,local3.*     /var/log/secure
    mail.*              /var/log/maillog
    cron.*              /var/log/cron
    kern.*              /var/log/kern.error
    uucp,news.crit          /var/log/spooler
    local6.*            /var/log/upnpd.log


Examples settings SYSLOG-NG program supports FIFO/Pipe exchange mode:

    # cat /etc/syslog-ng.conf

    @version: 3.4.8

    options {
        chain_hostnames(no);
        keep_hostname(yes);
        use_dns(no);
        owner("root");
        group("adm");
        create_dirs(yes);
        dir_perm(0755);
        perm(0640);
        flush_lines(0);
        stats_freq(0);
        stats_level(0);
        mark_freq (0);
        log_fifo_size(12800);
        log_msg_size(8192);
        time_reopen (10);
    };

    source s_sys {
        file ("/proc/kmsg" program_override("kernel"));
        unix-stream ("/dev/log");
        internal();
    };

    ...

    # Redirect to ipsetd-ng/sensord.ng log stream
    # xinetd: local2
    # apache: local3
    # system auth: authpriv

    filter f_filter0 { (facility(local2) or facility(local3) or facility(authpriv)); };
    destination d_ipset{ pipe(/var/log/syslog.fifo); };
    log { source(s_sys); filter(f_filter0); destination(d_ipset); };



Examples settings JOURNAL LOG system:

    # show available systemd unit

        $ journalctl -F _SYSTEMD_UNIT
        ....

    # insert available shared resource to ipsetd-ng or sensord-ng config file:
    # logparse -> source = _SYSTEMD_UNIT=service.name
    #   example: _SYSTEMD_UNIT=sshd.service
    #   example: _SYSTEMD_UNIT=vsftpd.service
    #   example: _SYSTEMD_UNIT=telnetd.service
    #   example: _SYSTEMD_UNIT=YOU.service


For obtaining logs from the HTTPD APACHE server, is used module mod_log_syslog:

    # cat /etc/httpd/conf/httpd.conf

    <VirtualHost *:80>
        CustomLog syslog:local3.debug common
        ...
    </VirtualHost>


Define XINETD general logging characteristics:

    # cat /etc/xinetd.conf

    defaults
    {
            log_type        = SYSLOG local2
            log_on_failure  = HOST
            log_on_success  = PID HOST DURATION EXIT

            cps             = 50 10
            instances       = 50
            per_source      = 10
    }
    includedir /etc/xinetd.d



To effectively block spam or hackers bots on the web server, you can use the following procedure:

  • 1. created with the CSS styles invisible visitors online form in main page, assign a sensor bots URL is not valid for execution normal users, example:
  • 
        <div style="display:none">
        <form action="/badpath/spampost.cgi" method="post" name="spampost">
            <input type="email" name="email" />
            <input type="text"  name="text" />
            <button type="submit" name="submit" value="true">hello spamer!</button>
        </form>
        </div>
    
    
    
  • 1.1 (optional) redirect SPAM sensor URL for virtual locations, or skip this step, and create real directory /badpath in Web root path:
  • 
        # Create virtual directory /badpath
    
        # for Apache Web server
        <Locations /badpath/spampost.cgi>
            RewriteEngine on
            RewriteBase /var/www/htdocs
            RewriteRule ^badpath/(.*)$ /$1 [H=cgi-script,NC,L]
        </Locations>
        # or
        Alias /badpath/ /var/www/htdocs/
    
        # for Nginx Web server
        location ~* ^/badpath/spampost.cgi$ {
            root    /var/www/htdocs;
            rewrite ^(.*)$ /spampost.cgi last;
        }
        # or
        location /badpath/ {
            alias /;
        }
    
        # or create real directory in Web root path
        mkdir -p /var/www/htdocs/badpath
        touch /var/www/htdocs/badpath/spampost.cgi
        chmod 755 /var/www/htdocs/badpath/spampost.cgi
    
    
    
  • 2. add form URL to file robots.txt to avoid blocking search engines:
  • 
        User-agent: *
        Disallow: /badpath/
        ...
    
    
    
  • 3. create a regular expression to block this URL, such as a LOG format Web server Apache:
  • 
        ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\s.*(?:badpath|spampost).*$
    
    
    
  • 4. and insert this block include regular expression to configuration file in section [logmatch]:
  • 
        progid = httpd
        family = ipv4
        raw = ([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\s.*(?:badpath|spampost).*$
    
    
    
  • 5. (optional) create a script, such as PHP, which allows you to control which addresses were blacklisted lock:
  • 
        <?php
        $f1 = fopen( __DIR__ . '/writable-dir/spampost.log', 'a+' );
        fwrite( $f1, print_r($_POST, true) );
        fwrite( $f1, print_r($_SERVER, true) );
        fclose( $f1 );
        ?>
    
    
    

Parse RULE: PCRE - Perl Compatible Regular Expressions

The PCRE is a set of functions that implement regular expression pattern matching using the same syntax and semantics as Perl 5, with just a few differences.
Regular expressions are essentially represent a powerful and very flexible language descriptions for the search string for a match.

Additional information about PCRE Regex

Example of Regular Expressions:

refused.*(?:telnet|ftp|ssh).*\s([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).?$
FAIL:.*(?:telnet|telnetd|ftp|vsftpd).*(?:libwrap\sfrom=)([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$
FAIL:.*(?:telnet|telnetd|ftp|vsftpd).*(?:libwrap\sfrom=)([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).?$
FAIL:.*(?:telnet|ftp|vsftpd|ssh).*\sfrom=::ffff:([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})$
FAIL:.*(?:telnet|ftp|vsftpd|ssh).*\sfrom=::ffff:([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}).?$
fake\sauth\srejection\sfor\suser.*tag=([0-9a-f]{8})$
fake\sauth\srejection\sfor\suser.*tag=([0-9a-f]{8}).?$
([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\s.*(?:wp-config.php|wp-login.php|bitrix).*$
([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})\s.*CONNECT\s.*(?:mail|mx).*:25.*$
\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b
^([0-9a-fA-F][0-9a-fA-F]:){5}([0-9a-fA-F][0-9a-fA-F])$
^([a-zA-Z0-9]([a-zA-Z0-9\-]{0,61}[a-zA-Z0-9])?\.)+[a-zA-Z]{2,6}$
(?i)^(?!^(PRN|AUX|CLOCK\$|NUL|CON|COM\d|LPT\d|\..*)(\..+)?$)[^\\\./:\*\?\"<>\|][^\\/:\*\?\"<>\|]{0,254}$
[-+]?(?:\b[0-9]+(?:\.[0-9]*)?|\.[0-9]+\b)(?:[eE][-+]?[0-9]+\b)?
^(?i:(?=[MDCLXVI])((M{0,3})((C[DM])|(D?C{0,3}))?((X[LC])|(L?XX{0,2})|L)?((I[VX])|(V?(II{0,2}))|V)?))$
(19|20)\d\d([- /.])(0[1-9]|1[012])\2(0[1-9]|[12][0-9]|3[01])

Example settings iptables and ipset to block an IP address

You can group multiple ipsets of different types into a single "setlist" (an ipset of type list:set) which is still treated as a single ipset by iptables (therefore, you only need one rule to match packets against multiple ipsets). You can bind ipsets together (e.g. a list of addresses and a list of ports). The man page ipset explains it all.
For userfw users, see IPSet modules ported to userfw

Example complete ipset and iptables rules on server:


    #!/bin/bash
    #
    # edited local net range
    NETLOC="192.168.0.0/16"
    # edited world wide net range
    NETWIDE="1.2.3.0/24"
    # use tcp ports for private services
    TCPPRIV="20 21 23 3306"
    # use udp ports for private services
    UDPPRIV="20 21"

    /usr/bin/buildblst.sh --init-ipset

    ipset -q -A whitenet ${NETLOC}
    ipset -q -A whitenet ${NETWIDE}

    iptables -A INPUT -p icmp -j ACCEPT
    iptables -A INPUT -i lo -j ACCEPT
    iptables -A INPUT -i eth0 -j ACCEPT

    iptables -A INPUT -j REJECT --reject-with icmp-host-prohibited
    iptables -A FORWARD -j REJECT --reject-with icmp-host-prohibited

    iptables -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
    iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
    iptables -A INPUT -m state --state NEW -p tcp --dport 443 -j ACCEPT

    iptables -A INPUT ! -s ${NETLOC} -m set --match-set whitelist src -j ACCEPT
    iptables -A INPUT ! -s ${NETLOC} -m set --match-set blacklist src -j DROP

    for i in ${TCPPRIV}
    do
        iptables -A INPUT -p tcp --dport ${i} ! -s ${NETLOC} -j DROP
    done
    for i in ${UDPPRIV}
    do
        iptables -A INPUT -p udp --dport ${i} ! -s ${NETLOC} -j DROP
    done

    /usr/bin/buildblst.sh --update-net



Highly recommend the use of built-in functions for working with iptables, it will increase productivity and free up hardware resources. Applications for these purposes executable script properly. The action script is needed to solve complex non-standard applications that require running external programs and arbitrary filesystem operations.

Example local access policy shell script:


    #!/bin/bash
    #
    # This script will be executed *SENSOR-NG* for manage IP address access.
    # Entered rule into the local access policy.

    # 0 - program path & name
    # 1 - IP address
    # 2 - command
    # 3 - type
    # 4 - match parse rule

    function chk_ip() {
        local ip=$1
        if [[ `echo $ip | egrep -E '[1-9]{1}[0-9]{1,2}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'` == "" ]] ;
        then
            return 1
        fi
        return 0
    }

    function del_ip() {
        local ip=$1

        iptables -t filter -D INPUT -s ${ip} -j DROP >/dev/null 2>&1
        while [ $? -eq 0 ]
        do
        iptables -t filter -D INPUT -s ${ip} -j DROP >/dev/null 2>&1
        done
    }

    function add_ip() {
        local ip=$1
        iptables -t filter -I INPUT -s ${ip} -j DROP >/dev/null 2>&1
    }

    if [[ ${1} == "" || ${2} == "" ]] ;
    then
        exit 1
    fi

    chk_ip ${1}
    RETVAL=$?
    if [ ${RETVAL} -eq 1 ] ;
    then
            exit 1
    fi

    case ${2} in
        9)
            del_ip ${1}
            add_ip ${1}
            notify-send "Adding ip: ${1}" "match rule: ${4} type: ${3}" -u critical
            exit 0
        ;;
        10)
            del_ip ${1}
            notify-send "Delete ip: ${1}" "match rule: ${4} type: ${3}"
            exit 0
        ;;
        *)
        ;;
    esac

    exit 0



Highly recommend the use of built-in functions for working with iptables, it will increase productivity and free up hardware resources. Applications for these purposes executable script properly. The action script is needed to solve complex non-standard applications that require running external programs and arbitrary filesystem operations.

Eexample rules for iptables to good work built in LOG parse engine:


    # command line:
    iptables -N LOGDROP

    iptables -A INPUT ! -s 192.168.0.0/24 -j LOGDROP
    # or include destination path:
    iptables -A INPUT -p tcp -d 1.2.3.4 --dport 80  -j LOGDROP
    iptables -A INPUT -p tcp -d 1.2.3.4 --dport 443 -j LOGDROP
    

    # ipsetd-ng settings:
    [`driver`]
        iptenable = yes     ; enable iptables add or delete direct to kernel
        iptchain = LOGDROP  ; chain of iptables
        ipttable = filter   ; table of iptables
        iptrule =  DROP     ; target of iptables
        iptmethod = append  ; method for adding to table: insert | append

    # sensor-ng settings:
    [`server`]
        iptenable = yes     ; enable iptables add or delete direct to kernel
        iptchain = LOGDROP  ; chain of iptables
        ipttable = filter   ; table of iptables
        iptrule =  DROP     ; target of iptables
        iptmethod = append  ; method for adding to table: insert | append
    

Eexample ipset to iptables filter rule:

To manage firewall rules based on iptables and ipset, recommend use the supplied utility from package IPSET-NG: buildblst.

    ipset create blacklist hash:ip
    ipset create whitelist hash:net

    iptables -A INPUT -m set --match-set whitelist src -j ACCEPT
    iptables -A INPUT -m set --match-set blacklist src -j DROP

    ipset -A whitelist 192.168.0.0/24
    ipset -A whitelist 10.1.1.0/24
    ipset -A whitelist 1.2.3.24/28

    ipset -A blacklist 5.6.7.7
    ipset -A blacklist 5.6.7.8
    ipset -A blacklist 5.6.7.9
    ...


Make sure to run this text in your firewall script. If you don't, the ipset blacklist, whitelist and the iptables rule to banned blacklisted IP addresses will be missing.

Check status for dropped packets and banned host:


    # iptables -L -v -n
    Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
     pkts bytes target     prot opt in     out     source               destination
      98M   35G ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0           match-set whitelist src
      444 82582 DROP       all  --  *      *       0.0.0.0/0            0.0.0.0/0           match-set blacklist src


Combining multiple ipset tables in one source:

  • 1. to create global list set, it must apply to the client, it will encompass all the black lists with different types of data:

    ipset -N blacklist list:set
    ipset -N whitelist list:set


  • 2. to create personalized tables for different types of addresses and sources:

    ipset -N blackip  hash:ip
    ipset -N blacklog hash:ip
    ipset -N blacknet hash:net
    ipset -N blackproxy hash:ip

    ipset -N whiteip  hash:ip
    ipset -N whitenet hash:net


  • 3. join tables in a single source:

    ipset -A blacklist blackip
    ipset -A blacklist blacklog
    ipset -A blacklist blacknet
    ipset -A blacklist blackproxy

    ipset -A whitelist whiteip
    ipset -A whitelist whitenet


  • 4. insert to personalized ipset tables IP addresses or net addresses depending on the type of..
  • 5. enjoy!
  • 6.1 (optional) later, you can add a black sheet with a lifetime of IP addresses for Apache or Nginx:

    # blocking one hours: timeout = minute * 60
    ipset -N nginx_blacklist hash:ip family inet hashsize 1024 maxelem 65536 timeout 3600
    # blocking one day: timeout = (hours * 60) * 60
    ipset -N nginx_blacklist_hack hash:ip family inet hashsize 1024 maxelem 65536 timeout 86400

    ipset -A blacklist nginx_blacklist
    ipset -A blacklist nginx_blacklist_hack


  • 6.2 (optional) Block setting the IP address for the HTTPD server Apache:

    <Location ~ "/(wp-|admin|eshop|configuration|webadmin|webmaster|fckeditor|phpmyadmin|phpadmin)">
        <IfModule mod_ipsetng.c>
            # add IP address to black list,
            # if called this resource
            IPSNGEngine on
            IPSNGLog on
            IPSNGCommand add
            IPSNGList add nginx_blacklist 403
            IPSNGDnsbl off
        </IfModule>
    </Location>


  • 6.3 (optional) Block setting the IP address for the HTTPD server Nginx:

    location ~* /(w00tw00t.at.blackhats.romanian.anti-sec|_utl_t=fb|p=73|HNAP1|wp-|admin|yandexml|eshop|configuration) {
        ipset_default_access deny;
        ipset_list add nginx_blacklist 3600;
    }        


StartUp ipset script:

To manage firewall rules based on iptables and ipset, recommend use the supplied utility from package IPSET-NG: buildblst.

    #!/bin/bash
    #
    # chkconfig: 345 80 20
    # description: run ipset rules as start and stop
    #

    ipset -h >/dev/null 2>&1
    RETVAL=$?
    if [ $RETVAL -eq 127 ] ;
    then
        echo "not found ipset executable in path"
        exit 1
    fi

    iptables -h >/dev/null 2>&1
    RETVAL=$?
    if [ $RETVAL -eq 127 ] ;
        echo "not found iptables executable in path"
        exit 1
    fi

    if [ -x /etc/rc.d/init.d/functions ] ;
    then
        ./etc/rc.d/init.d/functions
    fi
    
    ipsetstart () {

        ipset -q -N blacklist list:set
        ipset -q -N whitelist list:set

        ipset -q -N blackip  hash:ip
        ipset -q -N blacklog hash:ip
        ipset -q -N blacknet hash:net
        ipset -q -N blackproxy hash:ip

        ipset -q -N nginx_blacklist hash:ip family inet hashsize 1024 maxelem 65536 timeout 3600
        ipset -q -N nginx_blacklist_hack hash:ip family inet hashsize 1024 maxelem 65536 timeout 86400

        ipset -q -N whiteip  hash:ip
        ipset -q -N whitenet hash:net

        ipset -q -A blacklist blackip
        ipset -q -A blacklist blacklog
        ipset -q -A blacklist blacknet
        ipset -q -A blacklist blackproxy

        ipset -q -A blacklist nginx_blacklist
        ipset -q -A blacklist nginx_blacklist_hack

        ipset -q -A whitelist whiteip
        ipset -q -A whitelist whitenet

        ipset -q -A whiteip 127.0.0.1

        if [ -f /etc/ipset-ng/ipset-save.lst ] ;
        then
        cat /etc/ipset-ng/ipset-save.lst | ipset -R
        fi

        iptablesdel

        iptables -I INPUT -m set --match-set blacklist src -j DROP
        iptables -I INPUT -m set --match-set whitelist src -j ACCEPT
    }

    ipsetstop () {

        if [ ! -d /etc/ipset-ng ] ;
        then
        mkdir -p /etc/ipset-ng
        fi

        ipset -S >/etc/ipset-ng/ipset-save.lst

        iptablesdel

        ipset -q -F blacklist
        ipset -q -F whitelist
        ipset -q -X blacklist
        ipset -q -X whitelist

        ipset -q -F blackip
        ipset -q -F blacklog
        ipset -q -F blacknet
        ipset -q -F blackproxy
        ipset -q -X blackip
        ipset -q -X blacklog
        ipset -q -X blacknet
        ipset -q -X blackproxy

        ipset -q -F nginx_blacklist
        ipset -q -F nginx_blacklist_hack
        ipset -q -X nginx_blacklist
        ipset -q -X nginx_blacklist_hack

        ipset -q -F whiteip
        ipset -q -F whitenet
        ipset -q -X whiteip
        ipset -q -X whitenet

    }

    iptablesdel () {

        iptables -D INPUT -m set --match-set blacklist src -j DROP
        while [ $? -eq 0 ]
        do
        iptables -D INPUT -m set --match-set blacklist src -j DROP
        done

        iptables -D INPUT -m set --match-set whitelist src -j ACCEPT
        while [ $? -eq 0 ]
        do
        iptables -D INPUT -m set --match-set whitelist src -j ACCEPT
        done

    }
    case $1 in
        start)
            ipsetstart
        ;;
        stop)
            ipsetstop
        ;;
        restart|reload)
            ipsetstop
            ipsetstart
        ;;
        *)
        echo "Usage: $0 {start|stop|restart}"
        ;;
    esac

    RETVAL=$?
    exit $RETVAL


This script implemented in IPSET-NG package, and automatically loads when you start or stop IPSETD-NG, the script is in /etc/rc.d/init.d/ipsetd-ng.init

  Meta Tags: Parse LOG files engine