SENSORD-NG LOG parse engine, for fast blocking IP address in parse log files

SENSORD-NG is part of the package IPSET-NG, and is designed to parse and analyze log files in order to identify areas for banned IP addresses.

The logic of the analysis is based on regular expressions syntax supports PCRE style. Interaction and syntax filtering rules are described in detail in the manual for setting up parsing log files. Also provides examples of the most common formats of log files, to compile your own rules, we recommend using a OnLine constructor regular expressions.

When determining in the configuration file sections [userscript] -> base = /path/to/JavaScript/dir and [logparse] -> useusrscr = yes, IP addresses processing will be carried out with the help of JavaScript you specified in the parameter [logparse] -> jscript = name_of_execute_script.js. It is obvious that you have specified the script will be searched for in the directory you defined in the parameter [userscript] -> base.

More about Javascript framework API, there are also examples and libraries to work with IP addresses in the directory <ipset-ng-src-dir>/userscript, default, on installation process create user JavaScript directory in /etc/ipset-ng/userscript.

All IP address parse matches are sent to the main server IPSETD-NG and entered into the local access policy, require enable iptables operation in configuration, or executed by the script. Name and path of the script is specified by the configuration file sections [server] -> exec = /path/to/script.sh and parameters [server] -> execq = no sets to view output the result of the script for debugging.
SENSORD-NG executes access policy script has the following parameters:

  • 0 - executes script path and name
  • 1 - blocking IP address
  • 2 - command of ipset aware, is numeric value: test | add | del
  • 3 - type of ipset aware, is numeric value: white | black | add | del
  • 4 - numeric value match parse rule

If the configuration is defined iptables opportunity to communicate directly with the kernel, then the script event reasonably be canceled.
Although it is possible, it can perform a different role, such notification or update statistics or different configurations.
To activate the iptables in configuration file, use the following parameters: iptenable, iptchain, ipttable, iptrule, iptmethod.

  • Note: that for these iptables parameters SENSORD-NG placed [server] section, and for IPSETD-NG [driver] section.

Example configuration file can be found in <ipset-ng-src-dir>/misc/config/sensord-ng.conf, also available a detailed description of the configuration parameters.

To reduce the command line parameters using environment variables, they are set to ksh shell - export command team, they are set to csh shell - setenv command team.
Priority configuration source selector next: command line -> environment variables -> configuration file.

To construct this LOG sensor from the package IPSET-NG on another computer, you must download the file created after the server build,
unpack it and run the command: make clean ; make milterd
Read more about the build process, keys and options can be found in the section: tuning and compile options

Command line options:
-d, --daemon exec in daemon mode
-m, --monitor exec in monitor mode
-r, --remote=<arg> remote ipsetd-ng server IP address
-p, --port=<arg> remote ipsetd-ng port UDP/TCP-SSL port
-f, --family=<arg> select default family ipv protocol: ipv4 or ipv6
-a, --password=<arg> crypt chiper - access to server AES crypt algoritm
-l, --loglevel=<arg> log level disabled, critical, medium, full, debug, debugssl
-c, --userscript=<arg> exec for parse user JavaScript file
-s, --ssl SSL connection enable
-z, --sslcert=<arg> SSL certificate + ca + key
-i, --sslsni=<arg> SSL SNI server hostname
-u, --sslciph=<arg> SSL cipher string
-x, --memleak print memory leak diagnostic on exit
-e, --env display environment help
-h, --help display this help
Management daemon command line options:
-k, --kill stop running program copy (terminate)
Examples command line settings:

        # Example console monitor mode, pass secondary parse to JavaScript scenario parselog.js & no-crypt connection:
        /usr/bin/sensord-ng -r 1.2.3.4 -p 1919 -m -l full -c parselog.js

        # Example daemon mode, crypt connection:
        /usr/bin/sensord-ng -r 1.2.3.4 -p 1919 -d -l full -a "my access word!"

        # Example daemon mode, ssl connection:
        /usr/bin/sensord-ng -r 1.2.3.4 -p 5000 -d -l debugssl \
            -s -o 45 -z /etc/ipset-ng/ssl/full.client.pem -u "ALL:!ADH:!LOW:!EXP:!MD5:@STRENGTH"

        # Example unification SSL certificates and keys:
        cd /etc/ipset-ng/ssl
        cat ./cert.client.pem ./cert.ca.pem ./key.client.pem > ./full.client.pem


* Example ksh:
        export NG_SENSOR_port="1919"
        export NG_SENSOR_remote="127.0.0.1"
* Example csh:
        setenv NG_SENSOR_port="1919"
        setenv NG_SENSOR_remote="127.0.0.1"
NG_SENSOR_remote remote server host
NG_SENSOR_port remote UDP/TCP-SSL port
NG_SENSOR_family select default family ipv protocol: ipv4 or ipv6
NG_SENSOR_loglevel log level disabled, critical, medium, full, debug, debugssl
NG_SENSOR_userscript exec for parse user JavaScript file
NG_SENSOR_password crypt chiper - access to server AES crypt algoritm
NG_SENSOR_ssl SSL connection enable
NG_SENSOR_sslsni SSL SNI server hostname
NG_SENSOR_sslcert SSL certificate + ca + key
NG_SENSOR_sslciph SSL cipher string
NG_SENSOR_ssltm SSL connection timeout
Monitor command key functions:
[0,1,2,3,4,5] - set log level:
  • 0 disabled - disable log
  • 1 critical - warning only log
  • 2 medium - info level log
  • 3 full - full log
  • 4 debug - full log, include memory access
  • 5 debugssl - full log, include memory access + ssl
[6,7,8] - manual run script event:
  • 6 - run script onstart event
  • 7 - run script onreload event
  • 8 - run script onstop event
[e] - enable/disable console output
[s] - show configuration
[w] - show WhoIs request from processed IP address, three levels of detailed information, is governed by the number of hits
[g] - show GeoIP information from processed IP address
[d] - iptables chain print
  • only if the SENSOR-NG is compiled with the libiptc library
    and iptables enable is included in the configuration file
[x] - save iptables chain dump to file
  • only if the SENSOR-NG is compiled with the libiptc library
    and iptables enable is included in the configuration file
  • dump file stored in: /tmp/ipset-ng-dump-iptables-flush-DATA-TIME.sh
[f] - clear iptables chain
  • only if the SENSOR-NG is compiled with the libiptc library
    and iptables enable is included in the configuration file
  • The data stored in the file: /tmp/ipset-ng-dump-iptables-flush-DATA-TIME.sh,
    if you want to restore them - run it
[l] - show session statistic
  • only for data transfer mode SSL
[p] - show user certificates
  • only for data transfer mode SSL
[i] - show connect information
  • only for data transfer mode SSL
[a] - send crypt/no-crypt AES packet
  • crypt mode required password, is the same password on the ipsetd-ng server
[v] - check newer version available
[q,ESC] - stop milterd-ng and exit
[?,h] - help screen

Example local access policy shell script:


    #!/bin/bash
    #
    # This script will be executed *SENSOR-NG* for manage IP address access.
    # Entered rule into the local access policy.

    # 0 - program path & name
    # 1 - IP address
    # 2 - command
    # 3 - type
    # 4 - match parse rule

    function chk_ip() {
        local ip=$1
        if [[ `echo $ip | egrep -E '[1-9]{1}[0-9]{1,2}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}'` == "" ]] ;
        then
            return 1
        fi
        return 0
    }

    function del_ip() {
        local ip=$1

        iptables -t filter -D INPUT -s ${ip} -j DROP >/dev/null 2>&1
        while [ $? -eq 0 ]
        do
        iptables -t filter -D INPUT -s ${ip} -j DROP >/dev/null 2>&1
        done
    }

    function add_ip() {
        local ip=$1
        iptables -t filter -I INPUT -s ${ip} -j DROP >/dev/null 2>&1
    }

    if [[ ${1} == "" || ${2} == "" ]] ;
    then
        exit 1
    fi

    chk_ip ${1}
    RETVAL=$?
    if [ ${RETVAL} -eq 1 ] ;
    then
            exit 1
    fi

    case ${2} in
        9)
            del_ip ${1}
            add_ip ${1}
            notify-send "Adding ip: ${1}" "match rule: ${4} type: ${3}" -u critical
            exit 0
        ;;
        10)
            del_ip ${1}
            notify-send "Delete ip: ${1}" "match rule: ${4} type: ${3}"
            exit 0
        ;;
        *)
        ;;
    esac

    exit 0



Highly recommend the use of built-in functions for working with iptables, it will increase productivity and free up hardware resources. Applications for these purposes executable script properly. The action script is needed to solve complex non-standard applications that require running external programs and arbitrary filesystem operations.

Eexample rules for iptables to good work built in LOG parse engine:


    # command line:
    iptables -N LOGDROP
    iptables -A INPUT ! -s 192.168.0.0/24 -j LOGDROP
    

    # ipsetd-ng settings:
    [`driver`]
        iptenable = yes     ; enable iptables add or delete direct to kernel
        iptchain = LOGDROP  ; chain of iptables
        ipttable = filter   ; table of iptables
        iptrule =  DROP     ; target of iptables
        iptmethod = append  ; method for adding to table: insert | append

    # sensor-ng settings:
    [`server`]
        iptenable = yes     ; enable iptables add or delete direct to kernel
        iptchain = LOGDROP  ; chain of iptables
        ipttable = filter   ; table of iptables
        iptrule =  DROP     ; target of iptables
        iptmethod = append  ; method for adding to table: insert | append
    
Highly recommend the use of built-in functions for working with iptables, it will increase productivity and free up hardware resources. Applications for these purposes executable script properly. The action script is needed to solve complex non-standard applications that require running external programs and arbitrary filesystem operations.

  Meta Tags: SENSORD-NG LOG parse engine