Setting up A Network Monitoring and Inventory System

From Alpine Linux
Revision as of 13:52, 17 October 2010 by Jbilyk (talk | contribs) (→‎Steps: Started on Nagiosql steps)


This material is work-in-progress ...

Do not follow instructions here until this notice is removed.
(Last edited by Jbilyk on 17 Oct 2010.)

This guide will set up (list subject to change):

This guide will assume:

  • You have a knowledge of your network setup (at least know which subnets exist)
  • Your network allows ICMP ping packets to every device that you want discovered and automatically added to the system
  • Your firewalls (whether on your networking devices or end-user devices) allow traffic back to the server that is being set up, and if it is a Windows host, allows WMI queries in
  • If you are auditing a Windows domain, you have access to an account that has local administrator privileges on every computer (on Win XP a user with Power User privileges may be enough). If running the audits on Win7 Pro machines, double-check local firewall that it allows RPC traffic to/from OpenAudit computer (will allow audits to happen properly when triggered via the OpenAudit computer).

Goal

Basic concept is to combine the above apps/utilities to create an easy to set up and low upkeep monitoring system that also tracks inventory thanks to OpenAudit. Redundant entries in configs will be kept to a minimum (hopefully). New devices should be auto-added to monitoring systems, along with associated services (for example, a Windows server running MS Exchange might have basic services such as CPU load, HD free space, running services and Exchange queues monitored, whereas a router running on Alpine Linux would might need the routing table, throughput, and opennhrp monitored). Also, software licensing should be monitored so that Windows boxes do not exceed purchased licenses.

Steps

Install lighttpd, PHP and MySql 

apk add lighttpd mysql mysql-client php-mysql php-mysqli php php-xml php-json php-gettext php-gd curl php-curl php-mcrypt

Edit lighttpd.conf 

nano +46 /etc/lighttpd/lighttpd.conf

Uncomment lines: 

uncoment    include "mod_fastcgi.conf", save and exit

Starting http service and adding to boot 

/etc/init.d/lighttpd start && rc-update add lighttpd default

Configure MySQL 

/usr/bin/mysql_install_db --user=mysql
/etc/init.d/mysql start && rc-update add mysql default
/usr/bin/mysqladmin -u root password 'password'

Install Nagios, nagios-plugins and Nagiosql and other needed packages 

apk add nagios nagios-web nagios-plugins nagiosql php-mysqli php-mysql

Create soft-link for nagiosql virtual host' 

ln -s /usr/share/webapps/nagiosql /var/www/localhost/htdocs/nagiosql

Change permissions for nagiosql 

chown lighttpd:lighttpd /usr/share/webapps/nagiosql/configs

Browse to http://localhost/nagiosql and follow the setup instructions. Create folder /usr/share/webapps/openaudit and link to virtual host folder 

mkdir /usr/share/webapps/openaudit
chown lighttpd:lighttpd /usr/share/webapps/openaudit
ln -s /usr/share/webapps/openaudit /var/www/localhost/htdocs/openaudit

Download openaudit from http://downloads.sourceforge.net/open-audit/openauditrelease-09.12.23-SVN1233.zip and extract to /usr/share/webapps/openaudit. On a Windows server, create scheduled tasks to run ping-sweep-main.bat, lookup-main.bat and insert-hosts-main.bat on a regular basis. Since the insert-hosts-main.bat file runs RPC calls against other Windows servers, at the moment this section needs to run on a Windows server... Back on the Alpine monitoring server, start preparation of nagiosql for the Nagios paths used in Alpine Linux: 

echo "update db_nagiosql_v3.tbl_domain set binaryfile = '/usr/sbin/nagios';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set importdir = '/etc/nagios/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set basedir = '/etc/nagios/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set hostconfig = '/etc/nagios/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set serviceconfig = '/etc/nagios/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set backupdir = '/etc/nagiosql/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set hostbackup = '/etc/nagiosql/';" | mysql --user=root -p
echo "update db_nagiosql_v3.tbl_domain set servicebackup = '/etc/nagiosql/';" | mysql --user=root -p

Progress Log

Set up Alpine server with Lighttpd 1.4.28-r0, PHP 5.3.3-r2 and MySQL 5.1.50-r0 with OpenAudit 09.12.23. Windows box can be audited with results returned to Alpine.
Set up test cacti system and made sure the basics work with the new aport.
SNMPTT receives snmp traps properly and sends them on to Nagios but they aren't parsed in any way yet.
Installed Nagiosql, need to investigate potential usage + OpenAudit integration.
Got nmap results into OpenAudit. Needed a change in admin_nmap_input.php line 48 ("Interesting ports on" -> "Nmap scan report for"), for which a bug report was filed upstream. Got pingable/resolvable hosts automatically added into OpenAudit:

  • When auto-adding using scripts below, Windows versions 2000 (Pro and Server), XP Pro, 2003 (inc R2)(Std/Ent)(x86/x64) were detected. Windows 7 Pro only detected when the local firewall was set to allow RPC traffic through
  • Some printers (HP LJ3300, LJ8100) were detected whereas other models weren't
  • No networking devices (switches/routers) were added

Operational Concept

Nmap -> OpenAudit

  1. Scheduled nmap ping sweep of subnets (one at a time)
  2. Check if hosts have been audited recently
  3. If not, trigger an nmap OS detection
  4. If Windows, trigger a remote audit
  5. If other device type, do ___ to audit

OpenAudit -> Nagios (or Cacti, or Smokeping)

  1. Schedule a query of Windows Servers from OpenAudit + installed software
  2. Check that nagiosql has each server in it
  3. If it doesn't, add host monitoring + base services
  4. For each MS software (such as Exchange, SQL, DPM), add services

For bonus points:

  1. Query LLDP to find out where device is and set up dependencies automatically
  2. SNMPTT integration to Nagios (setup based on device info in OpenAudit)

Random scripts

Since part of implementation runs on Windows, a set of scripts that can run on both Windows and Alpine Linux are being put together.

host-detect.sh: a one-line linux command to replace all of the ping-sweep* and lookup* batch files. Ping-sweep*.bat: does an nmap ping sweep of each subnet in subnet.txt and writes all hosts that are up to a file lookup*.bat: does a A record DNS lookup for each host that's pingable and writes results to a file insert-hosts*: for each resolvable host, downloads the vbscript to insert the host information into openaudit (uses WMI to get hardware, OS and software info)

host-detect.sh (replaces all batch files below) 

#!/bin/sh
for IPADDRESS in `nmap -sP 10.14.8-39.0-255 | grep "Nmap scan report for" | awk -F ' ' '{print $NF}' | sed s/\(// | sed s/\)//`;do nslookup $IPADDRESS | grep "name =" | awk -F '=' '{print $2}';done

ping-sweep-main.bat 

cd C:\Program Files\WAMP\www\openaudit\scripts\scans
for /F %%i in (subnets.txt) do ping-sweep-backend.bat %%i

ping-sweep-backend.bat 

@echo off
cd C:\Program Files\WAMP\www\openaudit\scripts\scans
del %1.ip.txt
nmap -sP %1 | "C:\program files\utilities\bin\grep" "Nmap scan report for" | "C:\Program Files\Utilities\bin\awk" -F " " "{print $NF}" | "C:\Program Files\Utilities\bin\awk" -F "(" "{print $NF}" | "C:\Program Files\Utilities\bin\awk" -F ")" "{print $1}" >> %1.ip.txt

lookup-main.bat 

cd C:\Program Files\WAMP\www\openaudit\scripts\scans
for /F %%i in (subnets.txt) do lookup-subnet.bat %%i

lookup-subnet.bat 

@echo off
cd C:\Program Files\WAMP\www\openaudit\scripts\scans
del %1.host.txt
for /F %%i in (%1.ip.txt) do @lookup-backend.bat %%i >> %1.host.txt

lookup-backend.bat 

@nslookup %1 | "C:\program files\utilities\bin\grep" "Name:  " | "C:\program files\utilities\bin\awk" -F " " "{print $2}"

insert-hosts-main.bat 

@echo off
cd C:\Program Files\WAMP\www\openaudit\scripts\scans
for /F %%i in (subnets.txt) do insert-hosts-subnet.bat %%i

insert-hosts-subnet.bat 

@echo off
cd C:\Program Files\WAMP\www\openaudit\scripts\scans
for /F %%i in (%1.host.txt) do @insert-hosts-parse.bat %%i

insert-hosts-parse.bat 

@echo off
cd C:\Program Files\WAMP\www\openaudit\scripts\scans
echo %1 | "C:\Program Files\Utilities\bin\awk.exe" -F "." "{print $1}" > temp-out-insert.txt
for /F %%A in (temp-out-insert.txt) do @insert-hosts-backend.bat %%A
del temp-out-insert.txt

insert-hosts-backend.bat 

cd C:\Program Files\WAMP\www\openaudit\scripts\scans
"C:\Program Files\Utilities\bin\wget.exe" "http://camgmt01/openaudit/launch_local_audit.php?domain=can.wtbts.net&application=cmd&ext=vbs&hostname=%1&" -O %1.vbs
cscript %1.vbs
del %1.vbs

subnet.txt (snippet) 

10.14.8.0-255
10.14.9.0-255

To grab all Windows servers using MySQL: 

select system_uuid,system_name,net_ip_address,system_os_name from openaudit.system where system_os_name like '%Microsoft%Server%';
Retrieved from "https://wiki.alpinelinux.org/w/index.php?title=Setting_up_A_Network_Monitoring_and_Inventory_System&oldid=4515"