Veritas VCS and Nagios

Here is a quick insight into getting Nagios with Apache working on a Linux RHEL based system. I set this up on CentOS 4.4 and used YUM to install Nagios. I also created the Volumes and File systems in VxVM but I will leave that part out and show just how simple it is to get Nagios clustered in VCS. This assumes you have two VCS nodes setup called 'nagiosa' and 'nagiosb'. Once you finish this and add some storage and what else you need for your setup enable the resources and bring them online!

# Make the config writable
haconf -makerw

# Create New Nagios Service Group
hagrp -add NagiosSG
hagrp -modify NagiosSG SystemList nagiosa 0 nagiosb 1
hagrp -modify NagiosSG AutoStartList nagiosa
hagrp -modify NagiosSG Parallel 0

# Create Nagios Application resource
hares -add nagsvc Application NagiosSG
hares -modify nagsvc Critical 0
hares -modify nagsvc User root
hares -modify nagsvc StartProgram /sbin/service nagios start
hares -modify nagsvc StopProgram /sbin/service nagios stop
hares -modify nagsvc CleanProgram
hares -modify nagsvc PidFiles /var/run/nagios.pid

# Create Apache web server resource
hares -add apachesvc Apache NagiosSG
hares -modify apachesvc Critical 0
hares -modify apachesvc ServerRoot /etc/httpd
hares -modify apachesvc DetailMonitor 0
hares -modify apachesvc Port 80
hares -modify apachesvc Address 0.0.0.0
hares -modify apachesvc ConfigFile /etc/httpd/conf/httpd.conf

# Link Apache to Nagios so Apache must come up first
hares -link nagsvc apachesvc

# Save and close the config
haconf -dump -makero

FTP new files to remote server

It has been a while since I have added a good script so here is one that I put together for a friend who needed to send files in a directory that are updated daily to remote client. Have a look at the script and feel free to ask if you don't understand it. This is pretty straight forward.

#!/bin/bash

# Script variables
shopt -s -o xtrace
shopt -s -o vi
shopt -s -o nounset
SPATH="/path/to/script/home"
RUN_LOG="${SPATH}/ftp_log"
SENT_FILE="${SPATH}/send_files_log"

# update date on log files
touch ${RUN_LOG} || {echo "cannot create ftp_log"; exit 1;}
touch ${SENT_FILE} || {echo "cannot create send_files_log"; exit 1;}

# Set home DIR
cd ${SPATH}

# static variables
ALL_FILES=$(ls -l|awk '{if (NR == 1 || $1 ~ /^d/) {next;} print $9}')
DATE="$(date)"

# User defined variables
HOST='10.0.0.1'
USER='user'
PASSWD='password'
TMP_LIST="/tmp/ftp_xfer_log"
REMOTE_DIR="/remote/server/path"

# if tmp list exsists clean it up or create it
if [ -f ${TMP_LIST} ]; then
>${TMP_LIST}
else
touch ${TMP_LIST}
fi


# Create list to send
for file in ${ALL_FILES}
do
grep $file ${SENT_FILE} >/dev/null 2>&1
(($? == 1)) && echo $file >> ${TMP_LIST}
done

# Reformat tmp_list
XFILES="$(echo ${TMP_LIST} | xargs)"

# FTP Files to server
ftp -v -n ${HOST} <quote USER ${USER}
quote PASS ${PASSWD}
prompt
cd ${REMOTE_DIR}
bin
put ${XFILES}
quit
END_SCRIPT

# Place newly sent files into sent list
echo ${TMP_LIST} >> ${SENT_FILE}

# Update Log
echo "" >> ${RUN_LOG}
echo ${DATE} >> ${RUN_LOG}
echo "following files sent successfully" >> ${RUN_LOG}
echo ${TMP_LIST} >> ${RUN_LOG}

# End Script
exit 0

Solaris 10 RBAC for apache and sendmail

This document describes the setup of Solaris 10 roles for Web and Mail Administrators. It is assumed that Apache 2.0 and sendmail are already setup and working on the target server. Apache is also assumed to be running as user ‘webservd’. This configuration will allow Administrators/Developers to switch to a new account (RBAC Role) and run commands necessary to handle Apache, and sendmail. Commands will be run by the user specified in the preceding command prompt, i.e. “[root] # ls” – means the ls command should be run as root. Substitute user ‘mbaxter’ with the users needed for this document.

Step 1 Stop Apache and Coldfusion.

[root] # svcadm –v disable –s apache2
[root] # svcadm -v disable -s sendmail

Step 2 Create separate Administrative Role

[root] # roleadd –g webservd –d /home/webadm –m webadm
[root] # passwd webadm
New Password:
Re-enter new Password:
Passwd: password successfully changed for webadm
[root] # usermod –R webadm mbaxter

Step 3 Create authorizations for webadm.
Add the following lines to /etc/security/auth_attr file in vi

[root] # vi /etc/security/auth_attr
sunw.*:::Custom Authorizations::
sunw.grant:::Grant Custom Authorizations::
sunw.smf.manage.http/apache2:::Manage the Apache2 Service::
sunw.smf.modify.application.http/apache2:::Modify the Apache2 Application Properties::

Assign new authorizations to root user by editing /etc/user_attr
[root] # vi /etc/user_attr
root::::auths=solaris.*,solaris.grant,sunw.*,sunw.grant;profiles=Web Console Management,All;lock_after_retries=no

Step 4 Grant SMF-Specific authorizations to webadm

[root] # rolemod –A sunw.smf.manage.http/apache2,sunw.smf.modify.application.http/apache2 webadm

Step 5 Configure Apache2 with reduced privileges and required authorizations.

[root] # svccfg –s apache2

Install new properties:
svc:/network/http:apache2> setprop httpd/value_authorization = astring:
sunw.smf.modify.application.http/apache2
svc:/network/http:apache2> setprop general/action_authorization = astring:
sunw.smf.manage.http/apache2
svc:/network/http:apache2> setprop general/value_authorization =
astring: sunw.smf.manage.http/apache2

Configure reduced privileges:
svc:/network/http:apache2> setprop start/user = astring: webservd
svc:/network/http:apache2> setprop start/group = astring: webservd
svc:/network/http:apache2> setprop start/privileges = astring: basic,!proc_session,!proc_info,!file_link_any,net_privaddr
svc:/network/http:apache2> setprop start/limit_privileges = astring: :default
svc:/network/http:apache2> setprop start/use_profile = boolean: false
svc:/network/http:apache2> setprop start/supp_groups = astring: :default
svc:/network/http:apache2> setprop start/working_directory = astring: :default
svc:/network/http:apache2> setprop start/project = astring: :default
svc:/network/http:apache2> setprop start/resource_pool = astring: :default
svc:/network/http:apache2> end
[root] # svcadm –v refresh apache2
Action refresh set for svc:/network/http:apache2.

Step 6 Change ownership of log files.

[root] # cd /var/apache2/logs
[root] # chown webservd:webservd access_log error_log

Step 7 Configure Pidfile and Lockfile location.

[root] # mkdir –p /var/apache2/run
[root] # chown webservd:webservd /var/apache2/run
[root] # vi /etc/apache2/httpd.conf
Make these changes:
LockFile /var/apache2/logs/accept.lock
PidFile /var/apache2/run/httpd.pid

Step 8 Create Sendmail Profile by editing /etc/security/prof_attr with vi

[root] # vi /etc/security/prof_attr
Sendmail Management:::Sendmail Management Profile:

Step 9 Create Sendmail executable attributes by editing /etc/security/exec_attr with vi

[root] # vi /etc/security/exec_attr
Sendmail Management:solaris:cmd:::/usr/sbin/sendmail:uid=0

Sendmail Management:solaris:cmd:::/usr/sbin/dig:uid=0
Sendmail Management:solaris:cmd:::/usr/bin/mailq:uid=0

Step 10 Add Sendmail Profile to webadm

[root] # rolemod –P “Sendmail Management” webadm

Step 11 Change ownership of /etc/apache2 to webadm

[root] # chown –R webadm:webservd /etc/apache2

Step 12 Change ownership of /etc/mail to webadm

[root] # chown –R webadm:webservd /etc/mail

Step 13 Test configurations:

[root] # svcs –v apache2
disabled - 14:41:05 - svc:/network/http:apache2
[root] # su – webadm
[webadm] # svcadm –v enable –s apache2
[webadm] # svcs –v apache2
online - Sep_21 112 svc:/network/http:apache2
[webadm] # svcadm -v enable -s sendmail

Step 14 Add users to webadm role.

[root] # usermod –R webadm mbaxter

Script formatting

Had a few emails concerned with the spacing of the scripts supplied here and the layout. I do not go through and HTML code the scripts so that they are tabbed correctly or so that run on lines do to the blogs width limit are met. So if you have a problem reading it and cannot figure out the format send me an email and I will send you the script in its original form.

Application logging for Nagios output

I am sure there are dozens of ways to create application output logs for use with Nagios. I have a specific format that I use and a script that is called by apps to update the log. I will first post the log format then the script. don't forget to change the relative paths.

app_log:
BEGIN::Complete::0900030307::0244030507
PROCESS1::Complete::Good
PROCESS2::Complete::Good
PROCESS3::Complete::Good

monlog_update script:
http://malcolm.baxter.googlepages.com/monlog_update.pl

Solaris 10 RBAC

This document describes the setup of Solaris 10 roles for Coldfusion Developers. It is assumed that Apache 2.0 and Coldfusion are already setup and working on the target server. Coldfusion and Apache are also assumed to be running as user ‘webservd’. This configuration will allow developers to switch to a new account (RBAC Role) and run commands necessary to handle Apache, and Coldfusion. Commands will be run by the user specified in the preceding command prompt, i.e. “[root] # ls” – means the ls command should be run as root. Substitute user ‘mbaxter’ with the users needed for this document.

Step 1 Stop Apache and Coldfusion.

[root] # svcadm –v disable –s apache2
[root] # /etc/init.d/coldfusionmx7 stop

Step 2 Create separate Administrative Role

[root] # roleadd –g webservd –d /home/webadm –m webadm
[root] # passwd webadm
New Password:
Re-enter new Password:
Passwd: password successfully changed for webadm
[root] # usermod –R webadm mbaxter

Step 3 Create authorizations for webadm.
Add the following lines to /etc/security/auth_attr file in vi

[root] # vi /etc/security/auth_attr
sunw.*:::Custom Authorizations::
sunw.grant:::Grant Custom Authorizations::
sunw.smf.manage.http/apache2:::Manage the Apache2 Service::
sunw.smf.modify.application.http/apache2:::Modify the Apache2 Application Properties::

Assign new authorizations to root user by editing /etc/user_attr
[root] # vi /etc/user_attr
root::::auths=solaris.*,solaris.grant,sunw.*,sunw.grant;profiles=Web Console Management,All;lock_after_retries=no

Step 4 Grant SMF-Specific authorizations to webadm

[root] # rolemod –A sunw.smf.manage.http/apache2,sunw.smf.modify.application.http/apache2 webadm

Step 5 Configure Apache2 with reduced privileges and required authorizations.

[root] # svccfg –s apache2

Install new properties:
svc:/network/http:apache2> setprop httpd/value_authorization = astring:
sunw.smf.modify.application.http/apache2
svc:/network/http:apache2> setprop general/action_authorization = astring:
sunw.smf.manage.http/apache2
svc:/network/http:apache2> setprop general/value_authorization =
astring: sunw.smf.manage.http/apache2

Configure reduced privileges:
svc:/network/http:apache2> setprop start/user = astring: webservd
svc:/network/http:apache2> setprop start/group = astring: webservd
svc:/network/http:apache2> setprop start/privileges = astring: basic,!proc_session,!proc_info,!file_link_any,net_privaddr
svc:/network/http:apache2> setprop start/limit_privileges = astring: :default
svc:/network/http:apache2> setprop start/use_profile = boolean: false
svc:/network/http:apache2> setprop start/supp_groups = astring: :default
svc:/network/http:apache2> setprop start/working_directory = astring: :default
svc:/network/http:apache2> setprop start/project = astring: :default
svc:/network/http:apache2> setprop start/resource_pool = astring: :default
svc:/network/http:apache2> end
[root] # svcadm –v refresh apache2
Action refresh set for svc:/network/http:apache2.

Step 6 Change ownership of log files.

[root] # cd /var/apache2/logs
[root] # chown webservd:webservd access_log error_log

Step 7 Configure Pidfile and Lockfile location.

[root] # mkdir –p /var/apache2/run
[root] # chown webservd:webservd /var/apache2/run
[root] # vi /etc/apache2/httpd.conf
Make these changes:
LockFile /var/apache2/logs/accept.lock
PidFile /var/apache2/run/httpd.pid

Step 8 Create Coldfusion Profile by editing /etc/security/prof_attr with vi

[root] # vi /etc/security/prof_attr
Coldfusion Management:::Coldfusion Management Profile:

Step 9 Create Coldfusion executable attributes by editing /etc/security/exec_attr with vi

[root] # vi /etc/security/exec_attr
Coldfusion Management:solaris:cmd:::/etc/init.d/coldfusionmx7:uid=0

Step 10 Add Coldfusion Profile to webadm

[root] # rolemod –P “Coldfusion Management” webadm

Step 11 Change ownership of /etc/apache2 to webadm

[root] # chown –R webadm:webservd /etc/apache2

Step 12 Test configurations:

[root] # svcs –v apache2
disabled - 14:41:05 - svc:/network/http:apache2
[root] # su – webadm
[webadm] # svcadm –v enable –s apache2
[webadm] # svcs –v apache2
online - Sep_21 112 svc:/network/http:apache2
[webadm] # /etc/init.d/coldfusionmx7 start

Step 13 Add users to webadm role.

[root] # usermod –R webadm mbaxter

Simple Solutions

Sometimes a short Bash script is just what the doctor ordered. I Needed a quick script to check a files date on a remote machine and generate an error code if it was over 24 hours old. I will paste the code and a Nagios Expect plugin I wrote to call the script and interface with Nagios.

Remote test script in Bash:
http://malcolm.baxter.googlepages.com/nohup_test.sh

Nagios check script in Expect:
http://malcolm.baxter.googlepages.com/check_applogtime.exp>

Server list creation

While I was working at International Paper I became accustomed to a server tracking file called 'masterfile'. I used it many times in scripts to parse out data about individual servers or compile lists of servers to administer. Since then I have used a similar format here at Grumman to track my boxes and run scripts against them and have put together this Perl script to do it. The syntax can use Perl regular expressions to make it simple to create your lists. You will need Term::ANSIColor module from CPAN or delete the colored output and remove the module from the code. Also your masterfile should be '::' separated with the hostname as the first field. If you would like to see a dummy masterfile email me.

Use this script at your own will I assume no responsibility.

#!/usr/bin/perl

# master.pl - script to parse masterfile
#
# M.Baxter
# Version .90
######################################
######################################

# Complain about undeclared variables
use strict;
use warnings;

# Used modules
use Getopt::Long;
use File::Basename;
use Term::ANSIColor;

# Initialise global variables
our($opt_h, $opt_l, $opt_f, $opt_v, $opt_e);
our $script = basename($0);

# Declare subroutines
sub init;
sub usage;
sub list_servers;
sub full_master;

# Get the options
Getopt::Long::Configure('bundling');
GetOptions(
'v|verbose' => \$opt_v,
'h|help' => \$opt_h,
'f|full' => \$opt_f,
'l|list=s' => \$opt_l,
'e|exclude=s' => \$opt_e,
);

# Main
($opt_h) && usage;
($opt_f) || ($opt_l) || usage;
($opt_f) && full_master;
($opt_l) && list_servers;

# Subroutines
sub usage {
print `clear`, "\n\n";
print "Usage: \n\n";
print " -h : This Help Message\n";
print " -f : full masterfile in csv format created in scripts home dir.\n";
print " -l : list to be run i.e linux|sun|aix or any regular expression to search for.\n";
print " -e : server or regular expression to exclude from list.\n";
print " -v : Verbose output to full and list options.\n\n";
print color('bold green'), "example: $script -l linux -v\n\n\n";
print color('reset');
exit;
}

sub list_servers {
open (MASTER, "/usr/global/configs/masterfile") or die "cant open masterfile: $! \n";
while () {
if (/$opt_l/i) {
if($opt_e) {
if(! $opt_v) {
my @host = split(/:/);
print color('bold green'), "$host[0]\n" if(!m/$opt_e/i and !m/^#/);
} else {
print if(!m/$opt_e/i and !m/^#/);
}
} else {
if(! $opt_v) {
my @host = split(/:/);
print color('bold green'), "$host[0]\n" if(!m/^#/);
} else {
print if(!m/^#/);
}

}
}
}
close (MASTER);
print color('reset'), "\n";
exit;
}

sub full_master {
open (MASTERCSV, ">/usr/global/configs/masterfile.csv") or die "cant open masterfile.csv: $! \n";
open (MASTER, "/usr/global/configs/masterfile") or die "cant open masterfile: $! \n";
print MASTERCSV "Hostname,MachineType,Distribution,Application,Location,serialno,MACHINEMODEL,";
print MASTERCSV "CPUno,OSLEVEL,MEMORY,Startup_Order,Console\n";
while () {
s/:/,/g;
if (!m/^#/) {
print $_ if $opt_v;
print MASTERCSV;
}
}
close (MASTERCSV);
close (MASTER);
exit;
}

Nagios Application monitoring plugin

Recently I was tasked with bringing in the open source monitoring application Nagios into our environment to monitor our servers and hardware. After a small amount of configuration I had it setup on a HP DL360 running Centos 3.4. The Centos repository from DagWiers had all the necessary packages and I was able to simply use yum to install everything I needed. Over the course of a few posts I will show some scripts and apps I have put together to bring our in house applications into the monitoring process. I will not go over the Nagios installation though as I believe it has been covered well before. My first script for monitoring a COBOL application running on a Sun E6800 is called check_runitall. I wrote it in Perl and to use it you will have to install the Net::SCP::Expect module so fire up CPAN and install it before trying this out. Also this script asumes your Nagios plugins are installed to /usr/lib/nagios/plugins.

This code may be used as you like I assume no responsibility for it. Make sure to change the paths and so forth to suit your environment.

#!/usr/bin/perl -w
############################################################
############################################################
# check_runitall
# Nagios plugin to parse runitall
# runtime log
# Version .90
# M.Baxter
# USPS NCSC
############################################################
############################################################

# Complain about undeclared variables
# and set Perl requirements
require Net::SCP::Expect;
use strict;

# Used modules
use Getopt::Long;
use lib "/usr/lib/nagios/plugins";
use utils qw(%ERRORS $TIMEOUT &print_revision);
use vars qw($opt_f $PROGNAME);

# Initialise global variables
my $opt_H = undef;
my $opt_V = undef;
my $opt_t = undef;
my $opt_f = undef;
my $finc_code = undef;
my $message_log;
sub print_usage;
sub print_help;
$PROGNAME = 'check_runitall';

# Get the options
Getopt::Long::Configure('bundling');
GetOptions(
'V|version' => \$opt_V,
'h|help' => \$opt_H,
't|time' => \$opt_t,
'f|function=s' => \$opt_f,
);

if ($opt_V) {
print_revision( $PROGNAME, '$Revision: .90 $' );
exit $ERRORS{'UNKNOWN'};
}
($opt_H) &&amp;amp; print_help();
($opt_f) || ($opt_t) || print_usage();

# Secure copy log into local system
my $scpe = Net::SCP::Expect->new(
password => 'Password',
timeout => '25',
auto_yes => 1
);
$scpe->scp( 'User@Servername:/fast/monlogs/runitall_log',
'/home/nagios/runitall_log' );

# open log for reading
open( $message_log, "/home/nagios/runitall_log" )
or die "$PROGNAME cant open runitall_log $! \n";

# Search log for function and preform analysis
while (<$message_log>) {
if ($opt_t) {
if (/BEGIN/) {
chomp;
my @func = split(/::/);
if ( $func[1] eq "Complete" ) {
my @date = split(//, $func[3]);
printf("%-10.10s OK - Complete or waiting - Finished at %d%d\:%d%d %d%d/%d%d/%d%d\n", "Runitall", @date);
close($message_log);
exit $ERRORS{'OK'};
}
elsif ( $func[1] eq "Running" ) {
my @date = split(//, $func[2]);
printf("%-10.10s OK - Running - Start time %d%d\:%d%d %d%d/%d%d/%d%d\n", "Runitall", @date);
close($message_log);
exit $ERRORS{'OK'};
}
}
}
if ( ($opt_f) &&amp;amp; (/$opt_f/) ) {
chomp;
my @func = split(/::/);
if ( ( $func[1] eq "Complete" ) and ( $func[2] eq "Good" ) ) {
$finc_code = 0;
}
elsif ( ( $func[1] eq "Running" ) and ( $func[2] eq "Good" ) ) {
$finc_code = 1;
}
elsif ( ( $func[1] eq "Complete" ) and ( $func[2] eq "Failed" ) ) {
$finc_code = 2;
}
elsif ( ( $func[1] eq "Running" ) and ( $func[2] eq "Failed" ) ) {
$finc_code = 3;
}
}
}
close($message_log);
if ( $finc_code == 0 ) {
printf( "%-10.10s OK - Completed or waiting - Status Good\n", $opt_f);
exit $ERRORS{'OK'};
}
elsif ( $finc_code == 1 ) {
printf( "%-10.10s OK - Running - Status Good\n", $opt_f );
exit $ERRORS{'OK'};
}
elsif ( $finc_code == 2 ) {
printf( "%s Complete or Waiting In Failed State!\n", $opt_f );
exit $ERRORS{'CRITICAL'};
}
elsif ( $finc_code == 3 ) {
printf( "%s Running In Failed State!\n", $opt_f );
exit $ERRORS{'WARNING'};
}

# subroutines
sub print_help {
print "$PROGNAME\n";
print "Nagios Plugin to support log parsing from\n";
print "runitall application.\n\n";
print "Usage:\n\n";
print " -h : Help - Usage Menu\n";
print " -t : Parse Header for run times\n";
print " -f : Function name to be checked\n";
print " -V : Print Version\n\n";
print "example: $PROGNAME -f Function_name\n";
exit $ERRORS{'UNKNOWN'};
}

sub print_usage {
print "Usage: $PROGNAME [-h|-V|-t] [-f Function_name] \n";
exit $ERRORS{'UNKNOWN'};
}