Tag: Nagios

APT not automatically updating DigitalOcean

I’ve recently noticed a problem on 3 of my Digital Ocean Servers. The APT package lists are not automatically updating every day. I try to keep all servers upto date, and rely on Nagios to inform me when there’s packages needed to be updated and that’s the main reason I noticed something was broken.

The 3 servers in particular are newer builds to the rest of the system, and they dont have near as much installed as the others, so at first I didn’t pay too much attention when other servers were going into warning state on nagios indicating updates but these 3 weren’t. However I would still connect to these servers and run my normal command:-

apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove

A few times these servers did install updates and I just thought it must have been my timing, that the package lists hadn’t yet been updated by the cron.daily.

But after this happening a few times, I decided to not run the above and see how long these servers would take for nagios to throw an alert. It never did and that got me a little worried.

Over the last few days I’ve been diagnosing what’s wrong. I started out with making sure cron is working properly. Then kept an eye on the file timestamps

ls -ltrh /var/lib/apt/lists/

Eventually getting to /etc/cron.daily/apt and checking through what was was doing on the working servers compared to the broken ones. I turned on VERBOSE and got a bit of info when running /etc/cron.daily/apt but it seemed to exist quite quicky.

Comparing it to a working server the important bit seemed to be around

+ apt-config shell Debdelta APT::Periodic::Download-Upgradeable-Packages-Debdelta
+ eval
+ [ 1 -eq 0 ]
+ do_cache_backup 0</pre>
On the broken servers I was getting
<pre>+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]

Then it would exit. Further investigating was showing a few settings were being populated on the working servers but not on the broken ones.

So I compared the directory /etc/apt/apt.conf.d/ on both servers an found the following files missing from the broken servers

00aptitude

Aptitude::Get-Root-Command "sudo:/usr/bin/sudo";

10periodic

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";

15update-stamp

APT::Update::Post-Invoke-Success {"touch /var/lib/apt/periodic/update-success-stamp 2&gt;/dev/null || true";};

20archive

APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";

99update-notifier

DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo &gt; /var/lib/update-notifier/updates-available; fi "; };
(reverse-i-search)`apt-': apt-get update &amp;&amp; apt-get upgrade &amp;&amp; apt-get dist-upgrade &amp;&amp; apt-get autoremove

I think the main one being 10periodic but I didn’t fancy spending days/weeks adding each and waiting to see what happened, so I added them all in one go.

Then run /etc/cron.daily/apt again, and this time it’s taken a lot longer to run i.e I’ve written this entire post and it’s still running, as opposed to exiting within a few seconds earlier.

I’m pretty confident that this has now solved my APT package list not automatically updating problem, and providing it has Nagios will start warning on these 3 servers now.

Hyperion LED’s & Nagios (Part 3)

Hopefully you’d read Part 2. If not you’ll need to or this wont work.

So in this part we’re going to setup the nagios stuff to set off the alert LED’s. As a little bit of background my nagios installation is on a completely separate PI to the hyperion LED’s, but I have install hyperion on this pi to make use of hyperion-remote. Yes I was being lazy I could have used other methods instead of installing the whole thing.

First my nagios installation is in ‘/usr/local/nagios’, I’m not going to go through the commands to cd and edit, if you’ve installed and configured nagios I’ll assume you can do them 🙂

This is my /usr/local/nagios/libexec/notify-hyperion.sh

#!/bin/bash
STATE=$1
DURATION=23000
case $STATE in
"CRITICAL")
   EFFECT="Red Alert"
   ;;
"WARNING")
   EFFECT="Yellow Alert"
   ;;
"OK")
   EFFECT="Green Alert"
   ;;
*)
   ;;
esac

hyperion-remote -a osmc-l:19444 -d $DURATION -p 10 -e "$EFFECT" &amp;
hyperion-remote -a rasp-light:19444 -d $DURATION -p 10 -e "$EFFECT" &amp;
hyperion-remote -a webcam-pi:19444 -d $DURATION -p 10 -e "$EFFECT" &amp;

For the nagios saavy amongst you, you’ll see I account for CRITICAL, WARNING & OK. Yes I do need to add DOWN, UNREACHABLE & UP for the host alerts

The DURATION sets how long in ms hyperion will run this effect for (best worked out in conjunction with the speed, freq & step from the hyperion config. I’ve got this just right to cut off the alert after (I think) 4 fades. I force the priority ‘-p 10’ to 10, anything else I do with hyperion generally has a priority 50, 100 or 1000 so these will take over.

The last 3 lines are 1 each for my hyperion installs, you will need to change the name’s or replace them with the IP addresses dependant upon your network configuration.

Don’t forget to make the script executable, and you can test it with ‘./notify-hyperion.sh OK’

With the above tested and working, I’ve added the following to my nagios command.cfg

# 'notify-host-by-hyperion' command definition
define command{
        command_name    notify-host-by-hyperion
        command_line    /usr/local/nagios/libexec/notify-hyperion.sh "$HOSTSTATE$"
}

# 'notify-service-by-hyperion' command definition
define command{
        command_name    notify-service-by-hyperion
        command_line    /usr/local/nagios/libexec/notify-hyperion.sh "$SERVICESTATE$"
}

Then added the following to contacts.cfg

define contact{
        contact_name                    nagios-hyperion
        alias                           Nagios Hyperion
        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f
        host_notification_options       d,u,r,f,s
        service_notification_commands   notify-service-by-hyperion
        host_notification_commands      notify-host-by-hyperion
        }

define contactgroup{
        contactgroup_name       nagioshyperion
        alias                   Nagios Hyperion Notifications
        members                 nagios-hyperion
        }

For my installation I’ve then added

contact_groups                  nagioshyperion

To my template’s. You could add this to each service/host.

Within my setup I’ve stopped using email alerts, so changing the contacts to hyperion was fine. Within the templates I then have the notify_interval setup for 15 minutes. This means it will fire an alert to hyperion every 15 minutes. If you use email on your system too, you may not want to do this. an alternative could be changing the duration above, so that the red and yellow alerts are constant and the green run’s for a limited time before clearing down.

I did contemplate using event filters instead of contacts, so I could have the emails turned back on at some point, but decided against it as I would have to check before sending a green alert that it’s not already in green or I’d just end up with green all the time.

After all of the above make sure you restart nagios for the new config to take effect.

As a side note, I was sat watching TV this evening all of a sudden my room was yellow and I thought WTF. I do have hyperion setup in my room to start the LED’s at sunset but it was still light out and shouldn’t have fired. Then it clicked NAGIOS. and yes hey presto nagios had throw this site into warning status as there were updates available. I can see it getting annoying if my ISP drops out and I end up with alerts for hours. but on the whole I’m really happy it works, all I need now is a red alert klaxon 🙂

If your interest in setting up hyperion at sunrise/sunset I’ll be writing that one up separately.

Hyperion LED’s & Nagios (Part 1)

Part 1 is more Background story on my use of WS2801 WS2812b and Hyperion with the Raspberry Pi. Skip to Part 2 for the techy bit.

I’ve been using Hyperion for a while. I setup light behind the TV first off (using sticky tape) WS2801 and RaspBMC. This work brilliantly and I loved it. Spent hours tweaking the config so the LED’s were picking up the correct colour to the screen.

With all that working and a length of LED’s left over I decided to run some up the stairs. They sit just under the banister lip so you can’t see them, just the light on the stairs. I set these to Rainbow swirl and let it. They’re been running for months, occasionally changing the effect to show off what they can do.

Then disaster struck, the power adapter stopped working. Have to be honest I didn’t really notice until I was going to bed at 2am and almost fell over. They’ve been there giving off light (possibly a bit bright if anything) and I just got used to being able to see in the middle of the night without any other lights.

Anyway I digress, ordering a new power adapter I went searching for more LED’s (yes you can’t have enough of them once you’ve been playing). I decided that I’d really like to run some in my bedroom, the effects are cool and there would be plenty of light I wont need to use the main light with them on.

So I looking at where I originally bought my WS2801’s and nothing 🙁 so off to google, the obvious thing was I was going to have a hard time sourcing them in the UK. but why they’re great. So off I went to the hyperion git site for info and found there’s newer versions WS2811 and WS2812b. ah that may help, another search and I found someone selling a load on ebay. So I bought all he had 3xreels of WS1812b’s.

They turned up and I connected them up to try them as directed by hyperion. It was at this point I read the important bits RPi2 isn’t working yet and there maybe a problem with the PI communicating with them due to the voltage. I really thought I was going to have to make another little circuit to (buffer?) get them working. As a last ditched attempt it was mentioned try removing the resistor and try running them direct from python. I did both at the same time (not the best decision for troubleshooting. But to my surprise they worked.

So I killed the python program and restart hyperion, yep they’re working.

So off I went to stick them to the ceiling (they have sticky tape on the back). Done. If only I’d thought about connecting them before I stuck them up. I now had to work up in the air joining the cables. Not to worry I’ve done worse.

So I go to get what I need, by the time I got back up they’ve come down 🙁 bloody gravity! Now you’d think at this point I’d connect them up and sort out attaching them later Nope! (didn’t even enter my head) I was now on track to get them to stay up. Enter ‘SuperGlue’, applied little dots along the strip and stuck them up (yes I glued my fingers to the ceiling too). Finally they’re up and staying there. Oh I should have connected them when they were down!

‘Bugger it, where’s my screw driver’ I connected them up, put a power connector on the end and powered them and the PI.

Then installed hyperion on yet another pi. and it all worked like magic.

Have a look at the video, there’s no light other than the TV and it’s dark outside, but the room is really bright.

[youtube=https://www.youtube.com/watch?v=khfJW3vXcCE]

Click here for Part 2

Nagios Twitter Alerts

I’ve had nagios running for years, so decided to play around with the alerts.
Twitter seemed the obvious choice, it’s easy for a people to follow the twitter account that’s publishing the alerts, and great if you actively use twitter (I don’t, this was more of a ‘how would you’ rather than a need).
First thing is to register a twitter account that nagios will publish as. I setup https://twitter.com/NagiosStarB
Once you’ve registered you need to edit your profile and add a mobile phone number (This is needed before you can change the app permissions later. Once you’ve done that you can delete the mobile number).
Now head over to https://apps.twitter.com/ and create a new app
Fill in Name, Description and Website (This isn’t particularly important, as we’re not pushing this app out to users).
You’ll be taken straight into the new app (if not simply click on it).

We need to change the Access Level, Click on ‘modify app permissions’

I chose ‘Read, Write and Access direct messages’, although ‘Read and Write’ would be fine. Click ‘Update settings’ (If you didn’t add your mobile number to your account earlier, you;ll get an error).
Now click ‘API Keys’

You need to copy the API key and API secret (Please dont try to use mine).
Now click ‘create my access token’ close to the bottom of the page.

You also need to copy your ‘Access token’ and ‘Access tocken secret’

Now we move onto the notification script.
Login to your Nagios server via SSH.
You need to ensure you have python-dev & python-pip installed.

apt-get install python-dev python-pip
pip install tweepy

Then cd into your nagios libexec folder (mines at /usr/local/nagios/libexec) 

cd /usr/local/nagios/libexec/

We now add a new file called twitternagiosstatus.py 

nano -w twitternagiosstatus.py

Copy and paste the following code into the file 

#!/usr/bin/env python2.7
# tweet.py by Alex Eames http://raspi.tv/?p=5908
import tweepy
import sys
import logging

# Setup Debug Logging
logging.basicConfig(filename='/tmp/twitternagios.log',level=logging.DEBUG)
logging.debug('Starting Debug Log')

# Consumer keys and access tokens, used for OAuth
consumer_key = 'jNgRhCGx7NzZn1Cr01mucA'
consumer_secret = 'nTUDfUo0jH2oYyG8i6qdyrQXfwQ6QXT7dwjVykrWho'
access_token = '2360118330-HP5bbGQgTw5F1UIN3qOjdtvqp1ZkhxlHroiETIQ'
access_token_secret = 'rXjXwfoGGNKibKfXHw9YYL927kCBQiQL58Br0qMdaI5tB'

# OAuth process, using the keys and tokens
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret)

# Creation of the actual interface, using authentication
api = tweepy.API(auth)

if len(sys.argv) >= 2:
    tweet_text = sys.argv[1]
    logging.debug('Argument #1 ' + tweet_text)

    if len(tweet_text) <= 140:
        logging.debug('Tweeting: ' + tweet_text)
        api.update_status(tweet_text)
    else:
        print "tweet sent truncated. Too long. 140 chars Max."
        logging.debug('Too Long. Tweet sent truncated.')
        api.update_status(tweet_text[0:140])

Replace consumer_key with your API key, consumer_secret with your API secret, access_token with your access token and access_token_secret with your Access token secret.
Now save and exit the editor.

CTRL+x then Y then Enter.

With the file saved, we need to make it executable. 

chmod +x twitternagiosstatus.py

You can now test that the script works by typing 

./twitternagiosstatus.py "testy testy"

You should now be able to see the Tweet on your new account (you may need to refresh the page).
If all has gone well so far, you can now add your Nagios Configuration.
Change Directory into your nagios etc 

cd /usr/local/nagios/etc/

Edit your commands.cfg (mine is inside objects) 

nano -w objects/commands.cfg

Where you choose to place the new configurations doesn’t really matter, but to keep things in order I choose just below the email commands.
Copy and paste the following 

# 'notify-host-by-twitter' command definition
define command{
        command_name    notify-host-by-twitter
        command_line    /usr/local/nagios/libexec/twitternagiosstatus.py "$NOTIFICATIONTYPE$: $HOSTALIAS$ is $HOSTSTATE$"
}

# 'notify-service-by-twitter' command definition
define command{
        command_name    notify-service-by-twitter
        command_line    /usr/local/nagios/libexec/twitternagiosstatus.py "$NOTIFICATIONTYPE$: $SERVICEDESC$ ON $HOSTALIAS$ is $SERVICESTATE$"
}

You can adjust the specifics, but adding other $$ arguments (Use the email notification commands as an example). Save and exit 

CTRL+x, then Y, then ENTER

Now we add a new contact. Edit contacts.cfg 

nano -w objects/contacts.cfg

Copy and Paste the following 

define contact{
        contact_name                    nagios-twitter
        alias                           Nagios Twitter

        service_notification_period     24x7
        host_notification_period        24x7
        service_notification_options    w,u,c,r,f
        host_notification_options       d,u,r,f,s
        service_notification_commands   notify-service-by-twitter
        host_notification_commands      notify-host-by-twitter
        }

define contactgroup{
        contactgroup_name       nagiostwitter
        alias                   Nagios Twitter Notifications
        members                 nagios-twitter
        }

I decided to create a specific contact and contact-group for this, but you can adjust as you wish, add the contact to other contact-groups if you wish.
Now the last bit,
Add the new contact group to the hosts & services, templates or host-groups and service-groups.
How you decide to do this will depend on how you’ve set out your hosts, services, templates and contacts. For me I edit the each of the host files and add contact_groups  nagiostwitter to each host and service.
(IMPORTANT: this will override settings that are inherited from templates, so if you already have email notifications active you’ll either have to just add nagiostwitter to the template or add users to this). Dont forgot to , delimited
An example host of mine 

define host{
        use                     linux-server            ; Name of host template$
                                                        ; This host definition $
                                                        ; in (or inherited by) $
        host_name               excalibur
        alias                   Excalibur
        address                 192.168.1.27
        parents                 switch-netgear8
        hostgroups              linux-servers
        statusmap_image         linux40.gd2
        contact_groups          nagiostwitter,sysadm
        }

An example service on this host 

define service{
        use                             generic-service         ; Name of servi$
        host_name                       excalibur
        service_description             PING
        check_command                   check_ping!100.0,20%!500.0,60%
        contact_groups                  nagiostwitter,sysadm
        }

That’s it, hopefully if all’s done right you can restart the nagios service. 

/etc/init.d/nagios restart

Now your twitter feed will start to be populated with each alert. I can’t emphasis enough that if the nagios configuration is done wrong you may break other alerts that are already setup.
I really need to thank http://raspi.tv/2013/how-to-create-a-twitter-app-on-the-raspberry-pi-with-python-tweepy-part-1#install here as I used this as a starting point.

UPDATE:
A few weeks ago I received an email from twitter telling me my application had been blocked for write operations. It also said to check the Twitter API Terms of Service. I didn’t think this would cause a problem, I’m not spamming anyone other than myself or users I’ve asked to follow the alerts. So I read the Terms of Service, and it’s all fine. I raised a support request with Twitter and had a very quick response saying “Twitter has automated systems that find and disable abusive API keys in bulk. Unfortunately, it looks like your application got caught up in one of these spam groups by mistake. We have reactivated the API key and we apologize for the inconvenience.”
This did stop my alerts for a few days though.So just be aware of this.

UPDATE 2:
Thanks to a comment from Claudio to truncate messages over 140 characters. I’ve incorporated this recommendation into the code above.

NRPE compile error Cannot find ssl libraries

For years I’ve avoided SSL with NRPE because it just never seemed to work for me and on an internal network is it really needed?

I’m now doing a new fresh install of Nagios on a raspberry PI and decided after recently setting up SSL certificates on all my sites, to see if I can get this working with NRPE.

First things first, I’ve made sure the following are installed

apt-get install openssl libssl-dev build-essential

It’s also presumed you have already compiled and installed the nagios plugins and have a nagios user and group. Download and unpacked nrpe 2.13

mkdir nagios
cd nagios
wget http://prdownloads.sourceforge.net/sourceforge/nagios/nrpe-2.13.tar.gz
tar xzvf nrpe-2.13.tar.gz

Now onto the actual work:-

cd nrpe-2.13/
./configure

After a little time I got the nice error I’m used to seeing

checking for SSL libraries… configure: error: Cannot find ssl libraries

So after a little searching I hit upon libssl.so missing. Well not so much missing, it just doesn’t have a link where it’s expected. The answer to which is to create a new symlink to it

ln -s /usr/lib/arm-linux-gnueabihf/libssl.so /usr/lib/libssl.so

UPDATE: Noticed on another system it didn’t work, this was because /usr/lib/libssl.so was already present but pointing to the wrong place. This stopped the link being created. So I ran rm/usr/lib/libssl.so then reran the above, this create the link properly and the ./configure then runs as normal.

On my system it’s in the /usr/lib/arm-linux-gnueabihf folder, but on an x86 system this could be something like /usr/lib/x86_64-linux-gnu/

So now I rerun the configure step 

./configure

At the point I see

*** Generating DH Parameters for SSL/TLS ***
Generating DH parameters, 512 bit long safe prime, generator 2
This is going to take a long time
…………………+…………………+…………+…………………….++*++*++*++*++*++*

I know it’s taken the SSL stuff well. When it’s complete the configure run without errors you can continue to the make stage

make all

Update /etc/services inserting “nrpe 5666/tcp # NRPE” 

nano -w /etc/services

I’ve always run nrpe under xinit.d for all my installs, so make sure xinitd is installed 

apt-get install xinetd

Once it’s installed you need to add a new service. edit the file /etc/xinit.d/nrpe

nano -w /etc/xinit.d/nrpe

My sample configuration:- 

# default: on
# description: NRPE (Nagios Remote Plugin Executor)
service nrpe
{
  flags           = REUSE
  socket_type     = stream
  port            = 5666
  wait            = no
  user            = nagios
  group           = nagios
  server          = /usr/local/nagios/bin/nrpe
  server_args     = -c /usr/local/nagios/etc/nrpe.cfg --inetd
  log_on_failure  += USERID
  disable         = no
  only_from       = 127.0.0.1 192.168.0.0/16
}

You will probably need to change the user, group and only_from fields to suit your installation.
Next you’ll need a configuration. I’m just going to copy the sample for now 

mkdir /usr/local/nagios/etc
cp sample-config/nrpe.cfg /usr/local/nagios/etc/nrpe.cfg
chown -R nagios:nagios /usr/local/nagios/etc/

Lastly restart the xinitd service 

etc/init.d/xinetd restart

You can test your nrpe installation from your nagios server with the check_nrpe command. It’s probably worth also checking the syslog or messages for the system after restarting xinitd as any errors regarding startup will be reported.
I’ve tested from my server using 

/usr/local/nagios/libexec/check_nrpe -H titan

titan being the name of the server I have just complete an installation of plugins+nrpe on (I’ve already run through the above on titan itself). It responds with

NRPE v2.13

So all working.

UPDATE #2
I skipped through the steps above and was trying to just return the NRPE version, I kept hitting the error: “CHECK_NRPE: Error – Could not complete SSL handshake.”
Thinking something was wrong with the SSL I went back through everything, but same error. I then realised I hadn’t yet put a config in place for NRPE (as I was only trying to return the version number didn’t think it was too important) a silly mistake, but returns an off putting error.

Nagios Plugins fail to Compile

Getting the error

check_http.c:150:12: warning: ignoring return value of âasprintfâ, declared with attribute warn_unused_result [-Wunused-result]

When trying to compile the nagios plugins 1.4.16 in Ubuntu 12.04, it comes down to SSL.
Running : apt-get install libssl-dev
Then ./configure and make again fixed the problem.

As a side note I was also having problems with NRPE not installing. This one was down to not reading the error. the user and group nagios doesn’t get created by the nrpe install script. So just have to add the user and group and then the install was fine.