Ubuntu Upstart

Well I posted yesterday about handing over the control of my webcams script and the Zoneminder viewer xlib_shm to a daemon manager, as it seemed the right way to go and I noticed earlier tonight that my 2nd machine has again crashed out on the xlib_shm.
First I did a search for the daemontools to get an idea of what it’s like, but alot of the postings were older. So then decided to do an ‘alternatives’ search which brought up a few results, only one of which mentioned upstart in ubuntu, hey presto! I’ve heard upstart before but never looked at it (at least I thought).
Did a bit of looking around and yep I remember reading that Ubuntu is moving away from the init.d scripts a while ago just never put it together with the word upstart.
Basically there’s new files in /etc/init/ which is what upstart looks at and their the configs for each service that’s starting on the machine. Funnily enough I remembered then where why I’d done some searching previously on /etc/init/ and it was for gdm.conf because the gdm kept launching even though I’d removed it with the update-rc.d -f gdm remove. I was trying to kill it off because it was grabbing the screen before xlib_shm could.
Anyway that’s a bit of a side track. Looking at some of the configs there and the FAQ it looks quiet powerful and will do exactly what I need.

I was going to jump straight in and make a new config for the webcams stuff, but then noticed you can have a service start on another, well that should be good I only want webcams starting if zoneminder is running so let’s check that config, oh no there isn’t one yet. After 2 mins of searching for a config some clever person has already made I’m going to call it a night and have a proper look tomorrow. but it got me thinking of moving more over to Upstart like my asterisk stuff that will also need and depend on dahdi.
You can also depend on having a filesystem and networking. I can see the filesystem being used for the zoneminder config and both being used for the webcams config. So I’m really looking forward to getting into this a bit too. There’s only a handful of occasions when asterisk has shutdown so not a huge problem, but when it has I did once get left with no phone line for 3 days cause I didn’t know it wasn’t running (obviously enjoying the peace and quiet too much). To solve this I have Nagios monitoring now and using event handlers to restart via the init.d script but handing over to Upstart really does seem the way forward. I’d still keep Nagios monitoring to ensure it’s running I can’t live without my nagios watching over everything.

Zoneminder Webcams 2.0 Part 2

I said I’d put up the script I’m using so here it is. Couple of notes first.
I know PHP isn’t particularly the best language to use for scripts, but I’ve been using it for work within a website and since my head was already in PHP mode it seemed easier to just keep going and get the idea down and running than be typing stuff wrong all the time and have to keep correcting it to get each bit working.
I call this script to start running as zoneminder loads from the init.d/zoneminder file, exactly the same way I started the last one. Something I did hit though is if PHP tries to output and there’s no console it crashes out. So the diagnostic setting should be off if it’s live and only used if your running it from command line for testing. For some reason running it from command line with & to background it starts it off but on trying to continue in the terminal it stops the process. I’m sure there’s a perfectly good explanation for this like it’s passing the input into the php script which isn’t to output and falls over, but I don’t need to run it from command line unless testing so this bit doesn’t bother me.

Requirements:-
New folder, /var/cache/zoneminder/webcams, /var/cache/zoneminder/webcams/originals, /var/cache/zoneminder/webcams/resized
This also requires the ‘convert’ command (same as the previous but I forgot to note that). if you try to run convert under unbuntu it’ll tell you what to do to install it if you dont have it.

edit a new file ~/webcams2.php and paste the following:-

#!/usr/bin/php
<?php
$DIAG = 2;
$LOGPATH = "/tmp/webcams.log";
$host = "127.0.0.1";
$user = "root";
$pass = "";
$db   = "zm";

$directory = "/var/cache/zoneminder/webcams/";
$dblastid = "0";
$mainpull = "0";
$nextrotate = "10";
$nextpull = "60";
$autoadjustrefresh = FALSE;

$ms = mysql_pconnect($host, $user, $pass);
if ( !$ms )
        { echo ""; }

mysql_select_db($db);
//connection complete.

chdir ( $directory );

if ($DIAG > 0) { $LogFile = fopen($LOGPATH, 'a') or die("can't open file"); };
if ($DIAG > 0) { fwrite($LogFile, "Starting Logfile at : " . time(). " ...n"); };

function RefreshAllImages() {
 global $directory, $dblastid, $nextpull,$DIAG,$LogFile;
 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);

 if (!$r) { echo ("Problemsr"); } else {
  for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
    $filenamepath = $directory . "originals/" . mysql_result($r,$rows,'Monitor') . mysql_result($r,$rows,'ID') . ".jpg";
    $filename = str_replace($directory."originals/","",$filenamepath);
    $filetime = filemtime($filenamepath);
    if ($DIAG >= 2) { fwrite($LogFile, "Pulling File : " . $filenamepath . "n"); };
// old command    $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
    $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
    if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
    exec($cmd);
    clearstatcache(TRUE, $filenamepath);
    if ( (filesize($filenamepath)!="0") && ($filetime != filemtime($filenamepath)) ) {
      if ($DIAG >= 2) { fwrite($LogFile, "Converting File : " . filectime($filenamepath) . $filenamepath . "n"); };
      $cmd = "convert " . $filenamepath . " -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $filename;
      exec($cmd);
      touch ( $directory . "resized/" . $filename, filemtime($filenamepath) );
    };
  };
 $dblastid = mysql_result($r,$rows-1,'ID');
 };
if ($DIAG >= 2) { fwrite($LogFile, "Finished Pulling All Files!n"); };
$nextpull = (time() + 60);
};

function RefreshImages() {
 global $directory, $dblastid, $nextpull, $autoadjustrefresh, $DIAG, $LogFile;

 clearstatcache();
 if ($DIAG >= 2) { fwrite($LogFile, "Refreshing Images...n"); };

 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);
 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
  $start = mysql_result($r,$rows,'Start');
  $stop = mysql_result($r,$rows,'Stop');
  if ($DIAG >= 2) { fwrite($LogFile, "Start : $start. Stop : $stop. Time : " . date("H:i:s") . ".n"); };
  if ( ( ($start < $stop) && ($start < date("H:i:s")) && ($stop > date("H:i:s")) ) || ( ($start > $stop) && ( (($start < date("H:i:s")) && ($stop < date("H:i:s"))) || (($start > date("H:i:s")) && ($stop > date("H:i:s"))) ) ) ) {
   if ($DIAG >= 2) { fwrite($LogFile, "In timeframe running checks...n"); };
   $row_filename = mysql_result($r,$rows,'Monitor').mysql_result($r,$rows,'ID').".jpg";
   $row_id = mysql_result($r,$rows,'ID');
   $row_refresh = time() - mysql_result($r,$rows,'Refresh');
   $filenamepath = $directory."originals/".$row_filename;
   $filectime = filectime($filenamepath);
   $filemtime = filemtime($filenamepath);
   if ( (($filemtime > time() - 3600) && ($filectime < $row_refresh)) || (!file_exists($filenamepath)) ) {
     if ($DIAG >= 2) { fwrite($LogFile, "Pulling New File : " . filectime($filenamepath) . $filenamepath . "n"); };
// old command     $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
     $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
     if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
     exec($cmd);
     clearstatcache(TRUE, $filenamepath);
     if (( filemtime($filenamepath) > $filemtime ) && (filesize($filenamepath)!="0") ) {
       if ( $autoadjustrefresh == TRUE ) {
         if ($DIAG >= 2) { fwrite($LogFile, "Adjusting refresh time.n Calculating difference...n"); };
         $timedifference = filemtime($filenamepath) - $filemtime;
         if ($DIAG >= 2) { fwrite($LogFile, "Difference : " . $timedifference . "n"); };
         if ( ($timedifference > (mysql_result($r,$rows,'Refresh')+30) || $timedifference < (mysql_result($r,$rows,'Refresh')-30)) && ($timedifference < 900)) {
           if ($DIAG >= 2) { fwrite($LogFile, "Updating Database Refresh timer.n"); };
           $q2="UPDATE WebCams set Refresh='" . ($timedifference) . "' WHERE ID='$row_id'";
           $r2 = mysql_query($q2);
         };
       };
       if ($DIAG >= 2) { fwrite($LogFile, "Attempting resize : " . filemtime($filenamepath) . $filenamepath . "n"); };
       $cmd = "convert $filenamepath -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $row_filename;
       exec($cmd);
       touch ( $directory . "resized/" . $row_filename, filemtime($filenamepath) );
     } else {
       if ($DIAG >= 2){ fwrite($LogFile, "$row_filename Not Newer : " . filemtime($filenamepath) . " <= " . $filemtime . " Ignoringn"); };
     };
   };
  };
 };
$nextpull = (time() + 60);
};

function RotateImages() {
 global $directory, $nextrotate,$DIAG,$LogFile;
 $newimageset = FALSE;
 $imagefound = FALSE;

 clearstatcache();

 $q = "SELECT distinct(Monitor) from WebCams";
 $r = mysql_query($q);

 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
   if ($DIAG >= 1){ fwrite($LogFile, "Rotating Monitor " . mysql_result($r,$rows,'Monitor') . "n"); };
   if ($DIAG >= 1){ fwrite($LogFile, "Checking for New Files ...n"); };
   $newimageset = FALSE;
   $imagefound = FALSE;
   if (mysql_result($r,$rows,'Monitor') != 'C'){
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "  " . $filenamepath ); };
       if (filectime($filenamepath) > (time()-15) ) {
         if ($DIAG >= 1){ fwrite($LogFile, "n New file Found linking ... n"); };
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };

   if ($DIAG >= 1){ fwrite($LogFile, "Finished lookin for newer files.n"); };

   if ( $newimageset == FALSE) {
     $rotatefile = $directory."rotate".mysql_result($r,$rows,'Monitor').".jpg";
     if (file_exists($rotatefile) ) {
       if ($DIAG >= 1){ fwrite($LogFile, "nLooking for Link File : $rotatefile ...n"); };
       $currentfile = readlink($rotatefile);
       if ($DIAG >= 1){ fwrite($LogFile, "Current : " . $currentfile . "n"); };
       foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
         if ( $filenamepath == $currentfile ) {
           if($DIAG >= 1){ fwrite($LogFile, "Found Correct entry.n"); };
           $imagefound = TRUE;
         } elseif (($imagefound) && (!$newimageset)) {
           if ($DIAG >= 1){ fwrite($LogFile, "Checking next Image...n"); };
           if (filectime($filenamepath) > (time() - 1800) ) {
             if ($DIAG >= 1){ fwrite($LogFile, "Next Suitable Image " . $filenamepath . " Linking...n"); };
             exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
             $newimageset = TRUE;
           };
         };
       };
     };
   };

   if ($newimageset == FALSE) {
     if ($DIAG >= 1){ fwrite($LogFile, "Image still not set, Falling back to load first suitablen"); };
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "Checking File " . $filenamepath . "n"); };
       if (filectime($filenamepath) > (time() - 1800) && ($newimageset == FALSE) ) {
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };
 };
$nextrotate = (time() + 10);
};

$pulled = 0;
for (;;) {
 if (time() > $nextpull) {
  if ($pulled == 0 or $pulled == 60) {
   RefreshAllImages();
   $pulled = 1;
  } else {
   RefreshImages();
   $pulled++;
  };
 };
 if (time() > $nextrotate) {
  RotateImages();
 };
 sleep(2);
};
?>


Save the file then chmod +x ~/webcams2.php

You will likely need to change the database connection info, it’s not a good idea to connect to the database as root and I’ve removed my password.
An option I would leave off at the moment is ‘autoadjustrefresh’ This is meant to compare the times on the files it pulls and calculate the difference since the last pull.
The theory behind this is as follows:
Database is set to pull a new image every 300 seconds (5 mins). but the webcam itself is only updated every 15 mins. Now obviously you can sit and work this out yourself by just watching, but the files I’ve been pulling are generally modified timestamped with the servers time, so I can use this to calc the difference when a new image is obtained. Unfortunately this seems to go wrong and has made some of my feed not update for 30 min intervals even though I know the feed is refreshed every second and I’ve told it to pull every 120 secs. 
I figured it was adding a few seconds here and there for delays but it’s not that.
Another issue is on the rotating side it’s supposed to look if there’s a new image just come in (in the last 40 secs) if so display this as a priority. This theory again is sound, until I realised yesterday that some of the servers clocks are out by minutes, this means when I say the last 40 secs the file is already way beyond that so it doesn’t get shown as a priority. I have a fixed in mind for this though, which revolves around using the resized files creation time as a check as I only resize if a new image was pulled anyway.
Well that’s the script, to pickup the image in zoneminder same as the previous script I just have a new camera looking at a file /var/cache/zoneminder/webcams/rotateA.jpg which is set at 1fps (doesn’t need to be fastas it’s pretty static)
I much prefer this script over the last as it reads from the database and puts the camera name on the image, which is great for some of the motorway cams that give you no indication where they are.
it also means adding a new entry is straight forward and get’s picked up quickly.
It still needs some work so I’ll post changes up as and when. If anyone is reading and has suggestions drop me a comment.
I’ve been considering getting stuff like this to be run by a daemon manager that will restart it if it fails etc. may do that at some point, but for now starting it with zoneminder works for me. On the 2nd box I put it on I’m still running into some issues where xlib_shm is crashing, and I haven’t built any checks for that into this, that’s one reason I’m considering a daemon manager as xlib_shm and the webcams2.php script shouldn’t need to speak or check each other it was just a dirty way of keeping it running. Think about what maybe happening as I’m typing though, that server only outputs stuff from the webcams feed and I noticed with the previous script if the image was 0 bytes it obvious can’t convert. giving a dud image to zoneminder is probably enough for it to restart that monitor and as it’s the only one xlib_shm would probably fall over as it can’t read the memory zoneminder is using. kinda makes sense as my display occasionally come up blank or part converted image if it’s in the middle of running, but as I have live monitors zoneminder wouldn’t stop the lot. Will have to look into that a bit more, but handing to a proper daemon manager should be the way to go anyway it can handle crashes then.
Note: I need to do better formatting for code etc.

Zomeminder Webcams 2.0

Ok, following on from my post yesterday here’s the database table part of my new scripts.

CREATE TABLE `WebCams` (
`ID` INT(11) NOT NULL AUTO_INCREMENT,
`Name` VARCHAR(50) NULL DEFAULT NULL,
`URL` VARCHAR(200) NULL DEFAULT NULL,
`Start` TIME NULL DEFAULT NULL,
`Stop` TIME NULL DEFAULT NULL,
`Refresh` INT(5) NULL DEFAULT NULL,
`Monitor` VARCHAR(1) NULL DEFAULT NULL,
`Enabled` INT(1) NULL DEFAULT NULL,
PRIMARY KEY (`ID`) )
)

I place this under the zoneminder database itself, that way it’s all kept together.
A sample entry would be:-

INSERT INTO `WebCams` (`ID`, `Name`, `URL`, `Start`, `Stop`, `Refresh`, `Monitor`, `Enabled`) VALUES (1, 'Aberystwyth Pier', 'http://www.ceredigion.gov.uk/english/visiting/aber_webcam/img/994.jpg', '00:00:00', '23:59:59', 900, 'A', 1);

Pretty straight forward descriptions for Name, URL, Start & Stop. One thing to note is I made the script to interpret the start and stop so you can start at 10pm and finish at 2am and it will happily work (took a bit for my head to get around it though).
The Refresh column is how often it should grab a new image. I slowed the script down to run a refresh every 60 secs for a few reasons, first not to query the database constantly and 2nd as a safeguard that I can’t hammer any webcams trying to pull too often.
The Monitor I use as A,B,C this will be assigned to the name of what’s downloaded and to the rotate file that you reference in Zoneminder.
Enabled is either 0-No or 1-Yes.

That’s pretty much it. Next bit I’ll post up the actual script.

Zoneminder rotating Webcam images.

Note: I’ve made an entire new system running from a database that can accommodate multiple rotating outputs, because I liked seeing the webcams so much on my tv along with my own cctv. I leave this here as an idea script. btw, there was an update for 3 monitors based on the below that used this very script, but I’m going to jump straight to the database one as it’s much better.

I had the idea to include some public webcams onto my zoneminder monitor, I’m already running xlib_shm so was considering find a way within that code to pickup some jpegs and display them in one area. However this seemed overly complicated, it seemed a much simpler idea to create a new monitor in zoneminder that picks up from a local file and just add that monitor onto the xlib_shm line in the xinitrc file so it displays on bootup.

So the steps.
1) create a new directory /var/cache/zoneminder/webcams
2) create a text doc to hold the url’s /var/cache/zoneminder/webcams/urls.txt for example “http://www.ceredigion.gov.uk/english/visiting/aber_webcam/img/994.jpg” without “
3) create a text doc ~/webcams.sh with the following :
#!/bin/bash
DIR=/var/cache/zoneminder/webcams
PULLINTERVAL=300
ROTATEINTERVAL=13
CHECKINTERVAL=300
NEXTPULL=0
NEXTROTATE=0
NEXTXLIBCHECK=0
CURRENTIMAGE=1
echo Directory : $DIR
echo Pull Interval : $PULLINTERVAL
echo Rotate Interval : $ROTATEINTERVAL
cd $DIR
geturls()
{
FILENO=0
for i in `cat $DIR/urls.txt`
do
   FILENO=$[$FILENO+1]
   wget -q $i -O $FILENO.jpg
done
NEXTPULL=$[$DATE+$PULLINTERVAL]
}
rotateimage()
{
convert $CURRENTIMAGE.jpg -resize 640x480! rotate.jpg
if [ "$CURRENTIMAGE" -eq "$FILENO" ]; then
  CURRENTIMAGE=1
else
  CURRENTIMAGE=$[$CURRENTIMAGE+1]
fi
NEXTROTATE=$[$DATE+$ROTATEINTERVAL]
}
checkxlib()
{
PGREPXLIB=`pgrep xlib_shm`
 if [ -z $PGREPXLIB ]; then
   startx &
 fi
NEXTXLIBCHECK=$[$DATE+$CHECKINTERVAL]
}
while true; do
DATE=`date +%s`
   if [ "$DATE" -gt "$NEXTPULL" ]; then
   geturls
   fi
   if [ "$DATE" -gt "$NEXTROTATE" ]; then
   rotateimage
   fi
   if [ "$DATE" -gt "$NEXTXLIBCHECK" ]; then
   checkxlib
   fi
sleep 2
done
4) chmod +x ~/webcams.sh
5) Now add a new monitor to Zoneminder pulling from a local file /var/cache/zoneminder/webcams/rotate.jpg with dimensions 640×480.
6) After testing that this script runs while I’m connected via putty, I add the following to /etc/init.d/zoneminder under the startx & ‘/root/webcams.sh &’. That way when the computer starts up and everything is displayed the webcams script is also started and away it rotates.
7) Add the following to /etc/init.d/zoneminder under the stop section where I added kill -9 xlib_shm ‘killall -9 webcams.sh’. Now the script is stopped if zoneminder is stopped.
One thing to note is that restarting zoneminder via the interface is not the same as running /etc/init.d/zoneminder restart. So the script and does not get reset when restarting via the interface.