xlib_shm Zoneminder Got unexpected memory map file size X, expected Y

After upgrading my Ubuntu system first from 12.04 to 13.10 then onto 14.04.1 I started getting problems with xlib_shm
It wouldn’t start up, all the cameras were working fine from the web interface but nothing to the TV.

So I started troubleshooting and found the following error coming back at me.

shared_data_size=12166644
mem_size=884
Got unexpected memory map file size 4056640, expected 884

It doesn’t really make any sense though, it take you down the path of increasing the shmmax an shmall. But this was all working fine before the update and these values haven’t been changed. Still I increased them just incase.
No joy.
I also tried to recompile xlib_shm incase (being upgraded a few times) too much had changed. This unfortunately fell flat on its face when it can’t compile there’s stuff missing (we’ll come back to that).
After going round in circles for quite a while, I found while checking the camera configs that each camera had been set to 8bit greyscale. I set this back to 24bit colour. And restart. hey presto! my tv is once again filled with cameras.
I do still have a problem at the moment though, although the cams are now displaying they are frozen. I’m not sure though if this is more to do with how I start xlib_shm, and think it may be crashing out but not completely enough to take the display away.
So if your having problems with xlib_shm after upgrading either your OS or Zoneminder, check that your cams are setup properly. I almost missed the colour as it’s nighttime and they are grey anyway 🙂
As for the compile problems. It looks as though upgrading through 2 versions of Ubuntu has removed some packages, I’ll have to try and work out what incase I need to compile again.

Zoneminder Webcams 2.0 Part 2

I said I’d put up the script I’m using so here it is. Couple of notes first.
I know PHP isn’t particularly the best language to use for scripts, but I’ve been using it for work within a website and since my head was already in PHP mode it seemed easier to just keep going and get the idea down and running than be typing stuff wrong all the time and have to keep correcting it to get each bit working.
I call this script to start running as zoneminder loads from the init.d/zoneminder file, exactly the same way I started the last one. Something I did hit though is if PHP tries to output and there’s no console it crashes out. So the diagnostic setting should be off if it’s live and only used if your running it from command line for testing. For some reason running it from command line with & to background it starts it off but on trying to continue in the terminal it stops the process. I’m sure there’s a perfectly good explanation for this like it’s passing the input into the php script which isn’t to output and falls over, but I don’t need to run it from command line unless testing so this bit doesn’t bother me.

Requirements:-
New folder, /var/cache/zoneminder/webcams, /var/cache/zoneminder/webcams/originals, /var/cache/zoneminder/webcams/resized
This also requires the ‘convert’ command (same as the previous but I forgot to note that). if you try to run convert under unbuntu it’ll tell you what to do to install it if you dont have it.

edit a new file ~/webcams2.php and paste the following:-

#!/usr/bin/php
<?php
$DIAG = 2;
$LOGPATH = "/tmp/webcams.log";
$host = "127.0.0.1";
$user = "root";
$pass = "";
$db   = "zm";

$directory = "/var/cache/zoneminder/webcams/";
$dblastid = "0";
$mainpull = "0";
$nextrotate = "10";
$nextpull = "60";
$autoadjustrefresh = FALSE;

$ms = mysql_pconnect($host, $user, $pass);
if ( !$ms )
        { echo ""; }

mysql_select_db($db);
//connection complete.

chdir ( $directory );

if ($DIAG > 0) { $LogFile = fopen($LOGPATH, 'a') or die("can't open file"); };
if ($DIAG > 0) { fwrite($LogFile, "Starting Logfile at : " . time(). " ...n"); };

function RefreshAllImages() {
 global $directory, $dblastid, $nextpull,$DIAG,$LogFile;
 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);

 if (!$r) { echo ("Problemsr"); } else {
  for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
    $filenamepath = $directory . "originals/" . mysql_result($r,$rows,'Monitor') . mysql_result($r,$rows,'ID') . ".jpg";
    $filename = str_replace($directory."originals/","",$filenamepath);
    $filetime = filemtime($filenamepath);
    if ($DIAG >= 2) { fwrite($LogFile, "Pulling File : " . $filenamepath . "n"); };
// old command    $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
    $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
    if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
    exec($cmd);
    clearstatcache(TRUE, $filenamepath);
    if ( (filesize($filenamepath)!="0") && ($filetime != filemtime($filenamepath)) ) {
      if ($DIAG >= 2) { fwrite($LogFile, "Converting File : " . filectime($filenamepath) . $filenamepath . "n"); };
      $cmd = "convert " . $filenamepath . " -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $filename;
      exec($cmd);
      touch ( $directory . "resized/" . $filename, filemtime($filenamepath) );
    };
  };
 $dblastid = mysql_result($r,$rows-1,'ID');
 };
if ($DIAG >= 2) { fwrite($LogFile, "Finished Pulling All Files!n"); };
$nextpull = (time() + 60);
};

function RefreshImages() {
 global $directory, $dblastid, $nextpull, $autoadjustrefresh, $DIAG, $LogFile;

 clearstatcache();
 if ($DIAG >= 2) { fwrite($LogFile, "Refreshing Images...n"); };

 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);
 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
  $start = mysql_result($r,$rows,'Start');
  $stop = mysql_result($r,$rows,'Stop');
  if ($DIAG >= 2) { fwrite($LogFile, "Start : $start. Stop : $stop. Time : " . date("H:i:s") . ".n"); };
  if ( ( ($start < $stop) && ($start < date("H:i:s")) && ($stop > date("H:i:s")) ) || ( ($start > $stop) && ( (($start < date("H:i:s")) && ($stop < date("H:i:s"))) || (($start > date("H:i:s")) && ($stop > date("H:i:s"))) ) ) ) {
   if ($DIAG >= 2) { fwrite($LogFile, "In timeframe running checks...n"); };
   $row_filename = mysql_result($r,$rows,'Monitor').mysql_result($r,$rows,'ID').".jpg";
   $row_id = mysql_result($r,$rows,'ID');
   $row_refresh = time() - mysql_result($r,$rows,'Refresh');
   $filenamepath = $directory."originals/".$row_filename;
   $filectime = filectime($filenamepath);
   $filemtime = filemtime($filenamepath);
   if ( (($filemtime > time() - 3600) && ($filectime < $row_refresh)) || (!file_exists($filenamepath)) ) {
     if ($DIAG >= 2) { fwrite($LogFile, "Pulling New File : " . filectime($filenamepath) . $filenamepath . "n"); };
// old command     $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
     $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
     if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
     exec($cmd);
     clearstatcache(TRUE, $filenamepath);
     if (( filemtime($filenamepath) > $filemtime ) && (filesize($filenamepath)!="0") ) {
       if ( $autoadjustrefresh == TRUE ) {
         if ($DIAG >= 2) { fwrite($LogFile, "Adjusting refresh time.n Calculating difference...n"); };
         $timedifference = filemtime($filenamepath) - $filemtime;
         if ($DIAG >= 2) { fwrite($LogFile, "Difference : " . $timedifference . "n"); };
         if ( ($timedifference > (mysql_result($r,$rows,'Refresh')+30) || $timedifference < (mysql_result($r,$rows,'Refresh')-30)) && ($timedifference < 900)) {
           if ($DIAG >= 2) { fwrite($LogFile, "Updating Database Refresh timer.n"); };
           $q2="UPDATE WebCams set Refresh='" . ($timedifference) . "' WHERE ID='$row_id'";
           $r2 = mysql_query($q2);
         };
       };
       if ($DIAG >= 2) { fwrite($LogFile, "Attempting resize : " . filemtime($filenamepath) . $filenamepath . "n"); };
       $cmd = "convert $filenamepath -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $row_filename;
       exec($cmd);
       touch ( $directory . "resized/" . $row_filename, filemtime($filenamepath) );
     } else {
       if ($DIAG >= 2){ fwrite($LogFile, "$row_filename Not Newer : " . filemtime($filenamepath) . " <= " . $filemtime . " Ignoringn"); };
     };
   };
  };
 };
$nextpull = (time() + 60);
};

function RotateImages() {
 global $directory, $nextrotate,$DIAG,$LogFile;
 $newimageset = FALSE;
 $imagefound = FALSE;

 clearstatcache();

 $q = "SELECT distinct(Monitor) from WebCams";
 $r = mysql_query($q);

 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
   if ($DIAG >= 1){ fwrite($LogFile, "Rotating Monitor " . mysql_result($r,$rows,'Monitor') . "n"); };
   if ($DIAG >= 1){ fwrite($LogFile, "Checking for New Files ...n"); };
   $newimageset = FALSE;
   $imagefound = FALSE;
   if (mysql_result($r,$rows,'Monitor') != 'C'){
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "  " . $filenamepath ); };
       if (filectime($filenamepath) > (time()-15) ) {
         if ($DIAG >= 1){ fwrite($LogFile, "n New file Found linking ... n"); };
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };

   if ($DIAG >= 1){ fwrite($LogFile, "Finished lookin for newer files.n"); };

   if ( $newimageset == FALSE) {
     $rotatefile = $directory."rotate".mysql_result($r,$rows,'Monitor').".jpg";
     if (file_exists($rotatefile) ) {
       if ($DIAG >= 1){ fwrite($LogFile, "nLooking for Link File : $rotatefile ...n"); };
       $currentfile = readlink($rotatefile);
       if ($DIAG >= 1){ fwrite($LogFile, "Current : " . $currentfile . "n"); };
       foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
         if ( $filenamepath == $currentfile ) {
           if($DIAG >= 1){ fwrite($LogFile, "Found Correct entry.n"); };
           $imagefound = TRUE;
         } elseif (($imagefound) && (!$newimageset)) {
           if ($DIAG >= 1){ fwrite($LogFile, "Checking next Image...n"); };
           if (filectime($filenamepath) > (time() - 1800) ) {
             if ($DIAG >= 1){ fwrite($LogFile, "Next Suitable Image " . $filenamepath . " Linking...n"); };
             exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
             $newimageset = TRUE;
           };
         };
       };
     };
   };

   if ($newimageset == FALSE) {
     if ($DIAG >= 1){ fwrite($LogFile, "Image still not set, Falling back to load first suitablen"); };
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "Checking File " . $filenamepath . "n"); };
       if (filectime($filenamepath) > (time() - 1800) && ($newimageset == FALSE) ) {
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };
 };
$nextrotate = (time() + 10);
};

$pulled = 0;
for (;;) {
 if (time() > $nextpull) {
  if ($pulled == 0 or $pulled == 60) {
   RefreshAllImages();
   $pulled = 1;
  } else {
   RefreshImages();
   $pulled++;
  };
 };
 if (time() > $nextrotate) {
  RotateImages();
 };
 sleep(2);
};
?>


Save the file then chmod +x ~/webcams2.php

You will likely need to change the database connection info, it’s not a good idea to connect to the database as root and I’ve removed my password.
An option I would leave off at the moment is ‘autoadjustrefresh’ This is meant to compare the times on the files it pulls and calculate the difference since the last pull.
The theory behind this is as follows:
Database is set to pull a new image every 300 seconds (5 mins). but the webcam itself is only updated every 15 mins. Now obviously you can sit and work this out yourself by just watching, but the files I’ve been pulling are generally modified timestamped with the servers time, so I can use this to calc the difference when a new image is obtained. Unfortunately this seems to go wrong and has made some of my feed not update for 30 min intervals even though I know the feed is refreshed every second and I’ve told it to pull every 120 secs. 
I figured it was adding a few seconds here and there for delays but it’s not that.
Another issue is on the rotating side it’s supposed to look if there’s a new image just come in (in the last 40 secs) if so display this as a priority. This theory again is sound, until I realised yesterday that some of the servers clocks are out by minutes, this means when I say the last 40 secs the file is already way beyond that so it doesn’t get shown as a priority. I have a fixed in mind for this though, which revolves around using the resized files creation time as a check as I only resize if a new image was pulled anyway.
Well that’s the script, to pickup the image in zoneminder same as the previous script I just have a new camera looking at a file /var/cache/zoneminder/webcams/rotateA.jpg which is set at 1fps (doesn’t need to be fastas it’s pretty static)
I much prefer this script over the last as it reads from the database and puts the camera name on the image, which is great for some of the motorway cams that give you no indication where they are.
it also means adding a new entry is straight forward and get’s picked up quickly.
It still needs some work so I’ll post changes up as and when. If anyone is reading and has suggestions drop me a comment.
I’ve been considering getting stuff like this to be run by a daemon manager that will restart it if it fails etc. may do that at some point, but for now starting it with zoneminder works for me. On the 2nd box I put it on I’m still running into some issues where xlib_shm is crashing, and I haven’t built any checks for that into this, that’s one reason I’m considering a daemon manager as xlib_shm and the webcams2.php script shouldn’t need to speak or check each other it was just a dirty way of keeping it running. Think about what maybe happening as I’m typing though, that server only outputs stuff from the webcams feed and I noticed with the previous script if the image was 0 bytes it obvious can’t convert. giving a dud image to zoneminder is probably enough for it to restart that monitor and as it’s the only one xlib_shm would probably fall over as it can’t read the memory zoneminder is using. kinda makes sense as my display occasionally come up blank or part converted image if it’s in the middle of running, but as I have live monitors zoneminder wouldn’t stop the lot. Will have to look into that a bit more, but handing to a proper daemon manager should be the way to go anyway it can handle crashes then.
Note: I need to do better formatting for code etc.

Zoneminder XLIB_SHM

Ok it’s been a good while since my last post, and it’s not that I haven’t been doing anything, just not writing about it. Which just creates problems when I need to remember how what I did and how.

I needed something to get my monitors back on my similar to my windows setup. That used to rotate around the cameras and would bring to the front specific cameras when motion was detected.

After a bit of searching I found xlib_shm http://www.zoneminder.com/wiki/index.php/Xlib_shm
Although it says it only works with 1.22.3, I’m currently running 1.24.2 on Ubuntu 11.04.

So after installing it and trying to run via the command line “./xlib_shm -m 11 -m 12 -m 3 -m 13 -m 14 -c 2 -k 0x7a6d0000” it complained about not being able to open the display. of course I’m running it from a terminal on the server itself, but gdm is already lauched and I’d alt+ctrl+f1’d out of it. So I stopped gdm with “/etc/init.d/gdm stop” and tried again, hey presto it runs.

So now how do I get this to run on boot?

One option is to have gdm auto login and then auto run xlib_shm. I had a few problems with this approach, First the login counter, it takes a few seconds to auto login and kick in xlib_shm and it’s just a bit untidy. Second xlib_shm launches in a window doesn’t look very pretty and I’d prefer full screen. Third was the mouse pointer happily sat smack in the middle of the screen, not a huge problem for a 4 split, but single screen or 9 and it’s in the way of the main detail on the picture.

So kill that idea, I want it straight in no hassle as quick as it can. So the best thing I could come up with was stop gdm launching and make a new script to startup xlib_shm. unfortunately this approach seemed to hit it’s own problems. killing gdm wasn’t as easy as it should have been because of changes in the way it’s now started in Ubuntu 11.04, and adding xlib_shm into the init.d was proving unreliable sometimes it beat gdm and grabbed the screen othertimes it didn’t.

So I had to find another way, and my answer was to try and figure out how X was started.

I ended up editing ‘/etc/X11/xinit/xinitrc’
commenting out the ‘/etc/X11/Xsession’ line and adding ‘exec /usr/sbin/xlib_shm -m 11 -m 12 -m 3 -m 13 -m 14 -m 15 -m 18 -m 19 -m 20 -c 3 -k 0x7a6d0000’ to the bottom.
now starting X with “startx” fires up xlib_shm full screen with a 9 split.
Problem hit though with the screen turning itself off.
Found out that setting a couple of dpms options then starting x would solve this so added the following into the xinitrc file after the exec line

xset s off &
xset -dpms &

Now to get this starting with the boot. Solution edit the ‘/etc/init.d/zoneminder’
Within the start section add ‘startx &’ and in the stop section add ‘killall -9 xlib_shm’.
Now on each reboot X starts up with xlib_shm and the screen doesn’t shutoff at all.

I’ve no doubt theres bits in here that I’ve missed, I got this all running months ago. When I did the same similar on a 2nd system, it was pretty straight forward and similar but I did hit another issue.
For some reason xlib_shm would occasionally crash out and return to the command prompt. However between this change and installing that I’d setup scripts to pull a few webcams and display them on a zoneminder monitor rotating the image. As that script fires a rotate every 10 seconds, my answer to the crashing was to add a check into this script to see if xlib_shm was running and if not to startx &.

So that’s the brief outline of me getting xlib_shm to work with Zoneminder 1.24.2 on Ubuntu 11.04.

One thing to note, restarting zoneminder via the interface is not the same as running /etc/init.d/zoneminder restart. Because of this via the interface does not restart xlib_shm. This causes problem (mainly when I was testing) because the zoneminder feeds would shutdown and xlib_shm would then crash. I got used to only restarting via the commendline, but the script mentioned above would counter this problem should xlib_shm crash now.