Ajax woes

<updated see bottom>

Well it’s been a while since my last post, I’ve been working on a website that’s been taking up alot of my time and giving me a big headache.
Basically I started with a one page template that had a nice clean look. Unfortunately since I was last working on webpages, CSS has sprung up and taken over. So for every little change I had to copy an existing bit of code then change it, then refer to the CSS file and after alot of playing and going back and forth I finally got the template file looking how I needed it to to progress.

Now that I have my template page, I took that and created a few basic pages. To get the pages a bit more functional I changed the pages over to PHP. This worked well and now I had a whole login/register system, some basic pages displaying info, and some pages that interacted with a database.

All this was starting to come together and look good.
Things are never supposed to run smoothly though are they. The more I used the pages the more I thought that they need to load more smoothly, rather than going to a new page request for everything I needed to start using ajax.

So instead of finishing off the site as I was going, I decided to rewrite what I already had done to fit into an ajax framework.
Changing each of the tabs to pull the relevant page into the existing page was pretty simple. I got the tabs all working, highlighting the correct tab and changing along the way perfect.
Now the tricky bit, alot of the pages use forms and scripts. So I decided to try tackling the scripts first.
Basically I dont want to load all the script on the first load of the page as it’ll load up a lot of scripts that wont even be used. So I put the script in the page that gets pulled by ajax. seems like a simple solution.
However if the script isn’t present on page load the script isn’t recognised. ok I’ll stick the scripts to one side and work on the forms. oh no a problem, to use the forms inside ajax it needs a script to run.
So I fell back and decided just stick the script in the head of the first file at least it’ll get it all working. Nope that didn’t work either. As the elements aren’t present on the page when the script is first loaded the submit intercept stuff isn’t getting applied.

So I’ve started looking for a way to put scripts into ajax pages that will be relevant scripts for that page. I think this would be pretty common, but after hours of looking there is stuff out there mentioning it, but nothing explicitly saying how to have an initial script running to interpret other scripts inside an ajax page.

So there’s my ramblings for the night and the reason’s I stopped posting stuff for a few weeks. I don’t want to post details about the site just yet as it’s a complete work in progress, I have alot of the backend working but the entire site is currently ripped apart and non-functional. I know all the bits I want to get it to do, just not quite how to do them. I’ll solve it eventually, you’d have thought from the years working within IT I’d have loads of development contacts, unfortunately I have quite a few web designers, but none that do php/code development.

<update>
ok so after alot of searching around I found alot of stuff saying anything in a script should get interpreted as it’s loaded but that wasn’t happening for me. more and more searching I found a different way of loading the content instead of the httpxml or xmlhttp whatever it was, the following works for me:-

function loadajax(page,tab)
{
var page,tab;
$.ajax (
{
method: "get",
url: page,
dataType: "html",
success: function( strHTML ) {
$( "#content" ).html( strHTML );
}
}
);
}
</script>

Hey presto as the page gets loaded the script inside is now run too. Now to rework all the forms and stuff on the site.
<thought>I really do need to look at how you put script on a blog properly too.</thought>

Ubuntu Upstart

Well I posted yesterday about handing over the control of my webcams script and the Zoneminder viewer xlib_shm to a daemon manager, as it seemed the right way to go and I noticed earlier tonight that my 2nd machine has again crashed out on the xlib_shm.
First I did a search for the daemontools to get an idea of what it’s like, but alot of the postings were older. So then decided to do an ‘alternatives’ search which brought up a few results, only one of which mentioned upstart in ubuntu, hey presto! I’ve heard upstart before but never looked at it (at least I thought).
Did a bit of looking around and yep I remember reading that Ubuntu is moving away from the init.d scripts a while ago just never put it together with the word upstart.
Basically there’s new files in /etc/init/ which is what upstart looks at and their the configs for each service that’s starting on the machine. Funnily enough I remembered then where why I’d done some searching previously on /etc/init/ and it was for gdm.conf because the gdm kept launching even though I’d removed it with the update-rc.d -f gdm remove. I was trying to kill it off because it was grabbing the screen before xlib_shm could.
Anyway that’s a bit of a side track. Looking at some of the configs there and the FAQ it looks quiet powerful and will do exactly what I need.

I was going to jump straight in and make a new config for the webcams stuff, but then noticed you can have a service start on another, well that should be good I only want webcams starting if zoneminder is running so let’s check that config, oh no there isn’t one yet. After 2 mins of searching for a config some clever person has already made I’m going to call it a night and have a proper look tomorrow. but it got me thinking of moving more over to Upstart like my asterisk stuff that will also need and depend on dahdi.
You can also depend on having a filesystem and networking. I can see the filesystem being used for the zoneminder config and both being used for the webcams config. So I’m really looking forward to getting into this a bit too. There’s only a handful of occasions when asterisk has shutdown so not a huge problem, but when it has I did once get left with no phone line for 3 days cause I didn’t know it wasn’t running (obviously enjoying the peace and quiet too much). To solve this I have Nagios monitoring now and using event handlers to restart via the init.d script but handing over to Upstart really does seem the way forward. I’d still keep Nagios monitoring to ensure it’s running I can’t live without my nagios watching over everything.

Home Temperature Sensors

Not going to go into the technical stuff on this post, far too tired. but wanted to try posting up an image to see if I had to host elsewhere or if it just goes, and thought this would be a good try.

A few years back I decided I wanted to put some temperature sensors in the house as we just never seem to be able to get all the rooms at a decent temp at the same time. You can be boiling hot in the one and freezing in another. Not much has changed with that respect but I do have more of an idea of what’s going on day by day and longer term. It’s really quite interesting. As I posted yesterday about the Raspberry PI board, one of my many wishes was to somehow get an interface to take over control of the central heating. It should be pretty straight forward, the biggest downside I don’t want to be screamed at having no heating if the computer freezes so need to build in over-rides.
Anyways, that’s the grand plan but below is a simple image I knocked together a few years back that uses php to colour in each of the rooms and write the current temp

From inside my network you can click on each of the rooms to get some stats. and the little blue dot is the set point I’d programmed up for when I do sort out the controller.

Due to my bad coding at the time (first real time I’ve done php from scratch not just had to change someone elses code) and the fact that I keep everything (database is now at 860k entries) the detail page is quite slow to load. I should really rewrite it to store upto date stats elsewhere and then hourly just update that table with the totals. I may get around to it.
I did used to use MRTG to also connect into the database and pull the stats to have some nice graphs. This worked really well until I changed alot of my network kit that MRTG was polling, and I shut it down to give it a full clean up.

Anyway to get all this stuff going I use the OneWire Temp sensor DS18S20 I think it has a + on some of the chips I use but not all. I bought them online from a website with sheep in the title something like sheepwalk sheepcottage, can’t remember I’ll dig that out when I’m going through it a bit more. They were decent prices then maybe not now. but I received them very quick and they were helpful.

A thing to note that I’ll go into when I put the scripts up but this will prompt me, I do get alot of nulls coming back in the results. I know this is due to my mainly star configuration of the sensors and the fact that I dont provide power to the chips (bad idea when running a few). I did it because I was being lazy and using the 2 spare cables I had in the wire going to each of the alarm sensors for each room. It made sense at the time and it actually works really well as I didn’t have to run new cables and the temp sensor is hidden by the alarm sensor itself. Being close to the top of the ceiling I did expect it to skew the temp results (heat rises) but it actually seems to be pretty close when I compared them to an independent weather station sensor I used to see.

Oh and the green outside the house in the pic is also programmed to be changed. it’s green showing sensor fault (there isn’t one) I always planned to stick a sensor outside front and back and take an average reading for the colour. just havn’t got around to running a cable outside for that bit. So many things to do so much left unfinished 🙂

Zoneminder Webcams 2.0 Part 2

I said I’d put up the script I’m using so here it is. Couple of notes first.
I know PHP isn’t particularly the best language to use for scripts, but I’ve been using it for work within a website and since my head was already in PHP mode it seemed easier to just keep going and get the idea down and running than be typing stuff wrong all the time and have to keep correcting it to get each bit working.
I call this script to start running as zoneminder loads from the init.d/zoneminder file, exactly the same way I started the last one. Something I did hit though is if PHP tries to output and there’s no console it crashes out. So the diagnostic setting should be off if it’s live and only used if your running it from command line for testing. For some reason running it from command line with & to background it starts it off but on trying to continue in the terminal it stops the process. I’m sure there’s a perfectly good explanation for this like it’s passing the input into the php script which isn’t to output and falls over, but I don’t need to run it from command line unless testing so this bit doesn’t bother me.

Requirements:-
New folder, /var/cache/zoneminder/webcams, /var/cache/zoneminder/webcams/originals, /var/cache/zoneminder/webcams/resized
This also requires the ‘convert’ command (same as the previous but I forgot to note that). if you try to run convert under unbuntu it’ll tell you what to do to install it if you dont have it.

edit a new file ~/webcams2.php and paste the following:-

#!/usr/bin/php
<?php
$DIAG = 2;
$LOGPATH = "/tmp/webcams.log";
$host = "127.0.0.1";
$user = "root";
$pass = "";
$db   = "zm";

$directory = "/var/cache/zoneminder/webcams/";
$dblastid = "0";
$mainpull = "0";
$nextrotate = "10";
$nextpull = "60";
$autoadjustrefresh = FALSE;

$ms = mysql_pconnect($host, $user, $pass);
if ( !$ms )
        { echo ""; }

mysql_select_db($db);
//connection complete.

chdir ( $directory );

if ($DIAG > 0) { $LogFile = fopen($LOGPATH, 'a') or die("can't open file"); };
if ($DIAG > 0) { fwrite($LogFile, "Starting Logfile at : " . time(). " ...n"); };

function RefreshAllImages() {
 global $directory, $dblastid, $nextpull,$DIAG,$LogFile;
 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);

 if (!$r) { echo ("Problemsr"); } else {
  for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
    $filenamepath = $directory . "originals/" . mysql_result($r,$rows,'Monitor') . mysql_result($r,$rows,'ID') . ".jpg";
    $filename = str_replace($directory."originals/","",$filenamepath);
    $filetime = filemtime($filenamepath);
    if ($DIAG >= 2) { fwrite($LogFile, "Pulling File : " . $filenamepath . "n"); };
// old command    $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
    $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
    if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
    exec($cmd);
    clearstatcache(TRUE, $filenamepath);
    if ( (filesize($filenamepath)!="0") && ($filetime != filemtime($filenamepath)) ) {
      if ($DIAG >= 2) { fwrite($LogFile, "Converting File : " . filectime($filenamepath) . $filenamepath . "n"); };
      $cmd = "convert " . $filenamepath . " -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $filename;
      exec($cmd);
      touch ( $directory . "resized/" . $filename, filemtime($filenamepath) );
    };
  };
 $dblastid = mysql_result($r,$rows-1,'ID');
 };
if ($DIAG >= 2) { fwrite($LogFile, "Finished Pulling All Files!n"); };
$nextpull = (time() + 60);
};

function RefreshImages() {
 global $directory, $dblastid, $nextpull, $autoadjustrefresh, $DIAG, $LogFile;

 clearstatcache();
 if ($DIAG >= 2) { fwrite($LogFile, "Refreshing Images...n"); };

 $q = "SELECT * from WebCams where Enabled='1'";
 $r = mysql_query($q);
 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
  $start = mysql_result($r,$rows,'Start');
  $stop = mysql_result($r,$rows,'Stop');
  if ($DIAG >= 2) { fwrite($LogFile, "Start : $start. Stop : $stop. Time : " . date("H:i:s") . ".n"); };
  if ( ( ($start < $stop) && ($start < date("H:i:s")) && ($stop > date("H:i:s")) ) || ( ($start > $stop) && ( (($start < date("H:i:s")) && ($stop < date("H:i:s"))) || (($start > date("H:i:s")) && ($stop > date("H:i:s"))) ) ) ) {
   if ($DIAG >= 2) { fwrite($LogFile, "In timeframe running checks...n"); };
   $row_filename = mysql_result($r,$rows,'Monitor').mysql_result($r,$rows,'ID').".jpg";
   $row_id = mysql_result($r,$rows,'ID');
   $row_refresh = time() - mysql_result($r,$rows,'Refresh');
   $filenamepath = $directory."originals/".$row_filename;
   $filectime = filectime($filenamepath);
   $filemtime = filemtime($filenamepath);
   if ( (($filemtime > time() - 3600) && ($filectime < $row_refresh)) || (!file_exists($filenamepath)) ) {
     if ($DIAG >= 2) { fwrite($LogFile, "Pulling New File : " . filectime($filenamepath) . $filenamepath . "n"); };
// old command     $cmd = "wget -q -t 2 -T 5 "" . mysql_result($r,$rows,'URL') . "" -O "" . $filenamepath . """;
     $cmd = "curl -q -m 10 -R --retry 1 -s -f --url "" . mysql_result($r,$rows,'URL') . "" -o "" . $filenamepath . """;
     if ($DIAG >= 2) { fwrite($LogFile, "Using Command : " . $cmd . "n"); };
     exec($cmd);
     clearstatcache(TRUE, $filenamepath);
     if (( filemtime($filenamepath) > $filemtime ) && (filesize($filenamepath)!="0") ) {
       if ( $autoadjustrefresh == TRUE ) {
         if ($DIAG >= 2) { fwrite($LogFile, "Adjusting refresh time.n Calculating difference...n"); };
         $timedifference = filemtime($filenamepath) - $filemtime;
         if ($DIAG >= 2) { fwrite($LogFile, "Difference : " . $timedifference . "n"); };
         if ( ($timedifference > (mysql_result($r,$rows,'Refresh')+30) || $timedifference < (mysql_result($r,$rows,'Refresh')-30)) && ($timedifference < 900)) {
           if ($DIAG >= 2) { fwrite($LogFile, "Updating Database Refresh timer.n"); };
           $q2="UPDATE WebCams set Refresh='" . ($timedifference) . "' WHERE ID='$row_id'";
           $r2 = mysql_query($q2);
         };
       };
       if ($DIAG >= 2) { fwrite($LogFile, "Attempting resize : " . filemtime($filenamepath) . $filenamepath . "n"); };
       $cmd = "convert $filenamepath -resize 640x480! -pointsize 20 -fill yellow -draw 'text 10, 20 "" . mysql_result($r,$rows,'Name') . ""' " . $directory . "resized/" . $row_filename;
       exec($cmd);
       touch ( $directory . "resized/" . $row_filename, filemtime($filenamepath) );
     } else {
       if ($DIAG >= 2){ fwrite($LogFile, "$row_filename Not Newer : " . filemtime($filenamepath) . " <= " . $filemtime . " Ignoringn"); };
     };
   };
  };
 };
$nextpull = (time() + 60);
};

function RotateImages() {
 global $directory, $nextrotate,$DIAG,$LogFile;
 $newimageset = FALSE;
 $imagefound = FALSE;

 clearstatcache();

 $q = "SELECT distinct(Monitor) from WebCams";
 $r = mysql_query($q);

 for ($rows = 0; $rows < mysql_num_rows($r); $rows++) {
   if ($DIAG >= 1){ fwrite($LogFile, "Rotating Monitor " . mysql_result($r,$rows,'Monitor') . "n"); };
   if ($DIAG >= 1){ fwrite($LogFile, "Checking for New Files ...n"); };
   $newimageset = FALSE;
   $imagefound = FALSE;
   if (mysql_result($r,$rows,'Monitor') != 'C'){
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "  " . $filenamepath ); };
       if (filectime($filenamepath) > (time()-15) ) {
         if ($DIAG >= 1){ fwrite($LogFile, "n New file Found linking ... n"); };
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };

   if ($DIAG >= 1){ fwrite($LogFile, "Finished lookin for newer files.n"); };

   if ( $newimageset == FALSE) {
     $rotatefile = $directory."rotate".mysql_result($r,$rows,'Monitor').".jpg";
     if (file_exists($rotatefile) ) {
       if ($DIAG >= 1){ fwrite($LogFile, "nLooking for Link File : $rotatefile ...n"); };
       $currentfile = readlink($rotatefile);
       if ($DIAG >= 1){ fwrite($LogFile, "Current : " . $currentfile . "n"); };
       foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
         if ( $filenamepath == $currentfile ) {
           if($DIAG >= 1){ fwrite($LogFile, "Found Correct entry.n"); };
           $imagefound = TRUE;
         } elseif (($imagefound) && (!$newimageset)) {
           if ($DIAG >= 1){ fwrite($LogFile, "Checking next Image...n"); };
           if (filectime($filenamepath) > (time() - 1800) ) {
             if ($DIAG >= 1){ fwrite($LogFile, "Next Suitable Image " . $filenamepath . " Linking...n"); };
             exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
             $newimageset = TRUE;
           };
         };
       };
     };
   };

   if ($newimageset == FALSE) {
     if ($DIAG >= 1){ fwrite($LogFile, "Image still not set, Falling back to load first suitablen"); };
     foreach (glob($directory."resized/".mysql_result($r,$rows,'Monitor')."*.jpg") as $filenamepath) {
       if ($DIAG >= 1){ fwrite($LogFile, "Checking File " . $filenamepath . "n"); };
       if (filectime($filenamepath) > (time() - 1800) && ($newimageset == FALSE) ) {
         exec( "ln -sf " . $filenamepath . " " . $directory . "rotate" . mysql_result($r,$rows,'Monitor') . ".jpg" );
         $newimageset = TRUE;
       };
     };
   };
 };
$nextrotate = (time() + 10);
};

$pulled = 0;
for (;;) {
 if (time() > $nextpull) {
  if ($pulled == 0 or $pulled == 60) {
   RefreshAllImages();
   $pulled = 1;
  } else {
   RefreshImages();
   $pulled++;
  };
 };
 if (time() > $nextrotate) {
  RotateImages();
 };
 sleep(2);
};
?>


Save the file then chmod +x ~/webcams2.php

You will likely need to change the database connection info, it’s not a good idea to connect to the database as root and I’ve removed my password.
An option I would leave off at the moment is ‘autoadjustrefresh’ This is meant to compare the times on the files it pulls and calculate the difference since the last pull.
The theory behind this is as follows:
Database is set to pull a new image every 300 seconds (5 mins). but the webcam itself is only updated every 15 mins. Now obviously you can sit and work this out yourself by just watching, but the files I’ve been pulling are generally modified timestamped with the servers time, so I can use this to calc the difference when a new image is obtained. Unfortunately this seems to go wrong and has made some of my feed not update for 30 min intervals even though I know the feed is refreshed every second and I’ve told it to pull every 120 secs. 
I figured it was adding a few seconds here and there for delays but it’s not that.
Another issue is on the rotating side it’s supposed to look if there’s a new image just come in (in the last 40 secs) if so display this as a priority. This theory again is sound, until I realised yesterday that some of the servers clocks are out by minutes, this means when I say the last 40 secs the file is already way beyond that so it doesn’t get shown as a priority. I have a fixed in mind for this though, which revolves around using the resized files creation time as a check as I only resize if a new image was pulled anyway.
Well that’s the script, to pickup the image in zoneminder same as the previous script I just have a new camera looking at a file /var/cache/zoneminder/webcams/rotateA.jpg which is set at 1fps (doesn’t need to be fastas it’s pretty static)
I much prefer this script over the last as it reads from the database and puts the camera name on the image, which is great for some of the motorway cams that give you no indication where they are.
it also means adding a new entry is straight forward and get’s picked up quickly.
It still needs some work so I’ll post changes up as and when. If anyone is reading and has suggestions drop me a comment.
I’ve been considering getting stuff like this to be run by a daemon manager that will restart it if it fails etc. may do that at some point, but for now starting it with zoneminder works for me. On the 2nd box I put it on I’m still running into some issues where xlib_shm is crashing, and I haven’t built any checks for that into this, that’s one reason I’m considering a daemon manager as xlib_shm and the webcams2.php script shouldn’t need to speak or check each other it was just a dirty way of keeping it running. Think about what maybe happening as I’m typing though, that server only outputs stuff from the webcams feed and I noticed with the previous script if the image was 0 bytes it obvious can’t convert. giving a dud image to zoneminder is probably enough for it to restart that monitor and as it’s the only one xlib_shm would probably fall over as it can’t read the memory zoneminder is using. kinda makes sense as my display occasionally come up blank or part converted image if it’s in the middle of running, but as I have live monitors zoneminder wouldn’t stop the lot. Will have to look into that a bit more, but handing to a proper daemon manager should be the way to go anyway it can handle crashes then.
Note: I need to do better formatting for code etc.

Blog update.

ok, just run through enabling comments on most of my posts. Wondered why I’d never seen the option when posting I must have turned it off globally a good while back. Unfortunately when enabling it there’s no quick way I could see to turn it on then for each post so had to edit, set option and save. Just hope it hasn’t screwed with the dates on the posts now.

Apparently I’ve also turned on adsense for this blog, I didn’t really want it linked to this I wanted it link to another site. but if I get it on here first I can checkout how it works and hopefully move it over later even if it means cancelling and re-registering my email address to the other site. I did read something on it about no multiple accounts per site. hoping it still works the other way.