ramblings – J.Lo's Tech Blog

Raspberry PI 4 Boot from USB HDD not working GPT

This is more of a note for me than a guide or help for anyone else.

At present the PI4 does NOT support boot from USB, but you can work around this like the old days by using an SD card for the /boot and setting this to load the root from USB Stick/HDD.

However, this only appears to work for me using MBR partition type and just hangs when using GPT. This may be because I’m also encrypting the disk and using headless network boot. But that side does work and the disk/paritions are decrypt and list in /dev/disk/by-label but the OS just will NOT boot for me. Copying everything to the exact same partition layout and labels under MBR and it just works.

It seems a weird problem, as the SD card that’s actually booting and has the kernel is MBR and I don’t see why it would then care, the kernel has to be aware of GPT and as I said lists everything correctly. I can even decrypt and mount during the dropbear network boot bit, but the OS just wont boot. Workaround at present is to place the OS on it’s own drive/stick and use GPT on bigger disks purely for data. I’ve only found this while trying to use a 5Tb disk for both OS and Data. the 2Tb disk I can use MBR on without loosing space so the workaround isn’t really needed, just use MBR.

New to Ansible

So If you read my last post (it was really long sorry), you’ll see right at the end the current deployment. I had tried a few managers to be able to deploy/scale the whole system, but it really overcomplicated the whole thing. Chef looked really good (I can’t remember the other one), but it was problematic and just didn’t suit.

Instead I kept with the scripts I had written for the time being. They are in no way good enough to share as they are very customised to my setup but they achieve what I need. However to run them takes quite a bit of initial manual work.

So what do I need from a system:-

It has to just work, not go installing stuff it depends on to run.
It has to be able to split the setup into an initial and running level.
It has to be able to be told easily about a new server and what role it will be, then do the initial setup and onto the relevant running level.
It MUST be simple to use and understand. I’m getting a bit sick of having to read through how to configure weird stuff because someone decided to do things completely differently to how you’d expect it to work.
It must have very low resource requirements. I want to run this from a management droplet that already runs nagios, so it can’t sit there just eating resources while it has nothing to do.

I saw a very quick video on Ansible with someone using it for Raspberry PI’s (something else I have far too many of) and thought right away I should look into that. So here I am. I’m going to do some testing with Ansible.

Initially all I’m looking for it to do is handle the initial setup that I already have scripts for.

Create a new user, add it to the sudo group, set a password and copy the SSH keys.
Reconfigure sshd to deny root connections.
Set the server’s timezone.
Set the keypad option in nanorc.
Set the .bashrc for root to use a red prompt.
Install some basic packages such as screen and htop
Create a swapfile
Install and configure NTP
Setup IPTables (I dont use UFW, I prefer to deploy an iptables file and have this restored when the loopback interface is loaded).

I’ve had a quick look on DigitalOcean community (I love the resources there), but the stuff about using Ansible seems a little more throw it all in one file rather than properly split them out like I saw in the video. I think splitting out each task is a must to be able to understand what’s going on and making changes.

The video I’m referring to is https://www.youtube.com/watch?v=ZNB1at8mJWY

So that’s the start, let’s get going and see what I can screw up

Part 1 is Here

How I started using WordPress

I should make it clear from the outset, this post isn’t going to be solving anything. I’ve spent about 3 days working on stuff and this is just bugging me.

Let’s go back to around this time last year. A couple of friends and I were working on how to get some money back from a facebook page one of them had setup a few years prior. I had been working with him on it since a few weeks after he set it up, and we’d pushed around ideas to sell some merchandise alongside it a few times, but never really got anywhere.

He had made a rash decision one night to use an ‘online website creation’ provider to get a site online. From the start I hated it. Not the idea, I think it was about time to get our own site running, but he spent a few weeks tweaking about 8 pages to look really good, using their WYSIWYG editor. Then wanted me to change a few things in code that wasn’t right. It was an absolute nightmare! I can’t remember the name of the site but I think it had a W E X somewhere in the title.

It was a paid “solution”, and I think cost about £40 per month by the time he’d added a mailing list option to capture email addresses (not actually handle any emails) and a few other bits.

Coming from more of an IT position, my main concern was around load/spikes being handled. There was very little information about how well they could handle this (and I think we found the reason why). We finally put this live posting the link to about 40k followers.

Watching the nice google analytics (that I had to add to each page, because they didn’t have a drop in the tracking code option), within seconds we hit around 150 hits per second. This continued for a few hours, but 2 problems became apparent:

The site was struggling, and we would probably have been hitting a higher number. We were getting positive feedback and people understood it was busy, but I still wasn’t happy we’re paying for them to handle this and it’s just not.
And this really hits onto point 1. He’d setup a site that was pretty static! There was nothing to get people coming back for more. Yes there was a news page that we could update, but other than the mailing list form there was nothing interactive. (So hitting point 1, static content should really have been able to be handled 100x better).

Anyway we took that for what it was, a basic site with a bit of info and something to get us started.

We already had a ‘Shop coming soon’ page, so the next thing we were trying to figure out was what are we going to sell and how?

Initially we thought t-shirts, and started looking at some of the ways we could do this. 2 main providers seemed to jump out zazzle and cottonpress (I think, it was a while ago). While both had some good offerings, neither really grabbed us. I can’t remember which, but one of them deleted an image we uploaded for copyright (it was our logo, and we had it plastered everywhere and the account was signed up with a domain name using the same logo), but they wanted us to fill in and snailmail/fax some copyright forms and reupload the logo. Considering we were only seeing what we could do at the time, we decided to drop them as an option. If we have to jump through these hoops with everything we do, we’ll spend more time filling in their paperwork than anything else.

Time went on, visits to the site died down (did I mention lack of content/interactivity), and we still hadn’t sorted out products, a store, a business.

We continued with the facebook page and still poked around ideas on how to get a site/shop running. I spent a good few weeks working with oscommerce (I’d previously used it a little for another project idea, but it never went live) and finally had something to show, a semi working shop front (it had no products).

We discussed that neither of us really had a clue about setting up a proper business. I’m all IT and have no interest in writing business plans or doing business meetings (I should mention that a previous role I was an IT Manager and regularly had to be part of “grown up” meetings, I’m a Tech, I hate people, I hate meetings, give me something broken – I’ll fix it, give me a problem – I’ll work out a solution. But in NO WAY do I want to be taking part in any more business meetings).

A few months later I was helping him move. Another friend of his was also helping. I’d met him once before but didn’t get chatting then. He mentioned he was in his last year of university and was studying business and finance! Just how he hadn’t thought of this before I dont know, but instantly we knew he was coming on board 🙂

We spent about 3 hours in McDonalds discussing what we had, what we’d like to do, and just how crap we’ve been so far. Within this conversation we said about selling t-shirts/mugs/bags etc. Just like a genii out of a lamp our 3rd comes out with ‘oh my almost father in-law does printing stuff, I know he does mugs. Shall I speak to him?’ Just like a match made in heaven, we suddenly had our missing piece! someone who should have more of an idea on the business side (or at least know someone to ask) and connections to a printers for the kind of stuff we want to sell. You just couldn’t make it up, he’d known this guy for a few years and never thought to asking him about business stuff.

Things started moving forward, slowly at first but at least they were moving. We met up with “almost” father in-law and went through some designs and processes. We setup a business. I continued work on the shop website and we took down the other that was costing too much and not really doing anything.

In around October time we were set. Nothing spectacular, about 9 mugs and a few t-shirts. The mugs would be the easiest, we just send the order to “almost” and he takes care of printing, packaging and sending. The t-shirts would be a little different as we’d have to get a template made for them, and couldn’t afford the cost until we had some orders in.

We launched, I had tried to over-spec the server(s) but in itself this was tricky. There were no real stats on how well oscommerce could perform on certain hardware. And scalable VPS’s such as DigitalOcean’s current system just didn’t exist. Scaling would mean taking a new 30 day server and moving everything over to it. Certainly not a 5 min job, and definitely not something to start an hour after we’ve launched. We’ll just have to bite the bullet and see.

My memory of launch night is fuzzy to say the least. I think I’d been working 36-48 hours trying to finish stuff off. I had a big list of checks and can’t remember doing half of them.

Our page audience had grown to about 70k, so I was very nervous. We launched the shop and watched. ping, email – it’s an order, ping, another, ping another. It was working. I have to say the server(s) held up pretty well. It wasn’t without problems, we did start seeing the site timing out on new connections for about 10 mins. but a swift kick of apache sorted that and it didn’t cause a problem again.

Finally we were running. The feedback again was good. We had a bunch of concerns like

Will the system work
Will the servers(s) hold up
What happens if it goes mental and we sell a thousand mugs
Can we really do this

I think all in all it went well. We could have done better, but it also could have been a lot worse. We ran with oscommerce for a few months. Shortly after launching the shop I had a discussion on just how were we going to get facebook and the shop incorporated? There was no obvious answer, then we hit a problem. Once of the posts to facebook got reported (it’s a humorous page, and we only every post stuff sent in to us), this showed us just how much we’re reliant on facebook. Suddenly we’re all logged out and the poster was blocked for 24 hours. Luckily facebook pretty much left the page alone (just delete the one post), so we played on it that one of our admins was in the dog house for an earlier post. But it still didn’t take long to realise if facebook wanted they can delete the page at a whim and we’ve suddenly lost all our content and fans!

This just didn’t sit right with me, and I started looking at how the hell to get a backup of OUR page and content. There was nothing. So I started looking at how can we do things differently, enter WORDPRESS.

I’d seen this name floating about for the last few months, but never really saw the point in using it. I dont blog, we dont blog, so what’s the point? (I’m still not entirely sure I understand the point) but it’s close to one of the best things I’ve spent weeks fiddling with.

I’d installed wordpress on our VPS for me to have a look around, it still wasn’t a site that we could really use, but as a CMS maybe I can find a way to connect to facebook and backup our stuff. There must be people who do this right? WRONG. There’s loads of plugins for wordpress & facebook, but I’ve only ever found 1 that takes your page and puts it as posts. To make matters worse, it’s flakey as f**k hadn’t been worked on in god knows how long, and of the very few comments in the code their in Chinese.

Now I would never describe myself as a coder. I’ve used Delphi and VB for writing some functional programs in the past, and had to program a few in VB.net when I was IT Manager (the old problem/solution thing), I could also write some ASP and PHP, really most of my stuff was drag and drop boxes and program them up for stuff. I did quite a bit of databases stuff within them, but that was it. There was absolutely no such thing as using classes (I think even think they existed). But as part of my job I had a dev team who did develop in PHP and VB.net, and they were always amazed when trying to tell me how something couldn’t work, that I could not only follow along but tell them why they were wrong and on several occasions when something broke could actually read their code and work out (normally the simple thing) a temporary fix.

And so it begins I now have no dev team, a bunch of php code and classes that really didn’t make much sense to me. Bit by bit I managed to work out what each bit was doing, then moved on to changing it so that it would run for us. I know it will seem like simple stuff (especially looking back), but things like:

Changing a hard coded loop to only pull 10 facebook posts, to take a limit from a setup interface where you can specify how many to pull.
Adding in Date ranges to pull from and to.
Improving the cron job, so it look from the time of the lastest post it was + 1 second.
Downloading any attached image and saving it to the server (huge accomplishment).
Changing the posts content that gets published and updating links back to the post it just posted.

There’s load of stuff I’ve had to do to this plugin to get it working, let alone better and working for us. Eventually I’d finished (your never really finished, I have a list of new changes to get to sometime). After running it on a fresh wordpress install, we suddenly have a complete backup of our facebook page, around 9k posts and images, all sat in wordpress. and what’s more automatically grabbing new stuff 🙂

I showed off my new achievement, personally I dont think it was appreciated just how much time and effort I had put into this. but it went down really well. We now had a blog! we now had a blog alongside our store and it was really starting to come together.

Over the next few weeks I kept working on improving the blog while managing the shop. Then suddenly a new disaster, our server had for some reason gone offline. Trying to connect via the backup terminal access just gave me a blank screen, something was wrong and I couldn’t get access to see what. To make matters worse, our provider had very nicely decided to cut back on it’s 24/7 support, and now only operated 8-8. At around midnight, it’s not exactly what you want to be finding out that the support has changed and no-one bothered to tell you. The ONLY thing I could do was email them, and hope someone picked it up soon. They didn’t! I spent a good few hours trying everything I could think of to get hold of someone or find a way to the console, but nothing. This had the effect of making me sleep through my alarm at 8am, but I woke at 9am and called them. After a few choice words, I was assured the tech team would look at it right away. I was so tired I fell back to sleep and woke again about mid day. The first thing I did was open our site, or I should say attempted to! it was still down!! Another call, more choice words, and me advising them I’m not going anywhere and if they cut me off I’ll just keep calling until I can speak to a “Tech”, then explaining the problem and what I had tried to tier 1 monkey, quickly got me escalated. I couldn’t stop laughing when I finally did get to tier 2, their tier 1 had placed me on hold to get someone then come back to me and said ‘I need you to take this call, this bloke knows what the f**k he’s on about, I’d put him to tier 3 but I can’t direct transfer’ to which I replied ‘Yes and I know how to work a phone system, Line 1 is the customer, you should be on Line 2 for that conversation’ 🙂 I have to be honest just that mistake made my day. Tier 1, 2 and the manager that called me back an hour later were mortified, but as I explained to him I’ve been a senior tech on phone support, I’ve been an IT manager, I’m guessing I hit a newish person and scared the crap out of them. I only care in getting this back online. To be fair to Tier 2, I was connected to the console while he was apologising. (This part really could have had it’s own post).

Anyway getting over that failure I started looking for another VPS provider, I had no problem with their VPS and generally it was a very stable system, but 8-8 support with no out of hours we’re really f**ked option, forced my hand. It had gone down very badly with the others that this had costs us money and there was no way I could argue it as I agreed the situation was crap.

I found another provider and started moving stuff, but it just wasn’t right. It was actually a previous colleagues company, but something just wasn’t right. So I kept looking. Then I found Digital Ocean, initially I started using them to test some wordpress plugins, but I loved that I could very quickly bring up new servers in a matter of minutes. This surely has to be better than waiting hours. And it was. Testing was going well, so I started moving everything over. Everything just worked, and where I had to contact their support for a few little things (1 account related can’t remember the others), I had a reply very quickly sometimes within minutes other times within 30 mins. I couldn’t fault their support and I wasn’t bounced around, they knew exactly what I needed and sorted it.

So here we have a medium spec’d Digital Ocean server, running our WordPress and OsCommerce solutions and handling both pretty well.

But being one to never settle, I kept tweaking stuff and looking at out options. I setup another server (droplet) for testing, another wordpress install later and I’m going through trying out the ecommerce plugins. I was blown away with WooCommerce! yes OsCommerce worked for us. and yes I had put in quite a bit of time customising it and getting it to work with our processes. but the whole feel of the interface was crap. Woocommerce was like a breathe of fresh air. It had a bunch of functionality, there’s loads more plugins, it’s far easier to customise, it works from the wordpress themes, and fits right in with our blog and doesn’t look disjointed.

I proposed we move over to this and it went down well. Well enough infact the the others wanted to get more involved, we spent weeks working on changes to the theme (that’d we’d paid about $50 for), I moved the shop over and made it live without telling our facebook audience. We started getting some sales via Woocommerce, and it was obvious that this just integrated well.

We were going to have a relaunch to show off the new blog and store, I think I managed to p*ss the others off, when WordPress brought in a new standard theme that worked even better with Woocommerce and I changed to it to show them. It was obvious that it did and we should stick with it, but it also meant the last few weeks of customising was wiped out (and they still bring up the time I wiped out a few weeks work when I changed the theme).

I would never say wordpress/woocommerce is perfect, I’ve found many issues along the way and had to find work arounds for a lot of stuff. I still dont truly feel like I know what I’m doing and there’s no way that we use wordpress to it’s full potential. Currently we have the blog and shop running, we have somewhere in the region of 10k posts and around 15k sales. We still don’t publish to the blog independent of facebook/twitter but it’s on the roadmap.

One thing that has caught us out a few times is DigitalOcean scaling. Because we very often have little traffic, I always keep the servers scaled down with the intention of boosting them up before we push anything new. On at least 2 occasions, we’ve forgotten this and overloaded our site.

I’ve also gone through a few different configurations just trying to find the best solution.

1st We had 1 server, that was mid range and just worked, but I knew this alone wouldn’t handle the traffic.

2nd I brought up 2 web servers and a database server. This wasn’t an ideal setup, loadbalancing was at DNS level, syncing was done via cron jobs, and the whole thing held together via a VPN to keep database connections secure. This had a bunch of problems.

Next I moved back to a single web server but kept a separate database. This was better, and around the time DigitalOcean allowed you to scale up easier (but not down you still had to wipe out the server to do so).

Because having a single web server just wasn’t enough, I went back to 2, but added in the new(ish) cloudflare CDN in front of the servers. This really helped (though I’m still not convinced really does CDN for us)

As part of the above, I tried incorporating GlusterFS (absolute disaster). From every web search I did GlusterFS looks to be THE solution. In practice for us it took a website responding (with some heavy graphics) in 3 seconds longest avg 2secs, to 30sec longest 18secs avg. I know everyone rave’s on about how great it is and how if it’s slow it’s something you’ve done. I dont believe this for a second. I’ve spent days at a time trying to make it better, but the simple truth is if the files are pulled locally I get the 2/3secs above. When using a Gluster mount point to the files (which are still actually local, Gluster on both web servers, mounted back to themselves), I get the 18-30secs. Both web servers have a private lan connection to use gluster in the same Digital Ocean Location and NO amount of tweaking or testing seems to every really improve this. It was only made worse during testing when I took down one of the servers, so that the other could only use itself to serve the files and this managed to take out the mountpoint until I restart, and still it served up the pages slowly. I thought the whole point in using Gluster (at least for me) was HA, no single point of failure. Having both servers offline if one goes down does not seem very HA to me.

The ONE thing I really want Digital Ocean to sort out is their private lan. In order to solve the issue of anyone else on the private lan being able to see my traffic between servers I’ve had to use VPN’s between them. This adds complications to the entire setup, and a private lan per account would be very welcomed.

The setup I’m currently in the middle of deploying is:

a) Cloudflare

b) 2xNginx loadbalance proxies (also serve up maintence pages if they can’t connect back.

c) 2xNginx Backend servers

d) 2xMySql+Redis Servers

e) 2xNFS Servers

I’m happy with the load balancers, though I would love for DigitalOcean to offer a proper loadbalancing solution.

The MySQL servers took some config to get replication working properly while also using SSL for the connection to each other and from the backend web servers.

I still haven’t managed to configure MySQL to be HA from the web servers, so at the moment this would be a manual switch. I’ve found HyperDB for wordpress, while should resolve this, but since I had to slightly change the wordpress config to do SSL for MySQL, HyperDB doesn’t seem to be able to use SSL so I need to work out how to do this. I find this really weird as once of the suggestions is to have your database remote, I really would have thought being remote (especially if using something like Amazon for the database) would mean you’d want to use SSL to keep your database traffic secure. It seems strange that this isn’t a fundamental option in HyperDB (unless I’m just not seeing it).

And the last part NFS Servers, I still need to find out how to keep these in sync (without using Gluster), I’ve previously used Syncthing to keep servers in sync, it works but is pretty much held together with tape (my configuration of it not the actual program). Once I have the NFS sync’d I also need to find a way for the web servers to use both HA.

I do feel like this configuration is the closest to the best I can achieve on a budget. Once I have the MySQL and NFS stuff worked out, I will then be able to scale any server without completely taking the site offline. Which will really help in being able to deal with spikes. It is not much easier to scale with Digital Ocean, but I’d still really want to know doing so or taking a server out for maintenance is fine because everything will just keep running.

If you’ve got this far, I really thank you for reading. I hope the next couple of posts will be my solutions to the MySQL SSL and NFS problems. It’s not 2:41am and I think I’ve been writing this for about 2 hours, so I’m going to sleep 🙂 leave a comment if you got this far, include the words ‘sleep deprived’ so I dont think it’s spam.

Todays WordPress Adventures

Well it had to happen at some point, today I had a nice email from DigitalOcean saying they’ve disabled networking on one of my servers. This was because it’s ip address had been reported to RBL’s by several other servers.

Looking at the logs they included I was beating the s**t out of others wp-admin login pages. Now I know I wasn’t doing it personally, it was the first time in a long time I was in bed early and this seemed to start at 2am.

Luckily I could access my Droplet using the Console page, so after login I sat thinking ‘um…..’ where exactly do you start. The server normally has quite a bit of traffic so the logs are always cluttered. Needle in a haystack springs to mind.

I decided to run htop and see if the server was doing much without any traffic coming in. Oh yes /usr/bin/host is eating resources. So do I kill it or not. I decided not to at this point. Without networking I’m not doing anymore harm, and leaving it running may help find out what’s calling it.

It was a good call. I can’t give details of everything I did, I spent a hour hours checking through stuff. I do remember checking lsof and finding a link between a process id for host and a file within wpallimport’s uploads directory. So I had a look in there, followed by some further searching of google. 1 file in particular .sd0 seems to bring back results and this seems to be what’s caused it.

To get my server running again, I disabled the entire site within apache that was affected (luckily not a major site) and reboot the server. Once I was happy there were no cronjobs or anything calling on this script I mailed DigitalOcean and asked them to re-enable networking. They’re pretty speedy and within 15 mins had done it. A further reboot and my servers back online minus the one site I’ve disabled.

I expect the cleanup for this is going to take weeks of checking files, against backups while keeping as much as possible online.

I’m pretty confident I know what’s caused it, an out of date wordpress install with an out of date wpallimport install. It really goes to show that you have to check old stuff and keep it upto date.

The most annoying thing for me is that WordPress has a multisite option (which I use on 2 installs) and this allows me to keep plugins and everything upto date easily of sub-sites that are barely used. but it doesn’t extend to multiple domains which would really allow wordpress to be used across all my domains from one central console and then everything would be kept upto date in one go.

I know there’s a plugin for multisite domains, but I feel this is more of a hack of the wordpress system rather than wordpress properly designed to function with this in mind. I don’t want to install it and encounter many more problems.

It’s very bad admin on my part not having kept this site upto date, I’ll be the first to admit that but it’s easy to forget about installs you don’t use regularly. There must be some kind of nagios plugin to alert me to out of date plugins/versions for wordpress so I’ll be looking for that later in the week 🙂

Just can’t find it.

Another rambling entry.

I started this bog with the idea of keeping notes of things I may need to refer back to. Tonight I’ve spent 30 mins searching the entries for LDAP authentication on the PI. I do LDAP Auth on all my servers against 2 LDAP servers, and I’m sure I posted somewhere what I did to configure the PI’s to auth against LDAP but I just can’t find it.

What it has highlight though is 1) I don’t post nearly enough of what I do, and when I do it’s normally after I’ve done it and then write it up from memory. 2) The blog design just ins’t very user friendly.

So I’m going to try to write things up a bit more often as I’m doing them and tonight I’m going to look at some better designs. I’ve been installing wordpress quite a bit recently and it just seems to have themes and a better layout, so I’d like to get a similar design running on this blog. I would just run the blog on my own server, but I think there’s a good chance moving my blog will screw up any references to blogspot. Maybe something to look at though when I’m bored.

Raspberry Pi Scanner using HP PSC 2170

A few days ago I decided to have a good sort of all my old documents, statements and piles of paper that I’ve kept for years just incase I need it (I doubt I’m ever going to need my mobile phone statement from 2008).
I’ve been looking on ebay for a few months for a networked duplex document scanner. but these seem to easy break £70 and I don’t think I’d get alot of use out of it beyond initially scanning everything.
I already have a HP PSC 2170 Printer/scanner (not duplex), but the windows software is pretty crap, I use irfanview (which is a great bit of software), but the HP scanning software insists on loading for each scan then warmup the lamp and do a prescan before allowing me to actually scan. It’s always put me off using it.

So while thinking I’ve got papers everywhere, a scanner, and a few free PI’s a little light went off in my head, surely I can use a PI (I’m not expecting wonders with the scanner) but it’s something to play with. So google being your friend I went looking and came across:-
http://eduardoluis.com/raspberry-pi-and-usb-network-scanner/

ok so I could just take what he’s done and just use it. It looks like a good idea, and he/she has updated the process to include some form of web interface. Anyway it’s not what I need I just want to be able to scan to a device that’s on my network.

I put a rasbian image onto the PI, expanded the 16Gb SD card, named the PI and run updates.
Then I installed ‘sane’ via ‘apt-get install sane’
{note this is from memory, my history doesn’t contain all the commands I’ve used, I’ll need to recheck this on a fresh install}
Once sane was installed I used ‘scanimage -L’ to check my device is being detected. Nope nothing get’s listed. A quick google search for ‘sane HP PSC 2170’ I found a page listing HP PSC 2170 as good support. so I thought, well it should just be working. after a bit of scrolling around I noticed the table had an end column listing driver as hpaio, this wasn’t apparent against my printer, nevermind it’s another clue. So a bit more poking around to find out if the driver is installed I found ‘apt-get install libsane-hpaio’. Now when running ‘scanimage -L’ I get device `hpaio:/usb/PSC_2170_Series?serial=XXXXXXXXX’ is a Hewlett-Packard PSC_2170_Series all-in-one’.

With all this up and running I’m now onto getting an image. Running ‘scanimage -p –format=tiff –resolution=150 > test.tiff’ gives me a nice image within a few seconds. I was surprised at just just how quick the scanner sprang into action. No scanning the same piece of paper twice, no warming up the lamp.

Now that I have this, I’d much rather a jpg than a tiff. so I use ‘convert temp.pnm temp.jpg’ oh look another error, convert isn’t a valid command. Of course it’s not, I forgot to install anything. ‘apt-get install imagemagick’ and rerun the convert and now I have a much smaller jpg file. (I’m going to skip the steps I used to install and configure samba, to be able to pickup my images, but take it I did. I could have just as easily used WinSCP to access the files)

With all the tests done, and alot of {press up, press enter} I decided to write a quick bas script. First to just capture an image then convert it using a filename provided, then I improved it to save the filename based on the unix time, next I improved it to be able to group multiple pages into a single folder. Below is the script that when run, gives an option of single scan page or multi scan page, goes off and runs the scan and convert and returns either to the initial prompt or in multimode ask if there’s another page.

cat /usr/sbin/scan_it.sh
#!/bin/bash
scan()
{
image_date=`date +%s`
pwd=`pwd`
echo "  :  Scanning Image $image_date into $pwd"
scanimage --resolution=200 > temp.pnm
convert temp.pnm $image_date.jpg
rm temp.pnm
}
while [ true ]
do
  read -n 1 -p "Single, Multiple, Quit? " INPUT
  case "$INPUT" in
  s)
    cd /var/scans/
    scan
    ;;
  m)
    folder_date=`date +%s`
    mkdir /var/scans/$folder_date
    cd /var/scans/$folder_date
    YN=y
    while [ "$YN" == "y" ]
    do
      scan
      echo "Another Page? (Y/N) "
      read -n 1 YN
    done
    ;;
  q)
    echo " Quitting... Goodbye..."
    exit 0
    ;;
  *)
    echo " Not an Option! "
    ;;
  esac
done

This script presumes you have already made a folder ‘/var/scans’ using ‘mkdir /var/scans’

I did also setup sane to run as a network service making the scanner available from other machines, but I haven’t tested this yet. For my needs I decided to keep the images as jpgs rather than make PDF’s I still may make PDF’s out of all my documents, especially the multi paged ones. But I like the idea that I can access jpg’s on pretty much anything (including my recent experiments with python and pygame to display graphics on my TV via HDMI. I think PDF’s are a little more restrictive).

When doing the initial google search I also came across:-

http://blog.doctape.com/turn-your-raspberry-pi-into-a-scan-to-cloud-device-2/

Which looks like a really promising idea. To move away from needing terminal access to the PI to scan stuff.

The main reason I’ve decided not to go any further at the moment, is my printer/scanner I think is starting to die (not related to this setup) on a few occasions it’s refused to scan and needed a reboot, it’s also locked the scanner at the bottom of the page instead of returning after a scan, and went through alot of clunking (which just wouldn’t stop) when it was initializing after one reboot.

While it has the possibility of failing to complete a scan I’d rather be able to see any errors on screen.

I’ve spent tonight scanning over 400 pages, at the end of which I copied all the files to my server and run a backup then happily set fire to each piece of paper I’ve been keeping for years. It’s been something of a therapeutic exercise. I have no doubt I’ll be returning to this little project in time to actually incorporate sending the files directly to the server, having the option to email them, converting to PDF on request, and god knows whatelse. For the time being this script is doing me just fine. The only option I may incorporate is a selection of a ‘scan to’ folder before selecting single/multiple after scanning alot of bank statements, phone statements, payslips, etc I know I’m going to be sorting them all into subfolders and doing this at the time of scan just makes more sense.

Anyway that’s my rambling for today. Hopefully it will give someone some ideas.

Raspberry PI

After waiting all of Feb for news of when the Raspberry PI was going to be released, We finally got told that an announcement would be made at 6am on Wednesday 29th.
As I was still awake at 3am, I thought no point in setting the alarm I’ll just stay awake. Hoping that this would indeed be the launch day.
(As it was very early in the morning and I’ve definitely slept since the following times are approx.)
Sometime after 5am raspberrypi.com was replaced with a down for maintenance page.
Then about 5:45am, raspberrypi.org seemed to disappear off the net, then come back with a nice server error. but this only lasted a few mins.
As the time got closer to 6am, I could feel the pain of the PI’s servers. As I had at least 10 mins to go, I wasn’t hitting refresh 60 times per min (I’m sure enough people were). Every now and again the server wouldn’t respond, so I’d gave it 30 secs and try again.

All of a sudden (I’d say at about 5:58am), the raspberrypi.org page loaded up and was completely different. This is it the announcement we’re all waiting for.
As I scanned through quickly, I noticed them saying about licensing 2 resellers I kept looking quickly for the ‘where to buy’, then I got it 2 links. So I quickly right clicked on each and opened in a new tab.

Both loaded fairly quickly. Once into the Farnell site I searched raspberry, as the results come back it said 2 products but didn’t show them. So I click on the words products and it reloaded and displayed. Clicking the add to cart option threw up a nice little dialogue under the cart saying adding, then that was replaced with an ‘unable to find price in database. Item not added to cart’ error.
SHIT!! was the exact thought, never mind try the 2nd obviously farnell has an issue.
As I got to the RS site again there wasn’t anything on the homepage like I’d expect for a pretty big launch. So again type raspberry into the search and hit enter. It comes straight back with a specific RaspberryPI page. I scrolled quickly down looking for the add to cart button or buy button, but nothing only a form to fill out with my details to register my interest.
‘Surely that’s not right, I’m on the wrong page.’ I thought. Again type raspberry into the search box and back I come to the same page. Bugger it, as least I found it on the farnell site so I went back to that tab. reloaded the page and searched again, expanded the products and clicked the add to cart button. Again I got the nice error!! I’m seriously not impressed now, I’ve hit both sites very early on but not getting me any closer to having a nice order number.
Back I went to the RS site and thought well I’ll fill in the details and maybe it’ll direct me then to another page. Filled them all in and submit. Great but still no page where I can order it. So decided to go back to the start page for RS and look at the categories incase it’s not coming up on a search. It was at this point that the RS site started to load slowly for me. So back I jump to farnell, and refreshed back to the homepage.
Now the fun begins, Farnell just completely stopped loading anything for me. give it a couple of secs and refresh, oh I’m pushed to a page telling me their site is overloaded but I can phone an order through.
Great stick the number into my mobile and press dial. Nope it didn’t dial, instead I’m being sent to a page. Thinking to myself wtf, as the page sprung up with a list of alternative numbers, I quickly realised I installed No to 08 app ages ago (I dont ring 08 numbers very often and forgot I did it). Never mind though it’s given me an alternative so clicked it and dial commences. Ring-Ring, and I’m answered. Oh NO it’s an IVR happily telling me their lines open at 8am but I can place and order or enquiry at their website. Not F###ing likely I thought while hanging up.
Refreshed Farnell, dead. back I go to RS. It’s loading but the search is still giving me the same register my interest page.

All this has taken place within about 5-10 mins.

So thinking quite calmly, ‘Maybe they haven’t released yet, I should really read through the raspberry announcement page’. So back to that tab and scrolled to the top. This time paying attention, I take in all the info and yep they are on sale from today.
Back to Farnell tab and refresh = nothing.
Over to the RS tab and refresh = oooohh it’s slow, and now it’s gone.
After about 5-10 mins of jumping back and forth between tabs, and I have to admit I’d also opened additional tabs since I’d been refreshing the Product Search on Farnell, I’d opened a new one for their front page. and similar on the RS.
As I’m getting nowhere fast, I went back to the RPI page. Then noticed they’re twitter link. So I opened this (in a new tab).
oh there’s lots of people shouting on here that they can’t get anywhere. All of a sudden I noticed a posting from Liz saying if your getting the register your info page on RS your on the wrong page.
‘F#ck It!!’ I thought, back to the RS tabs and refresh, luckily the one did load and did another search still it took me back to the register interest page. Another search gave no results.
Back to Twitter I go, in the 20 or so seconds I’d been away twitter was very happy to tell me 80+ new posts.
Now I have to admit, I’ve gone through product launches from a tech side and it’s always worse than what the marketing people tell ya. I knew that the R-PI was going to be big and that the first 10,000 was going to sell out fast. I’d only judged that on the number of downloads that the (pre) software image had a few weeks back and the number of people talking in the forums. Having never looked at the number of followers the R-PI had on twitter, I didn’t expect the overwhelming chatter I now threw myself into.
I’m not a big fan of twitter personally, it just doesn’t appeal or click with me. But at 6:30 in the morning I have to say I was enjoying managing to read 2/3 comments before it was popping up another 10+ new comments, clicking that reading a few and repeating.
While doing this jumping to other tabs and trying to refresh, unlike early though where the pages did load but slowly, I was now hit with no response across all the Farnell & RS refreshes.

After another 20 mins of reading tweets and trying to refresh pages, I had a call from my cousin did I want to go for coffee and he’s now on the way home.
‘Yep, fine’, not getting anywhere with this now. 2 sites down, lots of angry tweets, and my hopes of getting a R-PI early was pretty much wiped out.

After an eventful trip to the coffee place, which consisted of the coffee girl in tears because a boss she doesn’t like is coming back to the store and she now has to transfer stores.
We’re back in the car, oh the times 7:59am. Well their lines open at 8am, I doubt very much that they’ll have any left now, but don’t ask don’t get so I pressed redial on my phone. It rang and rang then an announcement ‘higher call volumes than expect, please hold’, at least their open. Then I get a nice double beep off my phone 8% battery remaining.

Now I’ve been in much worse positions with little battery life, but the timing just sucks. never mind I’ll keep on the line until the battery is dead. Now I have to say a big thumbs up to HTC, every other phone I’ve had has pretty much died within seconds of telling me it’s low when on a call, but this one managed a 10 min journey home while still on the call (in a queue I should add), and my 5 mins fumbling around single handed trying to find my keys, while not dropping either my phone or tablet.
Alas after 15 mins of holding, I decided there’s little point in holding, if they had any left at 8am, 15 mins worth of calls would definitely wiped them out. So I hung up.
Went back to my laptop and refreshed. Both pages still out, so back to twitter I go.

Plenty more chatter to catch up on, but still there’s one hell of a constant flow of new stuff coming through.
All of a sudden a refresh of Farnell kicked in. It was my search one, so I quickly clicked add to cart. oh it’s added!!!!!
Checkout!! – it’s only gone straight through and given me the checkout.
So in I filled the address info, and proceed. it’s trying to load an invoice page but nope timed out. I can see in the addressbar I have a session ID, so maybe just maybe it will remember me if I refresh and it wont kick me back to the start.
5 refreshes later and I have an invoice page loaded. Quickly filled in my details and onto the next page.
All in all it took me about 5 mins to get all the way through and a hell of a lot of refreshes along the process. but finally it’s complete my order.

I received an email confirmed my new account, a few hours later I had some mails confirming my order and giving an estimated delivery of end of march (I’d be quite happy with that I thought).

Today I received another confirmation of my order, but now pushing estimated delivery back to the end of April. I’m kind of expecting that to get further back as time progresses, but we’ll see what happens.

This evening I received a mail from RS confirming my interest and saying they will be mailing out to take orders later in the week. They also put that the orders will be sent out in a first come first served.
Now I’m not entirely sure if they mean the register your interest order, or a new order starting with another gold rush when they mail people saying you can buy them from them.
If it’s the former, I’m hopeful that my submission for interest was early enough to be amongst the first while everyone else was still looking to buy it. It was after all done while they’re site was quite alive.

Time will tell. I’m still really excited with all this, and think that the enormous following it’s gathered has to be a good thing. I’ve got so many little projects that I want to do with it, if I could get a few by the summer-time it would definitely keep me busy all summer.

Ajax woes

Well it’s been a while since my last post, I’ve been working on a website that’s been taking up alot of my time and giving me a big headache.
Basically I started with a one page template that had a nice clean look. Unfortunately since I was last working on webpages, CSS has sprung up and taken over. So for every little change I had to copy an existing bit of code then change it, then refer to the CSS file and after alot of playing and going back and forth I finally got the template file looking how I needed it to to progress.

Now that I have my template page, I took that and created a few basic pages. To get the pages a bit more functional I changed the pages over to PHP. This worked well and now I had a whole login/register system, some basic pages displaying info, and some pages that interacted with a database.

All this was starting to come together and look good.
Things are never supposed to run smoothly though are they. The more I used the pages the more I thought that they need to load more smoothly, rather than going to a new page request for everything I needed to start using ajax.

So instead of finishing off the site as I was going, I decided to rewrite what I already had done to fit into an ajax framework.
Changing each of the tabs to pull the relevant page into the existing page was pretty simple. I got the tabs all working, highlighting the correct tab and changing along the way perfect.
Now the tricky bit, alot of the pages use forms and scripts. So I decided to try tackling the scripts first.
Basically I dont want to load all the script on the first load of the page as it’ll load up a lot of scripts that wont even be used. So I put the script in the page that gets pulled by ajax. seems like a simple solution.
However if the script isn’t present on page load the script isn’t recognised. ok I’ll stick the scripts to one side and work on the forms. oh no a problem, to use the forms inside ajax it needs a script to run.
So I fell back and decided just stick the script in the head of the first file at least it’ll get it all working. Nope that didn’t work either. As the elements aren’t present on the page when the script is first loaded the submit intercept stuff isn’t getting applied.

So I’ve started looking for a way to put scripts into ajax pages that will be relevant scripts for that page. I think this would be pretty common, but after hours of looking there is stuff out there mentioning it, but nothing explicitly saying how to have an initial script running to interpret other scripts inside an ajax page.

So there’s my ramblings for the night and the reason’s I stopped posting stuff for a few weeks. I don’t want to post details about the site just yet as it’s a complete work in progress, I have alot of the backend working but the entire site is currently ripped apart and non-functional. I know all the bits I want to get it to do, just not quite how to do them. I’ll solve it eventually, you’d have thought from the years working within IT I’d have loads of development contacts, unfortunately I have quite a few web designers, but none that do php/code development.

<update>
ok so after alot of searching around I found alot of stuff saying anything in a script should get interpreted as it’s loaded but that wasn’t happening for me. more and more searching I found a different way of loading the content instead of the httpxml or xmlhttp whatever it was, the following works for me:-

function loadajax(page,tab)
{
var page,tab;
$.ajax (
{
method: "get",
url: page,
dataType: "html",
success: function( strHTML ) {
$( "#content" ).html( strHTML );
}
}
);
}
</script>

Hey presto as the page gets loaded the script inside is now run too. Now to rework all the forms and stuff on the site.
<thought>I really do need to look at how you put script on a blog properly too.</thought>

Blog update.

ok, just run through enabling comments on most of my posts. Wondered why I’d never seen the option when posting I must have turned it off globally a good while back. Unfortunately when enabling it there’s no quick way I could see to turn it on then for each post so had to edit, set option and save. Just hope it hasn’t screwed with the dates on the posts now.

Apparently I’ve also turned on adsense for this blog, I didn’t really want it linked to this I wanted it link to another site. but if I get it on here first I can checkout how it works and hopefully move it over later even if it means cancelling and re-registering my email address to the other site. I did read something on it about no multiple accounts per site. hoping it still works the other way.

Blog stats

Just checked my blog stats, I’m amazed people are actually reading stuff I’ve posted. Can’t remember if I’ve allowed feedback and questions, so will check the posts tomorrow as I’d love to know what people think and if I’ve missed anything or can help anyone out.
One thing I have to remember to post about tomorrow is my problems with viewing zoneminder cameras remotely via the web browser.
So thanks to anyone who actually reads anything I’ve done.