APT not automatically updating DigitalOcean

I’ve recently noticed a problem on 3 of my Digital Ocean Servers. The APT package lists are not automatically updating every day. I try to keep all servers upto date, and rely on Nagios to inform me when there’s packages needed to be updated and that’s the main reason I noticed something was broken.

The 3 servers in particular are newer builds to the rest of the system, and they dont have near as much installed as the others, so at first I didn’t pay too much attention when other servers were going into warning state on nagios indicating updates but these 3 weren’t. However I would still connect to these servers and run my normal command:-

apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove

A few times these servers did install updates and I just thought it must have been my timing, that the package lists hadn’t yet been updated by the cron.daily.

But after this happening a few times, I decided to not run the above and see how long these servers would take for nagios to throw an alert. It never did and that got me a little worried.

Over the last few days I’ve been diagnosing what’s wrong. I started out with making sure cron is working properly. Then kept an eye on the file timestamps

ls -ltrh /var/lib/apt/lists/

Eventually getting to /etc/cron.daily/apt and checking through what was was doing on the working servers compared to the broken ones. I turned on VERBOSE and got a bit of info when running /etc/cron.daily/apt but it seemed to exist quite quicky.

Comparing it to a working server the important bit seemed to be around

+ apt-config shell Debdelta APT::Periodic::Download-Upgradeable-Packages-Debdelta
+ eval
+ [ 1 -eq 0 ]
+ do_cache_backup 0

On the broken servers I was getting

+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]
+ [ 0 -eq 0 ]

Then it would exit. Further investigating was showing a few settings were being populated on the working servers but not on the broken ones.

So I compared the directory /etc/apt/apt.conf.d/ on both servers an found the following files missing from the broken servers

00aptitude

Aptitude::Get-Root-Command "sudo:/usr/bin/sudo";

10periodic

APT::Periodic::Update-Package-Lists "1";
APT::Periodic::Download-Upgradeable-Packages "0";
APT::Periodic::AutocleanInterval "0";

15update-stamp

APT::Update::Post-Invoke-Success {"touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";};

20archive

APT::Archives::MaxAge "30";
APT::Archives::MinAge "2";
APT::Archives::MaxSize "500";

99update-notifier

DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi "; };
(reverse-i-search)`apt-': apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove

I think the main one being 10periodic but I didn’t fancy spending days/weeks adding each and waiting to see what happened, so I added them all in one go.

Then run /etc/cron.daily/apt again, and this time it’s taken a lot longer to run i.e I’ve written this entire post and it’s still running, as opposed to exiting within a few seconds earlier.

I’m pretty confident that this has now solved my APT package list not automatically updating problem, and providing it has Nagios will start warning on these 3 servers now.