I’ve recently noticed a problem on 3 of my Digital Ocean Servers. The APT package lists are not automatically updating every day. I try to keep all servers upto date, and rely on Nagios to inform me when there’s packages needed to be updated and that’s the main reason I noticed something was broken.
The 3 servers in particular are newer builds to the rest of the system, and they dont have near as much installed as the others, so at first I didn’t pay too much attention when other servers were going into warning state on nagios indicating updates but these 3 weren’t. However I would still connect to these servers and run my normal command:-
apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove
A few times these servers did install updates and I just thought it must have been my timing, that the package lists hadn’t yet been updated by the cron.daily.
But after this happening a few times, I decided to not run the above and see how long these servers would take for nagios to throw an alert. It never did and that got me a little worried.
Over the last few days I’ve been diagnosing what’s wrong. I started out with making sure cron is working properly. Then kept an eye on the file timestamps
ls -ltrh /var/lib/apt/lists/
Eventually getting to /etc/cron.daily/apt and checking through what was was doing on the working servers compared to the broken ones. I turned on VERBOSE and got a bit of info when running /etc/cron.daily/apt but it seemed to exist quite quicky.
Comparing it to a working server the important bit seemed to be around
+ apt-config shell Debdelta APT::Periodic::Download-Upgradeable-Packages-Debdelta + eval + [ 1 -eq 0 ] + do_cache_backup 0</pre> On the broken servers I was getting <pre>+ [ 0 -eq 0 ] + [ 0 -eq 0 ] + [ 0 -eq 0 ] + [ 0 -eq 0 ]
Then it would exit. Further investigating was showing a few settings were being populated on the working servers but not on the broken ones.
So I compared the directory /etc/apt/apt.conf.d/ on both servers an found the following files missing from the broken servers
00aptitude
Aptitude::Get-Root-Command "sudo:/usr/bin/sudo";
10periodic
APT::Periodic::Update-Package-Lists "1"; APT::Periodic::Download-Upgradeable-Packages "0"; APT::Periodic::AutocleanInterval "0";
15update-stamp
APT::Update::Post-Invoke-Success {"touch /var/lib/apt/periodic/update-success-stamp 2>/dev/null || true";};
20archive
APT::Archives::MaxAge "30"; APT::Archives::MinAge "2"; APT::Archives::MaxSize "500";
99update-notifier
DPkg::Post-Invoke {"if [ -d /var/lib/update-notifier ]; then touch /var/lib/update-notifier/dpkg-run-stamp; fi; if [ -e /var/lib/update-notifier/updates-available ]; then echo > /var/lib/update-notifier/updates-available; fi "; };
(reverse-i-search)`apt-': apt-get update && apt-get upgrade && apt-get dist-upgrade && apt-get autoremove
I think the main one being 10periodic but I didn’t fancy spending days/weeks adding each and waiting to see what happened, so I added them all in one go.
Then run /etc/cron.daily/apt again, and this time it’s taken a lot longer to run i.e I’ve written this entire post and it’s still running, as opposed to exiting within a few seconds earlier.
I’m pretty confident that this has now solved my APT package list not automatically updating problem, and providing it has Nagios will start warning on these 3 servers now.