Service Dying At A Specific Time

The Problem

I had another developer ask me about this a few months ago. He had a SystemD service that kept dying, there didn’t seem to be a consistent pattern to when it died but it always happened at 1AM. To put it another way it didn’t die every day or even every week but when it did die it did it at 1AM.

If you encounter an issue like this, where a service is repeatedly stopping at a specific time, especially if that time is a multiple of 5 or 15 minutes (e.g. the sort of time you might set on a cron job), then that should be a clue that it’s probably something that the server is doing on a schedule that’s causing the issue.

The Solution

In my case it was the automatic updates for the operating system, theses sometimes require a reboot to take effect and you can set a time for that to happen when configuring the updates. On Ubuntu servers this is usually handled via the unattended-upgrades package, you can check it’s configuration including the currently configured reboot time with the following command:

cat /etc/apt/apt.conf.d/50unattended-upgrades

However this isn’t a problem in and of itself as the server should be installing security updates and it may need to reboot for this to happen. The issue is that the SystemD service was not set up to automatically restart on reboot.

In SystemD there are two different commands, start and enable. The start command manually starts the service right now, however it does not set the service to automatically start on a reboot. The enable command will tell the service it is to run automatically i.e. when the server reboots.

To see the current status of a service, including where or not is has been “enabled” you use the following command:

sudo systemctl status example-service

You can enable the service with this command:

sudo systemctl enable example-service

And start it with:

sudo systemctl start example-service

For me this was a pretty quick one to fix but if you’re not familiar with SystemD or how automatic updates work in Linux (especially Ubuntu) then it could be a bit of a head scratcher so I thought I’d write a quick post on it.

Useful Links

• Fedora Docs – Understanding SystemD
• RedHat – SystemD Cheat Sheet