Symptoms
You might come across this issue on any number of cloud / virtual server providers, it’s usually an issue that occurs on quite small instances. You have a database, it’s running just fine, then out of the blue it crashes, it keeps happening intermittently at random times but it seems fine when you restart the service.
If you’re running MySQL you might see something similar to the below in your log files, however I have seen this issue with other databases too.
InnoDB: Fatal error: cannot allocate memory for the buffer poolThe Issue
In my case the issue was that the smaller instances available on AWS, Google Cloud, Digital Ocean etc usually don’t have that much memory and are often created without any swap by default. This means that if your server runs out of memory (even for a moment) there is now nowhere to put anything and things start crashing.
The Solution
Firstly check if you have swap file, if you have it installed the easiest way to do this is run htop which will show you both memory usage and swap, if not you can just run the following:
cat /proc/meminfo | grep SwapWhich should return the totals for your swap, if that doesn’t work either install htop or check out this Stack Exchange answer.
If you don’t have any swap here is how you can create it (note that this shouldn’t cause any issue but you might want to take a backup image of the server first just in case):
sudo /bin/dd if=/dev/zero of=/var/swap.1 bs=1M count=1024
sudo /sbin/mkswap /var/swap.1
sudo chmod 600 /var/swap.1
sudo /sbin/swapon /var/swap.1This will create a swap file (1 GiB), change it’s permissions and tell the OS to use it as swap. You’ll also want to enable it after a reboot `sudo nano /etc/fstab` and add the following line:
/var/swap.1 swap swap defaults 0 0Ideally you should reboot at this point to check.
I can’t be sure as it was some years ago that I originally came across this issue but I imaging the code for fixing it came from this Stack Overflow answer so refer there if you need additional info.
0 Comments