Badly written analytics script causes cascading homelab failure

This all started a couple weeks ago. Things started to slow down on my homelab server. My Minecraft server kept crashing. I thought that the Minecraft server was the cause of this because it has a lot of memory allocated to it and every time this would happen, the memory would fill up and the Minecraft server would crash. That thought was further reinforced when I noticed the memory was almost about to overflow when I stopped my Minecraft server and the memory usage significantly went down.

For about a week I stopped my Minecraft server. And then services started to fail and memory usage was high. So I opened the terminal to check on the programs using memory.

Investigation

I used the top command to find the memory hog. I found that /usr/bin/python /app/server.py was using 70% of the memory, but my server has no /app directory. After further tracing, I discovered that this program belonged to a docker container running my analytics backend. For those of you who don’t know, I wrote a custom privacy-friendly analytics script that is used on my websites. The backend is extremely simple. It just takes the data from the request and adds it to a JSON file.

But in order to add to the JSON file, it must load it into memory. This is the problem. After a while, the file has become so big that it hogs all the memory.

Temporary fixes

The temporary fix is to delete or move the database file every few months. This is a very easy and quick solution to the problem. So far, I have implemented this. I only deployed this analytics program in August of 2024 so I will probably encounter this issue again sometime in May or June 2025.

Permanent solutions

The only permanent solution is to rewrite the analytics backend to use better database handling and/or switch to something better like SQLite.