It was a chilly Monday morning. School was cancelled due to the ice on the roads. I thought it would be a perfect day to get some stuff done in my homelab. But as they say, anything that can go wrong will go wrong.

First signs

In the morning I installed Pi-hole on my raspberry pi. You can read my blog post about that. After I deployed the Pi-hole, I tried to install another app to deploy something else. When I was installing that, I was getting errors about some syntax error in a file. I opened the file in question using vim and it looked normal. I found a backup of that file and I copied the backup to the main file and everything worked.

For almost a year I have been using Pi-hole as the only DNS server in my network. For those of you who don’t know, Pi-hole is an open-source solution for custom DNS records and ad-blocking in your home network. My Pi-hole instance has worked just fine up until now. Recently, it has started slowing down the entire network and crashing.

Picking a server

Right now I have 3 servers. 2 of them are running linux, and the other one is running Home Assistant OS. The Home Assistant server is pretty locked down so it may be hard to get Pi-hole up and running on it. One of the linux servers is already running Pi-hole.

This all started a couple weeks ago. Things started to slow down on my homelab server. My Minecraft server kept crashing. I thought that the Minecraft server was the cause of this because it has a lot of memory allocated to it and every time this would happen, the memory would fill up and the Minecraft server would crash. That thought was further reinforced when I noticed the memory was almost about to overflow when I stopped my Minecraft server and the memory usage significantly went down.

New year (almost), new me blog! It feels like this is the millionth time I made a new blog. My previous blog was really bad. I had to write blog posts inside a JSON file. Nothing at all was automated.

Recently I started using Obsidian to take notes. It’s just a markdown editor but with a fancy UI and some more features. Then one day I realized I could write my blog posts in Obsidian and have them automatically pushed to my website.

This morning I woke up to my phone using mobile data and my home assistant automations not working. Initially I thought it the power was out, but I could turn on the lights just fine. I checked my UniFi app and saw that the server was not connected to the network at all. This meant that the cable got unplugged, the switch isn’t working, or the server isn’t working. It said the switch was connected and another device was connected to the switch so that narrows it down to just 2 cases. So I opened my server closet in the basement and immediately noticed something was wrong. I couldn’t figure out what was wrong but I just felt like something was wrong. Everything was plugged in, the network switch lights were blinking like normal, my raspberry pi was running just fine, even the server indicator lights were on. My main server is an old gaming PC so it has a glass side panel so I looked inside and I could see the fan spinning, but I could not hear it. Usually I have it set to full speed and I can hear full speed very well. I tried rebooting the server with the power button and the fans didn’t go to full speed. As a last resort, I brought down a keyboard and monitor. As soon as I plugged in the monitor, I saw that there was a prompt to set the time on the BIOS!
In my opinion, this was the stupidest reason for an outage.