← Go back to NixNet Status

Unknown issues

September 26, 2021 at 11:34 PM UTC

IRC Misskey Mumble Pleroma XMPP Etherpad Framadate Hedgedoc Mobilizon Bibliogram Nitter RSS-Bridge TinyTinyRSS fosspay Gitea linx MediaWiki PrivateBin searx Wallabag xBrowserSync

Resolved after 57m of downtime. September 27, 2021 at 12:32 AM UTC

Something has gone wrong with all services running on our Hetzner machine. We’re not entirely sure what the issue is but we’re investigating.

Programs like htop hang, logging in over SSH is successful but the shell hangs, LXC commands hang and more as well. As a temporary solution, we’re going to reboot the physical machine, immediately SSH in, and start poking around in the logs to see what exactly was going on. (23:34 UTC — Sep 26)

SSH doesn’t seem to be starting so we contacted Hetzner support; a datacenter technician will be attaching a KVM (Keyboard/Video/Mouse) to the machine so we can inspect the boot process and see what’s going on. If we’re unable to fix it using the KVM, we’ll boot a rescue image, chroot in, and start poking around. (23:57 UTC — Sep 26)

We’re still unsure what the root cause of the issue is, but we executed a hard reset of the whole server and everything is back up. We’re inspecting logs to figure out what happened. (00:32 UTC — Sep 27)


Last updated: September 27, 2021 at 1:05 AM UTC