SSH service not responding

Question

I have a headless Ubuntu running on a regular PC in my garage. I use SSH to access it. It's running a good handful of Docker containers, including dashdot which indicates the CPU is under normal load etc.

Since today it does not respond to PuTTY connection attempts, but the server is otherwise running fine, the Docker containers are running fine. The machine responds to ping requests, but taking ~700ms to respond.

I did not change ports or make firewall changes. I have no guess what might have caused this.

How can I access the machine, except by rigging up a monitor and keyboard in the garage? I could probably also solve it by just hard-rebooting the machine, but I'd like to try softer methods first.

Answer 1

Disclaimer: I don't endorse any of the products mentioned here - I just happen to know and/or use them.

The big challenge with a headless server is of course if the SSH daemon goes down, in which case your options could be rather limited. You may not be able to do anything about this with your current setup, but I have some suggestions that will allow you to connect the next time.

Option 1: Webmin

Webmin has an interactive shell (shiny new version added in Webmin 2.010) that works without SSH.

Setup Webmin repos:

sudo curl -o setup-repos.sh https://raw.githubusercontent.com/webmin/webmin/master/setup-repos.sh
sudo sh setup-repos.sh

Install Webmin:

sudo apt install webmin

And contrary to old news, the recent versions of Webmin are perfectly supported on Ubuntu.

Option 2: Mainboard with IPMI

I also use a SuperMicro server motherboard with IPMI. This very handy, because the IPMI interface has an integrated Web KVM module - so you can access your terminal screen directly through the IPMI interface.

Option 3: Raspberry Pi KVM

A third option (if your hardware doesn't support server mainboards) is to use a dedicated Raspberry Pi as KVM. There is a project called "PiKVM" that makes both software, a RasPi HAT, and a complete kit with everything.

This solution presents a Web KVM interface similar to that accessible with IPMI (but actually, the interface is much nicer than any IPMI solution I've seen).

(As of March 2023, Raspberry Pi's are still in shortage, so it might be difficult to find one until mid 2023...)