I can't connect to Ubuntu server from local network, but can from outside

Question

This is a weird one that has been driving me crazy for a while. I just can't work it out.

I'm running Ubuntu 22.04 (with all latest patches) on a Dell PowerEdge-r710 server. It's hosting a variety of services on a variety of ports. Many are being run under Kubernetes, some are directly under the host. This is on a domestic LAN behind a Verizon router. I've used port forwarding on the router to route all the ports of interest to the machine and that all works just fine.

What doesn't work, is when I try to access those ports from the internal network. It used to work, but now it doesn't.

Here are some example cases:

• Example 1, native service. I can ssh -p 8022 [email protected] just fine from inside or outside my house. But when I try ssh -p 8022 [email protected] from inside the house, it can't connect. Unless I do it from the command line of the server itself.

• Example 2, Kubernetes hosted web server. I can curl http://publicname.com:8181 fine, but curl http://192.168.1.173:8181 does not work. Ditto for working from the shell of the server.

• Example 3, the odd one out: ping publicname.com works fine, but so does ping 192.168.1.173. O_o

I have tried this from multiple machines inside the house. I have tried it from Windows machines and Mac machines. It's all the same. So I'm assuming it isn't workstation dependent.

I've tried it with the local name, but I'm just using the internal IP address here, to take DNS out of the equation. I've tried it when both the server are plugged into a switch, downstream from the router. I've rebooted my router. I don't generally make many settings to my router (other than port forwarding) and there's been no firmware upgrade for it for at least a year. So I'm pretty sure it isn't a router problem.

I've both enabled and disabled UFW, and it doesn't seem to matter. With UFW off, I've tried adding explicit access to, say 8022:

*A INPUT -p tcp -m tcp --dport 8022 -j ACCEPT*

No help.

There's a lot of other UFW, Docker, and Kubernetes rules in iptables (and iptables-legacy) which I can dump if it would be of use.

Here is a dump from ip -4 route list:

default via 192.168.1.1 dev eno4 proto static metric
100  blackhole 10.1.23.128/26 proto 80 
10.1.23.136 dev cali83503037a1d scope link 
10.1.23.137 dev cali94b8c5e0ec2 scope link 
10.1.23.141 dev caliebd7497b285 scope link 
10.1.23.142 dev cali224b6f89d2e scope link 
10.1.23.143 dev calia147e44c90e scope link 
10.1.23.146 dev caliebea71cd4c0 scope link 
10.1.23.147 dev calief22f96cfc6 scope link 
10.1.23.149 dev calif356e5d735e scope link 
10.1.23.154 dev cali9a9cf33aad8 scope link 
10.1.23.155 dev cali0ce1e6d1dc6 scope link 
10.1.23.156 dev cali5a5d94e5703 scope link 
10.1.23.159 dev calib51c35c69e3 scope link 
10.1.23.160 dev cali05a64ae21b6 scope link 
10.1.23.174 dev cali6d73584524e scope link 
10.1.23.179 dev cali3df919b30be scope link 
10.1.23.181 dev cali6eba53f48ff scope link 
169.254.0.0/16 dev eno4 scope link metric 1000 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.1.1 dev eno4 proto static scope link metric 100

Here's a dump of the IP address:

$ ip addr show dev eno4
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 84:2b:2b:61:b1:96 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
    inet 192.168.1.173/32 scope global noprefixroute eno4
       valid_lft forever preferred_lft forever
    inet6 fe80::862b:2bff:fe61:b196/64 scope link 
       valid_lft forever preferred_lft forever

I've put up with this for about a month now, while trying to work it. For ssh I can deal. But I've added a SAMBA drive for backups and I really don't want to expose that to the internet just so I can get to it from my LAN.

Does any of this make sense to anyone?

Answer 1

Raj's comment gave me an idea. It seems you have somehow defined your local IP address with a /32 subnet, as seen in the output of ip addr show dev eno4:

5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 84:2b:2b:61:b1:96 brd ff:ff:ff:ff:ff:ff
    altname enp2s0f1
    inet 192.168.1.173/32 scope global noprefixroute eno4
       valid_lft forever preferred_lft forever
    inet6 fe80::862b:2bff:fe61:b196/64 scope link 
       valid_lft forever preferred_lft forever

This means that the network has only a single IPv4 address and all traffic will go directly between the device with that IPv4 address and the default gateway (see here), again supporting Raj's comment.

So you should fix this in your network configuration. Most home networks use a /24 subnet (255.255.255.0 subnet mask) to allow for the forth octet to be used as IP addresses (gives 254 possible IP addresses in the range).

Since it doesn't make sense for a DHCP server to hand out IP addresses in a /32 subnet range, I think this is a wrong configuration in your Netplan (here assuming that you have set a static IP for the server). So set the subnet mask of your static IP to 255.255.255.0 instead of 255.255.255.255.