Route traffic on different ports through different network interfaces

Question

My situation is this: I’m setting up a server running at home (Ubuntu Desktop 22.04.3 LTS) to run an email server and a few other online services. As we all know, for my email to work reliably and not get blocked I need to have an unchanging public IP address. Due to my circumstances I am not able to get a static IP address through my ISP or change ISPs at the moment.

The solution I have found is to buy a 4G SIM card with a static IP (from an ISP that offers that), which I can then use with a USB dongle. However this 4G connection costs me substantially per MB to use.

But. Mail is the only server that needs a static IP address. For everything else using my home network connection and updating my DNS records with DDNS would be fine. I have tested this setup previously for other services and it has worked.

So. I was wondering. Would it in theory be possible to: connect the server to two network interfaces at the same time and route traffic depending on destination port. I.e. all outgoing connections to ports 25, 465, 587, and possibly 993 should be sent through the 4G dongle interface (enx344b50000000) and all other connections sent over eth0. Similarly, the server should listen for incoming connections on the same ports on enx344b50000000 and listen on all other ports (if allowed by ufw) on eth0.

I would then need DNS records from mail.mydomain.tld —> <4g static public IP> and mydomain.tld —> <home public IP> (updated with DDNS, and NAT configured on my home router).

Computers on the internet would then be able to seamlessly connect to these two IP addresses, not “realising” that they are in fact the same machine, as long as requests to mail.mydomain.tld are always on the above mentioned ports.

Question: Is this possible? Could it be a robust solution that works the way I hope? Would someone be able to help me set it up?

I have come across a few different guides in my DuckDuckGo-ing, I understand it has to do with setting a mark in iptables and assigning them to a table using ip route. However I haven't managed to get it to work yet, and many of these guides are for VPNs and they all seem to be slightly different to each other. So I thought I would ask about my own specific use case and see what people think?

Thanks!!

Edit 1: More info

Following the instructions in the answers to implement this (thanks!) I am trying to test it with port 443. The idea is that I will be able to run curl ipinfo.io/ip and curl https://ipinfo.io/ip and get different results.

After following all the instructions it seems I am successfully intercepting connections to port 443 but they are not being passed on to the internet. curl ipinfo.io/ip returns expected results, but curl https://ipinfo.io/ip returns:

curl: (7) Failed to connect to ipinfo.io port 443 after 1813 ms: No route to host

This is before and after running

iptables -t nat -A POSTROUTING -o enx344b50000000 -j MASQUERADE

---

The following steps have completed successfully:
Note I am currently using wlp2s0 interface in place of eth0.

iptables -t mangle -A OUTPUT -p tcp --dport 443 -j MARK --set-mark 1

Verified with:

$ sudo iptables -t mangle -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
MARK       tcp  --  anywhere             anywhere             tcp dpt:smtp MARK set 0x1
MARK       tcp  --  anywhere             anywhere             tcp dpt:2525 MARK set 0x1
MARK       tcp  --  anywhere             anywhere             tcp dpt:https MARK set 0x1

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

create the table 4g in /etc/iproute2/rt_tables, verified with

$ cat /etc/iproute2/rt_tables
#
# reserved values
#
255 local
254 main
253 default
0   unspec
#
# local
#
#1  inr.ruhep
#
100 4g  # 4G modem static IP

and run:

ip rule add fwmark 1 table 4g

verified with:

$ ip rule list
0:  from all lookup local
32765:  from all fwmark 0x1 lookup 4g
32766:  from all lookup main
32767:  from all lookup default

It's when I get to running

ip route add default via 192.168.0.0 dev enx344b50000000 table 4g

that I'm no longer sure...

What command can I run to verify the existence of the default route in table 4g I've just created?

What should the output of normal ip route be ?

Currently the output of ip route is:

default via 192.168.0.1 dev enx344b50000000 proto dhcp metric 100 
default via 192.168.193.97 dev wlp2s0 proto dhcp metric 600 
169.254.0.0/16 dev wlp2s0 scope link metric 1000 
192.168.0.0/24 dev enx344b50000000 proto kernel scope link src 192.168.0.178 metric 100 
192.168.193.0/24 dev wlp2s0 proto kernel scope link src 192.168.193.131 metric 600

And netstat -r produces:

Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         _gateway        0.0.0.0         UG        0 0          0 enx344b50000000
default         _gateway        0.0.0.0         UG        0 0          0 wlp2s0
link-local      0.0.0.0         255.255.0.0     U         0 0          0 wlp2s0
192.168.0.0     0.0.0.0         255.255.255.0   U         0 0          0 enx344b50000000
192.168.193.0   0.0.0.0         255.255.255.0   U         0 0          0 wlp2s0

This is before adding the default gateway on table 4g and it remains unchanged after.

It seemed wrong to me that there are two default gateways. So I tried:

sudo ip route del default dev enx344b50000000

but that made no difference. Is this being interfered with by Ubuntu's dynamic routing? I'm also not sure what the 169.254.0.0/16 IP is either.

Thanks for any further help!

Answer 1

Yes, it can be done, and you will have a lot of fun learning all about the nitty gritty of routing on Linux. From your question I gather that that's part of why you are doing this.

You will need to look into ip route and especially ip rule. Here are some issues you will run into.

• For inbound traffic things look easy: make postfix listen on your static interface and your other services on the DDNS interface. However, if you don't force them, response packets will go out over either interface, and your client will see responses coming from the wrong IP (and ignore them).
• Outbound: ip rule and/or netfilter marking should be able to force traffic to specific ports over specific interfaces.

BUT here comes the big catch. Once you have it all running, you will very likely discover that your outbound email gets caught in spam filters, even when you set up SPF and DKIM (which you must).

The point is that your sending IP will be in an ISP customer range, with no reverse DNS lookup from your IP to your domain name (unless you can convince your 4G provider to set this up). In my experience, spam filters will penalise you severely enough to not make this a workable set-up.

Postscript

Also, don't underestimate the amount of spam and hack attempts you will have to deal with on the receiving end. They'll run in the thousands a day - all eating your expensive 4G data. It'll definitely be a fun learning experience, but you'll need time on your hands.

Answer 2

It's policy-based routing, specifically. Yes, a combination of iptables and ip route can do it.

• Configure Ethernet (eth0) and USB dongle (enx344b50000000) interfaces correctly with their respective IP addresses.

• Set Up IPTables Rules: Use iptables to mark packets based on the destination port. (i.e. mark traffic for SMTP:)

iptables -t mangle -A OUTPUT -p tcp --dport 25 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -p tcp --dport 465 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -p tcp --dport 587 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -p tcp --dport 993 -j MARK --set-mark 1

This adds rules to the mangle table, marking packets destined for ports 25, 465, 587, and 993 with a mark of 1.

Create a custom routing table in: /etc/iproute2/rt_tables by adding:

100 4g

This creates a table named 4g with an arbitrary ID of 100.

Add routing rules to direct marked packets to use the 4g table:

ip rule add fwmark 1 table 4g
ip route add default via <4G-Gateway-IP> dev enx344b50000000 table 4g

Replace <4G-Gateway-IP> with the actual gateway IP of your USB dongle:

If your 4G ISP doesn't provide you with a public IP address directly on your dongle interface, you might need to set up Source NAT for traffic going out via the dongle to ensure responses come back to the right interface. This can be done with iptables in the nat table:

iptables -t nat -A POSTROUTING -o enx344b50000000 -j MASQUERADE

As you said, you will need to have two DNS A records: one for mail.mydomain.tld pointing to your 4G static IP, and another for mydomain.tld pointing to your home IP (updated via DDNS).

• Ensure your ufw or iptables firewall rules allow incoming connections on the required ports for both interfaces.

• Testing - You can use tcpdump to monitor the traffic and confirm it’s taking the right path.

You may need a startup script to make sure the rules are applied consistently!

---

Update 1:

Check the default route in table 4g with -t to specify the table:

ip route show -t 4g

Which should be something like:

default via <4G-Gateway-IP> dev enx344b50000000 table 4g

The ip route show command without the -t option show's you the default route in the main routing table, which is usually ID 254.

The 169.254.0.0/16 IP is a link-local address that is automatically assigned to an interface when it is not configured with a static IP address. It is used for communication between devices on the same link, and is not routable on the internet.

It's common to have multiple gateways on a system. The kernel will use the default gateway with the lowest metric value when there's multiple. In your case, default gateway on the enx344b50000000 interface is 100, while the wlp2s0 interface is 600.

The kernel will use the enx... as the default gateway, when available, and will use wlp2s0 interface as default gateway when enx... is not available.

The 192.168.0.0/24 and 192.168.193.0/24 entries in the routing table show that your system is configured with two subnets, 192.168.0.0/24 and 192.168.193.0/24, and that the default gateway for each subnet is 0.0.0.0 and 0.0.0.0, respectively.

Here's a breakdown (more helpful, certainly) of the output:

default via 192.168.0.1 dev enx344b50000000 proto dhcp metric 100

This is the default route, which specifies that packets destined for any IP address that is not explicitly listed in the routing table should be sent to the gateway at 192.168.0.1 using the enx... interface. The proto dhcp specifies that the route was obtained through DHCP.

default via 192.168.193.97 dev wlp2s0 proto dhcp metric 600

This is another default route, which specifies that packets destined for any IP address that is not explicitly listed in the routing table should be sent to the gateway at 192.168.193.97 using the wlp2s0 interface.

169.254.0.0/16 dev wlp2s0 scope link metric 1000

This is a route for the 169.254.0.0/16 subnet, which is a link-local subnet assigned to the wlp2s0 interface. The scope link specifies that the route is only relevant to the local network segment.

192.168.0.0/24 dev enx344b50000000 proto kernel scope link src 192.168.0.178 metric 100

This is a route for the 192.168.0.0/24 subnet, which is assigned to the enx... interface. The proto kernel specifies that the route was obtained through the kernel. The src 192.168.0.178 specifies the source IP address of the packets sent through this route.

192.168.193.0/24 dev wlp2s0 proto kernel scope link src 192.168.193.131 metric 600

This is a route for the 192.168.193.0/24 subnet, which is assigned to wlp2s0. The src 192.168.193.131 specifies the source IP address of the packets sent through this route.

And it's no problem at all, my friend. Hope this helps!

Note: You may need IP forwarding on the server to allow traffic to be routed between the two network interfaces:

echo 1 > /proc/sys/net/ipv4/ip_forward