ufw won't put custom rule in the correct place at reboot

Question

My general issue is that I lose contact with my Ubuntu 23.10 on ssh once I close my ports using knockd. I would like for it to maintain existing connections.

I have a custom rule

> iptables -I INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT

that will fix my issue when added. When I try and add the rule to

> ufw/before.rules

as the first possible rule so that it will load that way at boot time, upon

ufw reload

the rule will appear in the #2 position as it should and knockd performs as expected/required.

But upon reboot, ufw will place my custom rule into the #4 position and then knockd fails to work as expected, until I issue the

ufw reload

command. Then my custom rule will appear in the #2 position and the #4, while knockd behaves as it should.

yoda@email:~$ sudo iptables -L --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination         
1    f2b-sshd   tcp  --  anywhere             anywhere             multiport dports ssh
2    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
3    f2b-ufw    tcp  --  anywhere             anywhere            
4    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
5    ufw-before-logging-input  all  --  anywhere             anywhere            
6    ufw-before-input  all  --  anywhere             anywhere            
7    ufw-after-input  all  --  anywhere             anywhere            
8    ufw-after-logging-input  all  --  anywhere             anywhere            
9    ufw-reject-input  all  --  anywhere             anywhere            
10   ufw-track-input  all  --  anywhere             anywhere            

Chain FORWARD (policy DROP)
num  target     prot opt source               destination         
1    ufw-before-logging-forward  all  --  anywhere             anywhere            
2    ufw-before-forward  all  --  anywhere             anywhere            
3    ufw-after-forward  all  --  anywhere             anywhere            
4    ufw-after-logging-forward  all  --  anywhere             anywhere            
5    ufw-reject-forward  all  --  anywhere             anywhere            
6    ufw-track-forward  all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
num  target     prot opt source               destination         
1    ufw-before-logging-output  all  --  anywhere             anywhere            
2    ufw-before-output  all  --  anywhere             anywhere            
3    ufw-after-output  all  --  anywhere             anywhere            
4    ufw-after-logging-output  all  --  anywhere             anywhere            
5    ufw-reject-output  all  --  anywhere             anywhere            
6    ufw-track-output  all  --  anywhere             anywhere            

Chain f2b-sshd (1 references)
num  target     prot opt source               destination         
1    REJECT     all  --  agmk.uz              anywhere             reject-with icmp-port-unreachable
2    REJECT     all  --  178.128.84.59        anywhere             reject-with icmp-port-unreachable
3    REJECT     all  --  124.156.200.144      anywhere             reject-with icmp-port-unreachable
4    REJECT     all  --  162.62.135.19        anywhere             reject-with icmp-port-unreachable
5    REJECT     all  --  167.172.103.180      anywhere             reject-with icmp-port-unreachable
6    RETURN     all  --  anywhere             anywhere            

Chain f2b-ufw (1 references)
num  target     prot opt source               destination         
1    REJECT     all  --  scan-43n.shadowserver.org  anywhere             reject-with icmp-port-unreachable
2    REJECT     all  --  45-79-145-84.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
3    REJECT     all  --  143-42-1-52.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
4    REJECT     all  --  104-237-156-209.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
5    REJECT     all  --  143-42-1-123.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
6    REJECT     all  --  173-255-221-22.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
7    REJECT     all  --  194.33.191.29        anywhere             reject-with icmp-port-unreachable
8    REJECT     all  --  45-79-92-218.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
9    REJECT     all  --  80.66.83.49          anywhere             reject-with icmp-port-unreachable
10   REJECT     all  --  79.110.62.153        anywhere             reject-with icmp-port-unreachable
11   REJECT     all  --  79.110.62.184        anywhere             reject-with icmp-port-unreachable
12   REJECT     all  --  recyber.net          anywhere             reject-with icmp-port-unreachable
13   REJECT     all  --  apzg-0721m-038.stretchoid.com  anywhere             reject-with icmp-port-unreachable
14   REJECT     all  --  carthage.scan.bufferover.run  anywhere             reject-with icmp-port-unreachable
15   REJECT     all  --  173-255-210-89.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
16   REJECT     all  --  131.150.216.162.bc.googleusercontent.com  anywhere             reject-with icmp-port-unreachable
17   REJECT     all  --  115.146.127.123      anywhere             reject-with icmp-port-unreachable
18   REJECT     all  --  143-42-164-204.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
19   REJECT     all  --  proxychecker.vultr.com  anywhere             reject-with icmp-port-unreachable
20   REJECT     all  --  apzg-0721-a-076.stretchoid.com  anywhere             reject-with icmp-port-unreachable
21   REJECT     all  --  192-155-84-194.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable
22   REJECT     all  --  79.110.62.78         anywhere             reject-with icmp-port-unreachable
23   REJECT     all  --  ip-58-18.4vendeta.com  anywhere             reject-with icmp-port-unreachable
24   REJECT     all  --  45-56-83-149.ip.linodeusercontent.com  anywhere             reject-with icmp-port-unreachable

Chain ufw-after-forward (1 references)
num  target     prot opt source               destination         

Chain ufw-after-input (1 references)
num  target     prot opt source               destination         
1    ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-ns
2    ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:netbios-dgm
3    ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:netbios-ssn
4    ufw-skip-to-policy-input  tcp  --  anywhere             anywhere             tcp dpt:microsoft-ds
5    ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootps
6    ufw-skip-to-policy-input  udp  --  anywhere             anywhere             udp dpt:bootpc
7    ufw-skip-to-policy-input  all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST

Chain ufw-after-logging-forward (1 references)
num  target     prot opt source               destination         
1    LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warn prefix "[UFW BLOCK] "

Chain ufw-after-logging-input (1 references)
num  target     prot opt source               destination         
1    LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warn prefix "[UFW BLOCK] "

Chain ufw-after-logging-output (1 references)
num  target     prot opt source               destination         

Chain ufw-after-output (1 references)
num  target     prot opt source               destination         

Chain ufw-before-forward (1 references)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
2    ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
3    ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
4    ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
5    ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
6    ufw-user-forward  all  --  anywhere             anywhere            

Chain ufw-before-input (1 references)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere            
2    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
3    ufw-logging-deny  all  --  anywhere             anywhere             ctstate INVALID
4    DROP       all  --  anywhere             anywhere             ctstate INVALID
5    ACCEPT     icmp --  anywhere             anywhere             icmp destination-unreachable
6    ACCEPT     icmp --  anywhere             anywhere             icmp time-exceeded
7    ACCEPT     icmp --  anywhere             anywhere             icmp parameter-problem
8    ACCEPT     icmp --  anywhere             anywhere             icmp echo-request
9    ACCEPT     udp  --  anywhere             anywhere             udp spt:bootps dpt:bootpc
10   ufw-not-local  all  --  anywhere             anywhere            
11   ACCEPT     udp  --  anywhere             mdns.mcast.net       udp dpt:mdns
12   ACCEPT     udp  --  anywhere             239.255.255.250      udp dpt:1900
13   ufw-user-input  all  --  anywhere             anywhere            

Chain ufw-before-logging-forward (1 references)
num  target     prot opt source               destination         

Chain ufw-before-logging-input (1 references)
num  target     prot opt source               destination         

Chain ufw-before-logging-output (1 references)
num  target     prot opt source               destination         

Chain ufw-before-output (1 references)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere            
2    ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
3    ufw-user-output  all  --  anywhere             anywhere            

Chain ufw-logging-allow (0 references)
num  target     prot opt source               destination         
1    LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warn prefix "[UFW ALLOW] "

Chain ufw-logging-deny (2 references)
num  target     prot opt source               destination         
1    RETURN     all  --  anywhere             anywhere             ctstate INVALID limit: avg 3/min burst 10
2    LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 10 LOG level warn prefix "[UFW BLOCK] "

Chain ufw-not-local (1 references)
num  target     prot opt source               destination         
1    RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
2    RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type MULTICAST
3    RETURN     all  --  anywhere             anywhere             ADDRTYPE match dst-type BROADCAST
4    ufw-logging-deny  all  --  anywhere             anywhere             limit: avg 3/min burst 10
5    DROP       all  --  anywhere             anywhere            

Chain ufw-reject-forward (1 references)
num  target     prot opt source               destination         

Chain ufw-reject-input (1 references)
num  target     prot opt source               destination         

Chain ufw-reject-output (1 references)
num  target     prot opt source               destination         

Chain ufw-skip-to-policy-forward (0 references)
num  target     prot opt source               destination         
1    DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-input (7 references)
num  target     prot opt source               destination         
1    DROP       all  --  anywhere             anywhere            

Chain ufw-skip-to-policy-output (0 references)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere            

Chain ufw-track-forward (1 references)
num  target     prot opt source               destination         

Chain ufw-track-input (1 references)
num  target     prot opt source               destination         

Chain ufw-track-output (1 references)
num  target     prot opt source               destination         
1    ACCEPT     tcp  --  anywhere             anywhere             ctstate NEW
2    ACCEPT     udp  --  anywhere             anywhere             ctstate NEW

Chain ufw-user-forward (1 references)
num  target     prot opt source               destination         

Chain ufw-user-input (1 references)
num  target     prot opt source               destination         
1    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:ssh
2    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:http
3    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:https
4    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imap2
5    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:imaps
6    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3
7    ACCEPT     tcp  --  anywhere             anywhere             tcp dpt:pop3s

Chain ufw-user-limit (0 references)
num  target     prot opt source               destination         
1    LOG        all  --  anywhere             anywhere             limit: avg 3/min burst 5 LOG level warn prefix "[UFW LIMIT BLOCK] "
2    REJECT     all  --  anywhere             anywhere             reject-with icmp-port-unreachable

Chain ufw-user-limit-accept (0 references)
num  target     prot opt source               destination         
1    ACCEPT     all  --  anywhere             anywhere            

Chain ufw-user-logging-forward (0 references)
num  target     prot opt source               destination         

Chain ufw-user-logging-input (0 references)
num  target     prot opt source               destination         

Chain ufw-user-logging-output (0 references)
num  target     prot opt source               destination         

Chain ufw-user-output (1 references)
num  target     prot opt source               destination

How do I get knockd to load my rule at boot time to the correct position in my firewall?

Answer 1

So,

cat /lib/systemd/system/ufw.service

shows that ufw starts before the network is up (Before=network.target). But,

cat /lib/systemd/system/knockd.service

shows that knockd doesn't start until after the network is online (After=network-online.target).

You can create a simple systemd service that runs ufw reload after knockd is running.

Run the following command to create and edit a new systemd service file (or use your favorite text editor):

sudo nano /etc/systemd/system/ufwreload.service

Copy and paste the following into the file:

[Unit]
Description=Reload ufw after knockd is started
After=knockd.service

[Service]
Type=oneshot
ExecStart=/usr/sbin/ufw reload

[Install]
WantedBy=multi-user.target

When you are done editing, press CTRL+o to save the file and then press CTRL+x to exit nano.

Then, run the following commands to enable your service to automatically start when knockd is running after a reboot.

sudo systemctl daemon-reload
sudo systemctl enable ufwreload

The only complication I can think of is that the service might not start if knockd has issues starting up so please post any errors or issues in the comments — you might be able to change After=knockd.service to something else like After=NetworkManager-wait-online.service as a workaround.

---

Alternatively, you could run ufw reload as a cron job so many minutes or seconds after reboot to give knockd the opportunity to start up.

For example, run the following command to create and edit a new cronjob:

sudo crontab -e

Next, select your favorite text editor from the user prompt, or select the number for nano if you don't have a favorite text editor.

Then, copy and past the following at/to the end of the file:

@reboot sleep 180 && /usr/sbin/ufw reload

Save the file when you are finished and the ufw reload command should run 3 minutes (180 seconds) after your next reboot.

Use sudo crontab -e to edit your cronjob in case you need to adjust the start time after reboot (from 180 seconds to something else).

The issue you might run into here is that ufw reload will run at the set time, regardless of if knockd is running or your network is up or not. So you might occasionally need to run ufw reload again but the cronjob should prevent the need to manually reload ufw for the most part, assuming your set time isn't too soon after reboot (i.e., before knockd is running instead of after as it needs to be).