How to hide use of filtering on a given port? (nmap -sS targethost shows "filtered")

I have a software listening on port 1234 that I would like to keep hidden from outside world. I know that I can say e.g.

sudo iptables -A INPUT ! -s 1.2.3.4/32 -p tcp -m tcp --dport 9000 -j 1234 --reject-with icmp-port-unreachable

to make the port 1234 accessible only for the remote host 1.2.3.4.

However, other hosts can still run nmap -sS myhost, nmap -sW myhost or some other advanced test to see output such as

PORT     STATE    SERVICE
1234/tcp filtered someservice-guessed-from-port-number

For basic nmap scan (as normal user instead of root) the port 1234 appears to be closed.

Note that this output is identical regardless if any software is actually running on port 1234 or not.

Is it possible to use iptables to make the port 1234 appear totally identical to normal closed port for all hosts except the selected ones? I know that I could modify all ports to appear as filtered but that seems like a workaround instead of true solution.