Selfhosted

except that all requests’ IP addresses are set to the router’s IP address (192.168.3.1), so I am unable to use proper rate limiting and especially fail2ban.

I'd guess that however the network is configured, you have the router NATting traffic going from the LAN to the Internet (typical for a home broadband router) as well as from the home LAN to the server.

That does provide security benefits in that you've basically "put the server on the Internet side of things", and the server can't just reach into the LAN, same as anything else on the Internet. The NAT table has to have someone on the LAN side opening a connection to establish a new entry.

But...then all of those hosts on the LAN are going to have the same IP address from the server's standpoint. That's the experience that hosts on the Internet have towards the same hosts on your LAN.

It sounds like you also want to use DHCP:

Getting the router to actually assign an IP address to the server was quite a headache

I've never used VLANs on Linux (or OpenWRT, and don't know how it interacts with the router's hardware).

I guess what you want to do is to not NAT traffic going from the LAN (where most of your hardware lives) and the DMZ (where the server lives), but still to disallow the DMZ from communicating with the LAN.

considers

So, I don't know whether the VLAN stuff is necessary on your hardware to prevent the router hardware from acting like a switch, moving Ethernet packets directly, without them going to Linux. Might be the case.

I suppose what you might do

from a network standpoint, don't know off-the-cuff how to do it on OpenWRT, though if you're just using it as a generic Linux machine, without using any OpenWRT-specific stuff, I'm pretty sure that it's possible

is to give the OpenWRT machine two non-routable IP addresses, something like:

192.168.1.1 for the LAN

and

192.168.2.1 for the DMZ

The DHCP server listens on 192.168.1.1 and serves DHCP responses for the LAN that tell it to use 192.168.1.1 as the default route. Ditto for hosts in the DMZ. It hands out addresses from the appropriate pool. So, for example, the server in the DMZ would maybe be assigned 192.168.2.2.

Then it should be possible to have a routing table entry to route 192.168.1.1 to 192.168.2.0/24 via 192.168.2.1 and vice versa, 192.168.2.1 to 192.168.1.0/24 via 192.168.1.1. Linux is capable of doing that, as that's standard IP routing stuff.

When a LAN host initiates a TCP connection to a DMZ host, it'll look up its IP address in its routing table, say "hey, that isn't on the same network as me, send it to the default route". That'll go to 192.168.1.1, with a destination address of 192.168.2.2. The OpenWRT box forwards it, doing IP routing, to 192.168.2.1, and then that box says "ah, that's on my network, send it out the network port with VLAN tag whatever" and the switch fabric is configured to segregate the ports based on VLAN tag, and only sends the packet out the port associated with the DMZ.

The problem is that the reason that home users typically derive indirect security benefits from use NAT is that it intrinsically disallows incoming connections from the server to the LAN. This will make that go away

the LAN hosts and DMZ hosts will be on separate "networks", so things like ARP requests and other stuff at the purely-Ethernet level won't reach each other, but they can freely communicate with each other at the IP level, because the two 192.168.X.1 virtual addresses will route packets between each the two networks. You're going to need to firewall off incoming TCP connections (and maybe UDP and ICMP and whatever else you want to block) inbound on the 192.168.1.0/24 network from the 192.168.2.0/24 network. You can probably do that with iptables at the Linux level. OpenWRT may have some sort of existing firewall package that applies a set of iptables rules. I think that all the traffic should be reaching the Linux kernel in this scenario.

If you get that set up, hosts at 192.168.2.2, on the DMZ, should be able to see connections from 192.168.1.2, on the LAN, using its original IP address.

That should work if what you had was a Linux box with three Ethernet cards (one for each of the Internet, LAN, and WAN) and the VLAN switch hardware stuff wasn't in the picture; you'd just not do any VLAN stuff then. I'm not 100% certain that any VLAN switching fabric stuff might muck that up

I've only very rarely touched VLANs myself, and never tried to do this, use VLANs to hack switch fabric attached directly to a router to act like independent NICs. But I can believe that it'd work.

If you do set it up, I'd also fire up sudo tcpdump on the server. If things are working correctly, sudo ping -b 192.168.1.255 on a host on the LAN shouldn't show up as reaching the server. However, ping 192.168.2.2 should.

You're going to want traffic that doesn't match a NAT table entry and is coming in from the Internet to be forwarded to the DMZ vlan.

That's a high-level of what I believe needs to happen. But I can't give you a hand-holding walkthrough to configure it via off-the-cuff knowledge, because I haven't needed to do a fair bit of this myself

sorry on that.

EDIT: This isn't the question you asked, but I'd also add that what I'd probably do myself if I were planning to set something like this up is get a small, low power Linux machine with multiple NICs (well, okay, probably one NIC, multiple ports). That cuts the switch-level stuff that I think that you'd likely otherwise need to contend with out of the picture, and then I don't think that you'd need to deal with VLANs, which is a headache that I wouldn't want, especially if getting it wrong might have security implications. If you need more ports for the LAN, then just throw a regular old separate hardware Ethernet switch on the LAN port. You know that the switch can't be moving traffic between the LAN and DMZ networks itself then, because it can't touch the DMZ. But I don't know whether that'd make financial sense in your case, if you've already got the router hardware.