Maybe I am not thinking of the access control capability of VLANs correctly (I am thinking in terms of port based iptables: port X has only incoming+established and no outgoing for example).
I think of it like this: grouping several physical switch ports together into a private network, effectively like each group of ports is it’s own isolated switch. I assume there are routers which allows you to assign vlans to different Wi-Fi access points as well, so it doesn’t need to be literally physical.
Obviously the benefits of vlans over something actually physical is that you can have as many as you like, and there are ways to trunk the data if one client needs access to multiple vlans at once.
In your setup, you may or may not benefit, organizationally. Obviously other commenters have pointed out some of the security benefits. If you were using vlans I think you’d have at a minimum a private and public vlan, separating out the items that don’t need Internet access from the Internet at all. Your server would probably need access to both vlans in that scenario. But certainly as you say, you can probably accomplish a lot of this without vlans, if you can aggressively setup your firewall rules. The benefit of vlans is you would only really need to setup firewall rules on whatever vlan(s) have Internet access.
Out of curiosity, what switch are you using for your setup?
Last time I looked, I struggled to find any brand of “home tier” router / switch that supported things like configuring vlans, etc.