Part of the free-market attitude though is that you should be allowed to buy policy, so in that regard it’s consistent, you just have to account for corruption in the cost of doing business.

If people are ok with that then I guess it will stand, but it’s insane and anti-consumer in my book. A product costs what it costs, based on supply and demand, and if you can’t afford it you don’t buy it. This flimsy premise of “It lowers the bar to entry so users can upgrade later without having to replace!” will never come to fruition, and it’s too slippery of a slope to “put in a quarter to turn on your A/C”.

That is insane. If it costs the same to make, then lower range isn’t a reasonable area to pitch a lower cost vehicle. Wanting to lower the cost is fine. Putting in cheaper/smaller components to get there is fine. If you are using the same components and just software locking them to nickle and dime the users later, that’s anti-consumer and should not be tolerated. I can’t believe how people look at micro-transactions in games and think “wouldn’t this be cool with IRL stuff?”

My setup is a bit extreme, but here are my guardrails:

1. All users have the same UID’s on every system. I’m 1000, wife is 1001, son is 1002, daughter is 1003. All these exist on all systems. Our primary group is “family” (gid 10000). Our files are all owned by user:family. This matters because we let them have access to the share of things like home movies and pictures, and I have a TrueNAS with an NFS mount that their user folders rsync to nightly for backup. If you wanna get crazy, you can put in a whole LDAP/freeIPA setup, but that’s a lot (and I did all that as a learning experience).
2. They don’t have the account passwords. I have their password, and if they want to use it, the wife or I have to type the password. When we want them off, superkey+L to lock the computer, and if they reboot it comes to a login screen.
3. If you really go this route, and go the whole LDAP thing, you can also tie that into apps like Jellyfin. I have a huge library of movies and shows, but there’s a folder called “KidMedia” and I literally manually symlink things to that folder if I want them to have access. I set up the phones/tablet with their own jellyfin accounts, and when they log in they only see their media. I also NFS mount that share, so for the same reason, they can watch stuff on VLC from the computer with access control. We also do that with nextcloud, so we can use nextcloud talk to chat internally. The tablets/phones have built in android controls, so the idea is once they’re on their device, they’re free within the ecosystem I set up and they don’t enter credentials other than device unlock.

I built my kids potato computers from the time they were 3-5, which was during covid. They need computer skills nowadays, and it put them at an advantage for covid school. We got them on java Minecraft which was huge for reading, typing, and some basic math skills (they figured out multiplication for crafting things like doors). I made a chart which had icons of things they want, with the word next to it, so they could search and type in creative.

We used Ubuntu Mate. It’s simple, stable, and familiar. They do NOT have sudo on these boxes. As we’ve advanced, they now have firefox (behind a pihole which upstreams to opendns’ family protect), gimp (with a wacom tablet!), inkscape, calculators, tenacity, libre office, and they’re starting to get into some cad to make things to 3d print. You have to come to terms with doing a LOT of patient hand holding, but it has paid off dividends.

Yes, but it’s not an option yet. We’re heavily invested in veeam and are not looking to replace that piece yet.

Does that need to be true though? For like true “counting in how many 9’s” HA of course. But there’s nothing technically preventing high availability in 2 nodes; if the storage is shared and there’s a process to keep the memory in sync it should be possible with 2 nodes have some degree of high availability, even if it’s with big warnings.

Those are all fair, also the entire open-vswitch setup is very clunky. I always avoid the UI and just edit /etc/network/interfaces directly, especially for vlan networks. I dislike that it wants 3 nodes but I understand, still 2 nodes in the homelab is pretty reasonable. I wish in general the HA was more configurable, robust, and intuitive.

Yes, we are a medical/dental/pharmacy university and because of some of the specific data needs of our org we have a large on-prem ecosystem. We are currently a VMWare shop, but Broadcom’s business strategies have made us look for alternatives. I’ve used Proxmox in the homelab for years and have been feeling as its gotten more and more polished it’s ready to be considered for production work. Currently we have a lab environment of previous gen hardware which I want to use as a test-bed for possible production platform moves.

Proxmox isn’t VMware yet, but it’s close. The HA doesn’t work the same, I’ve struggled with something akin to DRS. If you use on-host storage, you have to constantly do replication work to keep them synced and even then a failover is essentially a storage rollback to the last sync. If you use iscsi storage, you have to be very careful. Snapshotting is only functional when backed by a few of the storage types, and we use ZFS. ZFS over isci is somewhat brittle, but we have a TrueNAS device which supports it here. We use Veeam as our enterprise backup solution, and I have no idea how these will work together. Veeam talks directly to our Nimble storage, does storage-based snapshots, and replicates them to our other site. Veeam theoretically does talk to TrueNAS, but without supporting Proxmox I don’t know what the backup/recovery flow would look like. Veeam is looking into this: https://community.veeam.com/discussion-boards-66/veeam-researching-support-for-vmware-alternative-proxmox-as-backup-buyers-fret-about-broadcom-6530 We tried to use TrueNas ZFS snapshots for just general VM semi-backup, but unless you want to rollback your whole dataset, it doesn’t work well. You have to make separate snapshot tasks for the specific zvol/dataset, otherwise you’re rolling your whole dataset back. Also, I tried mounting a snapshot, hoping to then share it as an iSCSI extent and remount it to a VM and pull out a specific file…this didn’t work at all, I can’t get the UI to show the promoted clone so I can try to present it to the host.

When coming back from a power-off, if your Proxmox hosts are in a cluster, there’s no cluster-aware startup order (HA disables the entire startup delay system). That’s not great, our apps have SQL dependencies which need to be started first.

That’s the issues, and it sounds negative, but ultimately for a zero-cost hypervisor that’s under active development those issues need to be viewed through the lens of the overwhelming achievement that the project is and continues to be.

We’re converting our workplace lab to Proxmox and it’s a great ramp for eventually leaving vmware. Great system.

Huge bummer that they’re all 5+ years old. We’ve been moving to libreelec with Disney+, Jellycon, Netflix, Youtube, and amazon prime plugins. It’s not the same, but it’s workable. If Amazon keeps MatterCast open and open source implementations get made, that’s where I’m focusing my attention. A raspberry pi with libreelec that can be a casting target feels, to me, like the holy grail:

https://www.theverge.com/2024/1/9/24030324/amazon-matter-casting-echo-show-fire-tv-prime-video

No joke can you share those results? I’m holding out for matter cast

We tried to host it ourselves to save cost, and it’s a beast but it mostly works. It certainly lags behind in features and uses a lot of resources, but when you compare with the cost it’s certainly passable.

He is allowed to state his personal opinions despite being the leadership of one of the most populous state. I appreciate that opinion.

Ok, good news, I re-imaged and after about an hour of tinkering it’s working. (My wife is a doctor who does tele-medicine from home so it was tricky to get a downtime, even riskier if I couldn’t get back to working; usually she works when kids are in bed and that usually my window for these kind of projects). I still have my old config backup; I have a lot of firewall rules and services to put back in (I had redirects for google trying to reach their dns from chromecasts to my pihole, I had a zabbix client pointing to my zabbix server, I had wireguard working and want to see if I can restore existing key exchanges, it was tied to my LDAP server, etc). I really want to compare my old backup with a new one when this is done and see if I can’t figure out what was broken. I want to document that because I found a bunch of people with similar questions that only had incomplete answers:

1. From the CLI, the WAN interface was DHCP, I set up the lagg between my 2 ports (lagg0), created a vlan 99 interface off of it (lagg0_vlan99) and made that the LAN interface with a static IP and no gateway.
2. I made a gateway for my 10.99.1.254 LAN gateway, had to assign it to the LAN interface when I made it. It is not tagged as upstream. One thing I noticed, WAN_GW is priority 255; it was 254 before. Just a difference I noticed.
3. I made an alias for each of my VLANS that might need internet access
4. In Outbound NAT, I switched it to Hybrid and made rules to allow traffic through to each VLAN.
5. Under Firewall->Rules->LAN I created a pass rule for each VLAN (This will get tuned later)

With this, LAN clients access the WAN, after putting in a port forward WAN clients can access things on the LAN, the firewall can ping both LAN and WAN.

If I go to my LAN interface and set the gateway to “LAN_GW” at 10.99.1.254, everything works (but I can’t ping anything on the LAN from the firewall itself, including the client I’m ssh’d from). If I set that to Auto, all LAN clients lose WAN access.

I’ve got a backup, but I think I’m gonna try to rebuild from scratch :/ I just worry I’m gonna end up in the same spot since I don’t understand how it all got here and don’t know what to avoid.

I’m totally with you…and I have that, which is why I think I’m hitting some sort of bug, or a firewall rule that is somehow breaking this:

I probably need to burn it down and restart, but I need to find a time the family will tolerate an extended outage. I did share some things on the opnsense forum though which might be useful here.

1. My diagram. At the bottom you will see why I have /16; in truth, it’s from back when I only had a single subnet, and I made it /16 so I could use the third octet to form DHCP scopes. That’s how the network worked in my head and I knew the IP scheme, so when it came time to add VLANS much later, I just made those the 2nd octet, and that’s how we are here today. Maybe one day I’ll re-do that, but it’s not in scope right now: https://nextcloud.surfrock66.com/s/txnZdzxHaiA5t65
2. I did an experiment with static routes last night. I have the static route in, so I untagged the “LAN_GW” as an upstream gateway, and tagged “WAN_GW” as an upstream gateway. No change in the ability for opnsense to ping anything (it can ping WAN, not LAN), however all my LAN clients lost internet. In this state, from opnsense, I ran a “ping -S 10.99.1.40 10.2.2.213” (that’s my DNS server). This failed, but interestingly enough I was looking at the live logs, and even though the interface is LAN, the source IP was the WAN IP. I’m very confused; I’ve confirmed the LAN and WAN interfaces are correct and they have correctly assigned default gateways. See the attached picture. This would make sense; is opnsense doing something to switch the LAN and WAN somehow? I’m blown away how this is the case; that being said, it makes sense that tagging the LAN interface as upstream allows traffic out.

It feels like somehow opnsense is treating LAN like WAN or something? I don’t know the obfuscation feels like it’s hiding things. A “ping -S 10.99.1.40 10.2.2.213” shouldn’t show in the logs with a source of the WAN address, right???

Ok, it’s definitely an issue with the firewall not sending traffic from itself to LAN. It’s weird, it’s passing traffic, but it cannot ping or access anything on the LAN including things on the 99 VLAN (so it’s local VLAN). The DNS requests are for sure failing from the firewall…but they work fine for the rest of the LAN. Any client can get a DNS response from the DNS server on the 2 VLAN, and can access the resulting site.

For now, I’m just excluding the wireguard thing, I think it’s a distraction to the problem that the firewall has some sort of bad routing going on.

I have a diagram, but at this point it’s pretty local to the firewall itself and I think it’s around the gateway/route configuration. I got some advice on the opnsense forum that my static routes are wrong; they say to make a single static route of 10.0.0.0/8 instead of one for each VLAN, turn off “upstream gateway” on the LAN GW (which when I do that I lose all WAN connectivity…which is a concern but I can revert). When I do the cli configuration, and I assign an IP for LAN, it asks if I want to put a gateway; it kind of says “it should be yes for wan, but no for LAN” but if I do no, I can’t access the internet from any clients, and if I do yes, it ticks “upstream gateway” on the lan gateway. Something is awry, but I’m gonna try again after making some static route changes.

Ok, lot to go over. The /16 thing is just history; before I started this, I actually had a full /16 for my whole house as I thought I’d have hundreds of IoT devices one day, and used that third octet as a logical separator. I’ve kind of got that stuck in my head, so when I moved to a 10. system, I made the vlans/subnets the 2nd octet because I have so much IP memory of that third and fourth octet. It’s unnecessary, but tbh I know most of my IP’s by heart, and I went into this trying to drive complexity up a bit to further my learning. I don’t think necessarily changing them to /24 would solve the problem, because the complexity wouldn’t really collapse much. It’s things like the 3 network is for our minecraft servers/services, and 10.3.2.* represents the main one, 10.3.3.* represents the one my son runs, etc. It’s just muscle memory at this point. The L3 stuff is mostly good I think, I’m mostly concerned about the firewall.

I know that opnsense can be the L3 device, but 1) part of my learning in this was to use kind of raw cli switch commands and not some web UI, and 2) I had the original L3 device before adding the opnsense box (I used to have a comcast modem as the upstream from the L3, now the L3 has a 0.0.0.0/0 route to opnsense, and that should upstream to the comcast device). I have a full VM dedicated as my DHCP/DNS device running bind9 and isc-dhcp-server which has been maintained for over 10 years; I’m not looking to offload that to another device and it works flawlessly on the lan (with an IP helper on each vlan).

I am definitely confused how it does gateways. My understanding is, in opnsense, gateways are the part of the route definition, so you define the opnsense gateways to point to the gateways on the L3 device, they’re not on the opnsense box itself. When you add an interface, you select the default gateway for that interface from a dropdown, consisting of the gateways you defined elsewhere. Where I get goofed up and lock myself out is when I change the “upstream” checkbox or mess with the priority. I don’t know how it selects one or the other as “active” either. I’ve iterated on that a lot; the further I get, the more it feels like the obfuscation of opnsense is adding to my complexity rather than reducing it.

It seems the only thing having routing problems are packets essentially originating from an interface on the opnsense; things on the LAN reach the WAN, things on the WAN reach the LAN, but wireguard clients terminating at the opnsense box can’t hit the WAN, and the opnsense box can’t hit the LAN (despite passing traffic).