It can never be on the VPN if the outer, encrypted VPN packets are not allowed on the connection. I mention it because it is one of the more complex bits to handle about your requirement, depending on if you always connect to the same IP via VPN or need to connect to a dynamic peer, possibly one you need to look up via DNS first.
Ah, so you don’t care about software running locally on the router being able to access things without the VPN but do care if routed traffic does? You might also want to consider if the router offers services like a caching DNS server, a HTTP or SOCKS proxy,… that might look like local traffic to the network layer because a local process initiates connections at the instruction of some other system.