There are many reports of live lyrics rolling out to YouTube Music for Android (version 6.15) and iOS (6.16) today. You might want to Force stop YTM from App info before checking, and be sure to try multiple songs if you don’t see it immediately.
This rollout is not tied to the updated Now Playing UI that adds comments that’s also being tested.
Only if you listen to a video with the “audio only” setting enabled, or if you’re searching for a song that isn’t available in the official library, with the video being the fallback.
The actual song library is a song library, no video.