_Anyone_ averaging 20MPH on the trails is probably riding unsafely. There are plenty of people who can do it on non-e-bikes, and more who can do it on e-bikes. It’s not the “e”, but the behavior that makes it a problem.
On roads, in order to average 20MPH for the trip, you’re probably talking about averaging 25mph or so while you’re going, to make up for stop signs, etc. There are plenty of people who can do that on non-e-bikes (but not as many as 20mph), and again, more people who can do that with e-bikes. So e-bikes make that commute accessible to more people.