I ride all of the major horse trails, particularly along the Vienna/Hunter Mills section routinely on my CX (but not the narrow side-of-trail-trails). I’ve never had anyone so much as look at me askance, particularly since they were the trail detour for so long. Just be careful entering/exiting the trails as some of them require crossing the flow of traffic.
Normal rules of dirt apply- yield to everyone, stop cold for horses and don’t ride them muddy.
If you are already in that section and want to put your handling skills to the test and avoid pretty much all people, jet up the DR trail to Colvin Run and into LFP.