I do the same thing for the group rides. We rides two or three abreast fast and form single line when call out “Car Back”. We’re not blocking the roads. The drivers have to slow down for few seconds before they can pass. Some drivers are not happy about slow down before passing or don’t want to see the cyclists on the road.
There are a lot impatience drivers in this DC area. When I rode in Portland OR, this is not an issue at all. When I ride on trail that cross the road, all drivers would stop and yield. I didn’t have to wait.
I think the drivers need to be more educated and aware of cyclists on the road.