I made this mistake, er, learned this lesson on my kid’s old bike: it won’t scour through the paint, but it will dull a glossy clearcoat.
Never again.
I’d expect it to cloud shiny metal as well, but I’ve never needed to find out – solvents and soft rags get those plenty clean.
(One of my bikes – the one I cosmetically abuse – has a steel frame with a matte white finish. It gets all kinds of black splotches from the brakes, drivetrain, cable rub, as well as a lot of mud and muck on the underside, not to mention a bit of oxide from the weep holes, and only melamine sponges can get it back to mostly whitish. They affect the finish on this bike as well, making it GLOSSIER than original, but it’s not noticeable nor do I actually care.)