Gender roles and expectations fuck everyone, it’s true. Nevertheless women are still seen as inferior, and femininity is seen as inferior to masculinity.
Thus a man wanting to be feminine or an outright woman is seen as transgressing cultural ideals, if not outright insane (for wanting to be “lesser”). Being a trans man is more understandable to some people.
Trans men might also have an easier time blending in or passing, due to the facial feature-hardening effects of testosterone. It’s probably less of a factor though.