Human bodies can and very often are sexually attractive, but the difference is that in cultures (and Western subcultures) where nudity is common, it is not automatically equated with sexuality. Or, if in the case of say, swinger's clubs, even if it is equated with sexuality, it's done in such a way as to view sexuality as a normal aspect of being a human being, rather than the prudish but also sexually fetishized way that often takes place in Anglo-American cultures (and some others).

Sexual feelings neurologically can and often are "switched off" in inappropriate circumstances - something I've already talked about quite a bit. Read up on the the Dual Control Model of sexual response and the Sexual Inhibitor system in particular for more. The notion that men can't do this because they are at their mercy of their penises is both mysandric and absurd. What's taking place (as already noted in the OP) is that men in cultures that equate nudity with sexuality have not been taught to consider attractive women they encounter in non-sexual or non-dating scenarios as something that should trigger their SI system - they've been taught the exact opposite.

We constantly message the culture that it's inappropriate to be sexual around or in front of children - but no such messaging takes place around an attractive stranger on the street (even if she's 9 years old) because "that's what women are for" - even if she's 9 years old. If you lived in a culture that didn't automatically equate nudity with sexuality and didn't objectify women on a mass scale in the media, this doesn't mean that nudity or attractiveness would never equal sexual feelings, it would mean that they were more likely to be context specific and something that you had more inner resources to manage.