This isn't "my" logic - it's what RAINN and other organizations who study this stuff say. Your personal data set of women you know, notwithstanding, lots (but clearly not all) of sexual violence is perpetrated by repeat offenders. Ergo, the percentage of men who commit sexual violence does not necessarily correlate with the percentage of women who experience it. From my perspective, the crux of the problem is the men who uphold rape culture and turn a blind eye to the actual perpetrators, whether they themselves are engaged in sexual violence or not - they are the ones who maintain the culture that allows it to be so rampant.