It came with patriarchy, which didn't arise until about 5k years into agriculture. As noted in The Dawn of Everything, "What happens if we accord significance to the 5,000 years in which cereal domestication did not lead to the emergence of pampered aristocracies, standing armies or debt peonage, rather than just the 5,000 in which it did?"

Read more about that here:

Human History Shows a Long Heritage of Personal Autonomy