Since most anthropologists believe there was no slavery until well into the Neolithic, I guess it's going to depend on when those megalithic structures were built, but it's a myth (concocted by Hollywood) that the ancient Egyptians used slave labor to build the pyramids, so not sure if there is any reason to believe other sacred sites would have been built by enslaved people.