I don't think you did anything wrong. Someone who is that freaked out that you have a life that isn't about him when you just met him is probably not someone it would have worked out well with anyhow.