I do too, and sometimes I even learn something, but it's got to be happening in good faith - which clearly that guy you describe was not - and which too many times is the case.

I used to never want to block for the same reasons you have cited, but it has absolutely helped my mental and emotional equilibrium to do it more often. One of the most recent ones was a guy who I was constantly proving wrong with statistics and reality checks that he couldn't refute, but it didn't stop him from coming back around on the next story and trying again to somehow "best" me. It just got tiresome. I don't want to spend my time and effort dealing with that. He wasn't rude or offensive (beyond having a bevy of completely wrong ideas) but he also had no interest in anything but his own off-base narrative that he couldn't actually support in any meaningful way - I think because he'd picked most of it up in manosphere chat rooms. I don't owe him access to my time or anything else and I'm much happier knowing that I won't have to deal with him any longer.

I highly recommend it.