You're treating people as if they are only individuals instead of also (and in fact, mostly) a part of a society. You cannot be neutral as a member of a society the way you possibly could be in a dispute between two friends. Most racism in not overt hatred, it's implicit bias and structural inequality, and so if you are continuing as a more privileged person to benefit from that (whether or not you intend or want to) without doing anything to challenge it, you are not behaving in a neutral way at all.
Presumption of innocence is a legal term that only applies to who has the burden of proof during a trial. If the state presumed the person was innocent, they wouldn't be on trial in the first place.
"According to the U.S. Supreme Court, the presumption of the innocence of a criminal defendant is best described as an assumption of innocence that is indulged in the absence of contrary evidence (Taylor v. Kentucky, 436 U.S. 478, 98 S. Ct. 1930, 56 L. Ed. 2d 468 [1978]). It is not considered evidence of the defendant’s innocence, and it does not require that a mandatory inference favorable to the defendant be drawn from any facts in evidence."
The "fence" as you call it was created about 6-9k years ago when patriarchal dominance hierarchies first came into existence. Prior to that time, there was no class system, and humans lived in egalitarian groups/tribes/villages. Creating and maintaining a stratified social system requires that somebody be designated as the lower part of the pyramid of power and then kept in that place through coercion and violence. How can anyone in good conscience pretend that they are neutral in the face of that.
As Eli Weisel said, “We must always take sides. Neutrality helps the oppressor, never the victim. Silence encourages the tormentor, never the tormented.”
