I stay away from conflict by being very pragmatic, that is, doing what needs to be done and only that. I would be neutral until someone crosses the lines you have drawn, and then take action from there. For example, if someone were to be violent I would just knock them out with one punch, but people arent violent with me because they know that it would be very very stupid to provoke me into a physical fight. If I were you, if one of them lays a hand on her other than a pat on the back, I would call the police.
Just carry on like normal and if one of your brothers steps out of line, dont hesitate to contact authorities. They will see a girl and her mentally ill brother being messed with and will undoubtedly take the side of a young woman and a mentally ill man.
Anyone can use words to try to hurt you, but we all have heard evil voices so like I give a ■■■■ when people think words hurt. Just DO NOT ACCEPT PHYSICAL ABUSE and call the cops if they harm or threaten either one of you. You two, the dynamic duo, Kid Sis and James don’t deserve any ■■■■■■■■ from your brothers.