My mom is showing signs of dementia, and my brother cannot handle this and is yelling at her. What can I do?

If you don't want to move mom, and you can't get brother out, you're kind of stuck. I too have a very loud, overbearing brother who "knows everything". That's who mother lives with. It is VERY hard to talk to mother if he is helicoptering and having his 2 cents in every conversation.
It sadly sounds like you are in a no-win conundrum. Brother isn't going to listen to you, he lives with mom and she's unhappy, but you get involved and he blows up. Poor mom!
It really does sound like moving her and selling the home it the only thing to do. Everyone wants their parents as independent as possible as long as possible, but maybe you need to look at this situation with fresh eyes. If she isn't happy now, and showing signs of dementia, what makes you think she's going to be happier with this dynamic just continuing on and on? Calling APS will probably enrage your brother. Your mother sounds clearly capable of making her own decisions, yet she cannot stand up to brother. She needs your support! Better to do it now (move her) when she can be a real part of the decision making, than later when she cannot. Brother is a big boy and should not be bullying and intimidating people. Your mother is actually being abused, she should NOT be afraid to go to her own home!!
Good luck with this!

They are co-dependent. Should you move him out she will be wailing about being alone. Since he is the cop, let him decide when she should stop driving. You can take the man out of the force, but you can't take the force out of the man. Be aware that mom may be using "divide and conquer" strategy. I'm willing to bet she complains to him about you. Keeping you two at odds makes her able to control both of you. My MIL was able to keep a 3 way split going for years. Finally all 3 called each other, compared notes and got on the same page.

When you say senior living- do we look for retired people (who are still healthy) or edlery, who will soon need care. We don't want her in a home yet. We want to push that off as much as we can.

Look into a small senior living (ind. living) situation with transportation options for her. Put her house on the market and bank that money for her future care. Brother dearest now needs to find other options.