What should I do with my brother who has trashed my mom's house? It looks like hoarders heaven.

Stop talking to him about how he is caring for Mom (or not). Give no hint that you are about to call APS to report him for Elder Abuse.

Are you hesitating because you know the responsibility for Mom’s care will fall on you? Your hesitation to report him indicates you are or this would already be a done deal. Make quick plans about Mom’s care when Brother is out of the picture. Can she stay with you for a time? Can she stay in her home with caregivers? Should you apply for Medicaid Nursing Home Care? Who has POA? If no one, consult an Elder Law Attorney and have one drawn up.

Brother isn’t going to have an epiphany and all of the sudden start being nice to Mom, taking care of her and paying rent simply because you “talk to him”. He needs to be gone from the picture for Mom’s health and safety.

You feel like you should evict him and report abuse.

So, why wouldn't you do that? I'm not criticising, just wondering what is making you hesitate.

Report elder abuse and then with a police officer along, evict him. Press charges if possible.

I have a brother who’s a long time hoarder. They don’t change, at least not without loads of psychological therapy, which they most often believe they don’t need. Talking to him is useless. Time to look out for mom. Your thoughts on having him evicted and looking out for her care are spot on

He is a "Nester" Getting him out may not be that easy
They may remove her faster than him.

There was a "Hoarders " episode similar to this, but it was a daughter and mom living in AZ. The daughter was a truck driver.

then like everyone is saying. call authorities and report. otherwise you also become part of the problem. what are the other family members doing?

get together with your family members and make a group complaint (?)
If he wont talk to you and he is isolating her from family - step up and step in quick.