How can my mom's house go into probate if she has not died yet?

Probate is an after death filing.

Maggie - your backstory is that you cared for mom in her home for years and then after several falls she got hospitalization and your brother came in and became mom’s legal guardian and wants house sold.... is that it???

If so - assuming your brother went through the proper legal to become mom’s guardian & a judge awarded him guardianship - your brother is now totally in charge of mom’s situation. He may decide to sell home to pay for mom’s care. Unfortunately guardians priority is to his ward (your mom) & not you. All those family disfunction surface.
BUT if you are considered to be a tenant (in the house), then he would have to abide by whatever laws protect tenants. It could stall for months his ability to sell house or even put it in the market. Savvy renters can make selling property difficult.... like difficult enough they pay you to leave and live in another place for months. You need an atty to represent you in all this.
Oh & atty could get a order placed on your SIL from coming over so no more showing up to yell at you. Really not a DIY project, you need your own legal.

You can't probate until the person dies. If actual probate u need to call the Surrogates office and find out what is going on. Has Mom been paying her taxes, water-sewage bill. I think you need to find out who and why house is being sold.