Mom has ALZ and has decided she doesn't need to have "baby sitter" when Dad isn't home. She really can't be home alone. Suggestions?

Well obviously you have to have someone there, so you will just have to put up with her fussing over it. Would she be happier if you told her the caregiver was there for some other reason, perhaps to help your dad, (as he obviously needs it LOL)?
Have you talked to the doctor about her hallucinations/delusions? Perhaps she needs to be seen by a geriatric psychiatrist, there could be medications available to bring her back a little closer to reality.

Yes, talk to her doc about meds, but also start looking at a memory care facility for her. This is only going to get worse as time goes on . Be prepared.

I agree about possibly looking into meds, though I would do that if her behavior is really disturbing to her or harmful. If she's just annoying, I might just deal with that for as long as possible. If she doesn't get too anxious or worried, I might just tell her whatever works to relieve her annoyance. Maybe say the lady is lonely and loves her company. Or the lady needs a place to stay until her husband gets off work. Any explanation that might help her accept her presence, even if it's not true. As long as it distracts her and relieves her concern. Eventually, she is likely to forget why the lady is there and it might not even bother her. It's difficult to predict what the progression will bring.

Get a comprehensive geriatric evaluation and prepare to spend some quality time alone with the SW. You are going to need an active POA (probably requires incapacity letters) or even guardianship because Mom will not be able to make good decisions but will be entitled to unless something is established. If you can convince her not to throw out the help that is making it safe for her to be home instead of in a facility, great, but at some point she might put her foot down and not allow the supervision she needs which will be a huge problem. At least occasionally, treatment of some medical problem or medication adjustments will help at least for a while. Sorry this is happening to you and your folks!!