My 89-year-old mother, who has vascular dementia, hates her caretakers. What is the best way to stop her aggressive behavior?

Your mom's situation is hard for you to detach from, but you must for your own sake.
It's all about your mother's dementia. She can't help her illness and you want to do your best for her, but you can't sacrifice your life, your health, or your marriage for her increasing needs. None of those sacrifices will help anyone.
It may be easier to put up with her increasing lack of reality if she is in a good AL, though she may need to be in one with a memory unit. The staff in these facilities are trained to cope with dementia, and your mother's attitude won't be that different than that of many other people they help.
You'll have to make a decision and stick with it, whether it's hired in-home care, AL or even a nursing home, if that is where her health puts her. You can be a part of her life - daily, if that works for you. You will be an important part of her care. But you will also have a life of your own. It seem like you are due.
Your mother's challenges will just increase. She will, unless other health issues take her soon, need more care that you can give, just to keep her safe.
It seems to me this is decision time. Good luck. I know it's hard, but tough choices abound in elder care.
Carol

Your Mom's attitude toward any type of care will never change. The Dementia has robbed her of her cognitive abilities and social graces. So I would recommend finding a solution that works well for all of you.
What Carols says is true. Regardless of your parent's ailments, they will only get worse and their medical needs will increase to the point beyond your expertise.
There are so many options that did not exist awhile ago. There are smaller facilities that cater to memory needs. The follks their are used to residents who have outbursts and other extreme behaviors. They know how to redirect and help them live quality lives without the use of heavy meds.
After reading so many posts on this site, I have come to the conclusion that a parent living with his or her children may not be the best choice. I noticed that, while my Mom was getting excellent physical care from us, she was isolated and did not have much social contact because of her mobility issues. She now has several people, in addition to family, to interact with now.
Trust your instincts. If you are feeling burned out make some changes now. Not being able to have a vacation or even a getaway is grueling...I know because I did the same thing. You really need a "team" to help take care of an ailing parent...there is absolutely nothing wrong with this. You are letting the experts handle the medical/behavioral issues so you can go back to being the loving and helpful family. It's a win-win.
(btw, You mentioned that assisted living did not work. How much time did you give it? Was it a memory care facility? It usually takes about 6 months for a new place to feel like "home." )
good luck