Mom vomits when I try to brush her teeth. Should I stop?

"the dentist warned me about her dental hygiene and plaque."

Your dentist is insane. Seriously your mother has stage 6 Alz I think worrying about plaque build up is ridiculous. Skip the flossing and try and brush her teeth as best as you can without making her puke.

Please, oh please, let me make my Final Exit BEFORE I develop late-stage dementia or any disease that introduces the possibility of others needing to brush my teeth, the use of feeding tubes, "thickened" liquids and the like! If mom still had any awareness whatsoever, I think she'd likely be shouting "STOP!" with the toothbrushing at this point. If mom isn't already there, comfort care only (nausea/pain/anxiety meds) would be this 86 Y/O's vote.

Coughing a lot while eating means she is having trouble swallowing and the vomiting is caused by gagging . Mom needs a swallow eval by speech pathology and recommendations as to what consistencies of food and liquid that she can tolerate . She may also not being chewing enough . She may need chopped or purée foods and thickened liquids . Mom’s dementia is advancing , I would not get crazy about the teeth brushing . Or perhaps do not use toothpaste as it’s apparently going down her airway and making her cough, gag, vomit . All this food and toothpaste going down her airway could cause aspiration pneumonia , which is way worse than plaque on teeth at this late stage of life .
They also have those soft disposable foam mouth swabs for comatose patients you could try . Yes Mom can lose weight from vomiting .

There comes a time when dental care isn't really a priority and less than perfect is good enough, I'd go with minimal brushing just to ensure there is no pocketed food and skip the flossing altogether. Dipping the toothbrush or an oral sponge in a peroxide based mouthwash and being gentle rather than scrubbing away with gobs of toothpaste may reduce the gag reflex.

And as the others have already pointed out she needs a swallow evaluation ASAP, she almost certainly needs a modified diet.

https://www.agingcare.com/articles/dysphagia-how-to-help-a-loved-one-eat-and-drink-safely-187010.htm

https://www.agingcare.com/articles/swallowing-disorders-tips-for-using-food-and-beverage-thickeners-208048.htm

It really makes no sense to me that old-old people cannot legally accomplish a Final Exit with their personhood and dignity more or less intact--IF that's their choice. I'm not there yet but I could find myself facing that unwelcome predicament in the foreseeable future. We have the "live to 100" crowd pushing for ever-increased longevity. Yet, there is absolutely NO plan as a society for how to care (or PAY) for the rapidly escalating population of potentially addled, frail, incontinent, hearing, vision and mobility impaired old-old people.

I'm right in the middle of The Silent Generation (YOB 1937). We were a small cohort to start with. Not many of us may see our 90th b'day. That may not be the case for millions of the Baby Boom generation, yet society is no better prepared or equipped for them to reach old-old age than it was for us Silents. Some politicians are looking for ways to CUT Social Security and Medicare. A growing number of homeless Americans are 55+. Now THAT is scary--or should be!

It sounds like she's having issues with swallowing, which can cause aspiration pneumonia and be fatal. Her doctor can test her to see if that is the issue. I'm surprised the nurse didn't suspect that as it's quite common with folks who have any of the dementias.
My late husband(who had vascular dementia)had aspiration pneumonia and his doctor said because of his dementia that his brain was forgetting to tell his throat to close when he was eating or drinking, thus allowing it all to go into his lungs and not his stomach.
And yes my husband almost died from it and developed sepsis and septic shock leaving him completely bedridden and under hospice care for the last 22 months of his life. It's nothing to mess around with.
I would make sure that all of her foods are now pureed and her drinks thickened with the product Thick-It.
And as far as the brushing of her teeth goes, I personally wouldn't worry about it at this point as I think there comes a point as the disease of dementia progresses that one has to pick their battles carefully and I don't think brushing her teeth and making her vomit is a battle worth fighting at this point. Otherwise just get her front teeth best you can.
Please have your moms doctor order a swallowing test for her ASAP.

Has she had a swallowing assessment? If not, ask for it to be done asap.

Thank you everyone. I will get the doctor on it. I didn't think about the aspiration pneumonia issue until you all brought it up.

Try a swab, they make ones for this purpose.
They are foam and very slightly ribbed so they can be used for any number of things.
Clearing food from the area between the gums and cheeks. (Many people begin "pocketing" food)
The swabs can be used to brush the teeth and the roof of the mouth. BUT if this causes her to gag and begin to vomit stop. Vomiting can lead to aspiration pneumonia.
And getting to the fact that she is vomiting...is she having problems swallowing liquids or is it food? If it is liquids they may have to be thickened. With dementia often the body can not close off the trachea when swallowing so food enters the lungs. I hope the food is being cut into pieces or possibly even pureed if needed.
I think at this point you no longer have to worry about dental hygiene.
I doubt that she would sit still or be compliant for any dental work anyway. (Heaven forbid that the dentist suggests pulling any teeth! Trying to care for mouth wounds, any wounds on someone with dementia is almost impossible without them picking, poking and doing all sorts of stuff. Not even going to get into anesthesia and dementia

When I was pregnant I gagged and threw up when I tried to put a toothbrush in my mouth. The finger brushes worked well.