Any advice on how to get my mom to relax more when she is having medical tests?

This is a tough one, because anxiety is such a large part of Alzheimer's disease for most people who have it. Likely some of her anxiety is caused by confusion and memory issues, so if you can write down what will happen at the appointment and why they do each test, that may help. Let her keep the list for referral.
Some people don't like to be touched by strangers, and this could worsen with dementia. Obviously, that can't be helped during a doctor appointment. Let the doctor know ahead of time about your concerns so that he or she can make any adjustments that are possible. If lying down for tests would help, maybe they can do that. My heart goes with you. You may want to check with her primary doctor to see if a mild sedative would be wise. Watching this kind of struggle and the worry that goes with it is one of the hardest parts of caregiving.
Take care,
Carol