I was taking my mother to visit her sister ( she's 94) who happens to have Alzheimers. The trips were starting to drain me physically and mentally as well. I have my own disabilities, in fact I may have suffered another heart attack just recently, these trips were just becoming too much for me to handle. My aunt is in
excellent physical condition, in fact, her cardiac function is probably better than mine. I don't think I'm being selfish, I just want to live a little while longer, it that's possible.
With your health issues to consider, do you have a backup person to care for your mom? Maybe that person could take her and relieve you of some of your responsibilities.
It's great that you are able to recognize that this is something that really bothers you. It's no disgrace and I would heed the signals that this kind of thing is not for you. My cousin has friends and relatives who have never visited her, so, it's a common problem. It can depressing. Don't blame yourself.
Are you sure this is something that your mother is up to? Is she okay with the visits or is she bothered as well? I'd explore that just to make sure.
If letting mom visit her sister while you go elsewhere isn't an option, do you think they would be able to face chat or skype? Does your aunt have someone on her end who can set it up and supervise it? If they were able to just hello every week, maybe the need for frequent in person visits might decrease.
Depending on your aunt's progression, I'm amazed that a long visit would be productive. My cousin, does pretty well for about 20 minutes, but after that, she gets tired, distracted unfocused, etc. Would your mom accept that a shorter visit would be advisable? Who's in charge of your aunt? What is her position on the visits?