Been caregiver for several years and now family is causing problems and bullying. Any advice?

I had stopped school and stopped working so I could be the caregiver for most of the day, unti my mom comes home from work. We have been living with grandparents for the past atleast 15 years. I’ve been my grandparents caregiver for the past several years. And my aunts weren’t involved and never had voiced their opinion. Now that my grandpa is dying they are talking about me and my mom behind our backs. And now they want to come in and take control and change things, and are having issues with us. My mom and I are done being bullied and are wanting to move out and stop doing it. I want to be there for my grandparents but don’t want to deal with the issues from my aunts and uncle.