We volunteered live-in care for a family member who'd been completely abandoned. Any advice?

DH's family abandoned their bachelor 84yr old uncle due to a family spat. He chose the wrong side. 2 years ago he almost died. He has no dementia. Just health issues and fear of being alone. I couldn't stand that he was completely alone. He had to call a renter for help. His closest relatives are 3 minutes from his door. We offered to move from my entire family, older children, new grandbaby, parents, sisters and all their kids, an hour away to be his care givers. He had lived alone for years. His house was disgusting. It took a team of family members 2 months to make it livable. Before we moved in he asked if we would retire there on the family home place. He had his will changed to leave it to DH. DH, youngest son (who is away at college except 2 months tops) and our HS daughter moved in. Since then, 2 years and a 2nd near death experience have passed. In his mind, we were homeless and he took us in. Our 2nd son is living in and keeping up our home. It's on my family farm. We are making the house payments for that house, (it's for a family of 6 not a young guy starting out). My son is keeping it up so we can move back "someday". He is also helping care for my mom (I should be there doing that! Not a 23yr old grandson). We are paying for so many things that the old farm house doesn't have "or need" like further cleaning, up keep a safe bathroom and internet for school. We are struggling. His uncle is ugly to me but not in front of anyone else. He's a bigot and sexist (please don't say it's the age he was raised in, I know PLENTY of people his age who would NEVER use the words or phrases he uses.). I have begun complaining (as if you couldn't tell from above rant). My DH, college son and daughter have heard some but don't believe how ugly he can be. They get upset with me. I'm the one left here when they leave. I'm the caregiver for a man who doesn't need it. I'm the housekeeper for a man more disgusting than all 3 of my boys and all their friends rolled together ever were. I have adult children. I should be enjoying my granddaughter. I should not have to worry if he washed his hands before he touches her or her toys. I should not have to bleach hardwood because she is crawling or buy new carpeting for the basement to keep her clean. All because he won't wipe his shoes when he comes in from parking in the barn where Goodness Knows What deadly chemicals were spilled over the last 100 years.
I don't know how to suck it up. I don't know how to deal with someone who is ugly like this. I'm not used to being dismissed by my family. Now, I'm not sure if this is a rant or actual question.