People are blaming me because I'm my brother's health directive. They say I did things wrong, what can I do?

We need more information before we could offer advice. Who are the “people” who are blaming you? What are they saying you did wrong? Can you add a little more information to your post?

Without knowing all or any details all I can say is...
Did you follow what his wishes were or are?
Did you do the best that you could under the circumstances?
If you had to do it all over again given the same circumstances and without the benefit of 20/20 hindsight would you have made the same decisions?
Would you have wanted the same decisions made for you in the same circumstances?
If you would not have changed anything then you did the best you could given that you are human.
I do not know who the "people" are but if your brother trusted YOU and not them then rest easy. Forget what "people" say. We are always so worried about what others think that we forget to trust ourselves. Your brother trusted you now you need to trust in yourself.

Walk away when they start making verbal accusations. Hang up the phone. File the texts and emails away without reading them. Pointless discussions about the past don’t solve anything. It might help them to air their complaints, but if it doesn’t help you, don’t participate.
Sometimes we find ourselves in situations that require us to make difficult decisions. All we can do is do the best we can.
I’ve suffered this way for the past few years. I feel like I have been beat up emotionally and take advantage of financially. I’ve done the best I can. I’ve sought consel from professionals and knowledgeable others, but I didn’t choose to listen to young adults who think they know it all. Now I enjoy not being in their company.
Some things are not up for discussion. Some people aren’t fit for social interaction.
One authority says I’ve experienced Elder Abuse from these yahoos going off about what I’m doing wrong - as I struggle.
Tell them you don’t want to hear it and keep your car parked so you can hit the street. Sometimes a slow drive through the country is the best thing.

Belong to organization, that helps disabled person, my brother is Intellectual disabled age 72, I'm 66,  I was diagnosed with Borderline Intellectual Functioning, have been taking care of his needs we have been living together 2003.  I'm physical disabled too.  These people want to take the health directive away from me for them to decide was best for him. I'm also his Representative Payee, Social Security OK for me to take care of his money, these people want to take that away from me, one thing is Social Security told me to have my name on his accounts.  What I did wrong, not letting doctors give him strong meds when he was in the hospital for UTI, complaining his knees, ankle and back pain.  I hope this helps, sorry it's long.

Butterfly, sounds like you've been doing your best for your brother. He is lucky to have you!

Is there a social worker or another person like that you could talk to about this, someone not from the organization that is wanting to change your brother's health directive?

This lady who called me on the phone, told If I tell anyone what she said to me, and it goes to the wrong persons. She'll  tell a health corporation lies about me, that I have mental problems, that I'm the wrong person to be in charge of my brother.   Since February, I told his counselor, that my brother has been in bed, not being mobile to walk around, they forgot about that, If he didn't have a health problems, I'm the one called the ambulance, for him to go to the hospital, it's like the whole county knows about me, that's what she said!   When she called me, I have caption call phone, I wrote what she said to me.  I have a lawyer, called he said he wants to get in touch with Adult Protected Services, too many lies, I do feed my brother, he's undernourished and fragile, very picky eater, I'm also in fault because he's sick, going to get IHHS, also have home health.