I'm being made to go into a patients house and he verbally abuses me and my co-worker. Is there anything I can do?

He is of sound mind and refuses care.  I'm being made by management even though we have reported it several times. Other carers are also refusing now to go in. All we do is walk his dog and feed his dog he refuses all personal care.