What are the insurance issues that need to be considered when hiring a caregiver independently instead of through an agency?

You'll need worker's comp. Homeowner's policies do not cover anything approaching full time employees. They only cover "occasional workers". Like plumbers, the gardener and the pool guy. The policy should spell out what an occasional worker is.

Worker's comp is not cheap. But a necessity. Otherwise if the caregiver is injured in hour home leading to the loss of her ability to make a living, the consequences would be dire.

Make sure your Homeowners insurance will cover any accidents.
If this person will be driving your loved one, or using your car for any reason make sure they are covered and have no driving issues with the State.
Make sure you are taking out the proper taxes.
Workers comp and unemployment
Keep awesome record of her hours so there is no problems later with the number of hours worked.
Make sure you follow the proper schedule for the number of hours she can work. If you pay over time make sure that is documented

Please clearly find out if caregiver signed some type of a non-compete for a set period of time (6 months usually) with old agency.

Most agencies do these routinely along with a NDA (non disclosure agreement) for all workers. It’s enforceable.

The caregiver can go independent and work for the family or individual that previously they worked for as employee of the agency as states have “right to work” laws. But the non-compete can make them or you as the new employer pay a % of their earnings - which would have been the agency’s commission on the earnings - to the old agency. The agency will find out in some way..... likely the caregiver tells one of their old coworkers or send you a certified “demand” letter as to status on old caregiver... and will send followup letter from a law firm. Could you ignore it..?... maybe but it’s enough to get everybody flummoxed and worried so it gets paid.

First, call your homeowners agent, and make certain you are covered for liability.  Second, in many states, you will need workers comp and unemployment insurance.   These generally do not cost much.  You will also have to withhold tax and pay that in.   

You may be able to find a service to handle this stuff, I would not go to a CPA, I think that is overkill