Here is a link to an MSDS for it - you can read for yourself on the toxicity and side effects. Basically, has not been found to be toxic or cancer causing, but breathing it can cause a number of effects as is says under the Inhalation item, and when it contacts heated surfaces like a furnace or water heater flame is can for acids. It also carries a lubricant with it in the air conditioning system (commonly mineral oil) and vaporized oils are known to be a significant respiratory hazard, both chemically and because the aerosolized oil coats the lungs causing edema and gradual suffocation.
Certainly a leak should not be allowed to continue, plus repeated filling of a leaking system constitutes a release of the gas that is prohibited under the Clean Air Act.
However, if leaking a pound every yesar or few, unlikely to have a noticeable effect unless you are breathing right at the leak point. I would say more likely, the respiratory effects are from inadequate filtration of dust and lint that is being recirulated in your home, or from mildew/mold growth on the evaporator coils. That is a common source of respiratory issues, as is forced air ventilation in general because it tends to pull inand recirculate more allergens from outside and inside than direct electric or steam/hot water radiator or baseboard systems.