Being restricted to slugs is FAR SAFER than using centerfire rifles in heavily populated area's as the range in which the bullet remains dangerous after missing the intended target is half that of the .243 example used. Then you have to factor in ricochets as a centerfire bullet will travel much father after a ricochet when compared to a slug.
Hunter takes a shot with his 20 gauge that's zero'd at 100 yards when the deer is 150 yards away and makes a clean miss. The slug loses steam very quickly and will be in the ground at around 250 yards or less.
Hunter takes a shot with his .243 at a deer that is 150 yards away and makes a clean miss. The bullet will retain velocity and energy for much longer meaning the bullet won't be in the ground until it's maybe 450-500 yards away, essentially doubling the distance that the bullet can potentially hit an unintended target.
Bullet A (20 gauge) will go 250 yards before hitting the ground.
Bullet B (.243) will go 500 yards before hitting the ground.
It seems very clear which one would have the higher risk of hitting an unintended target, no?