I agree with the sentiment of if you didn't shoot it, why would you want it? In the case of a hunt camp or a gimmicky restaurant you might want a few antlers hanging around for whimsy, so if the shooter doesn't want it, then I guess it doesn't hurt to donate to that.
In the case of more than one person shooting the same deer, I've always gone with the first person to hit the deer gets the rack because he likely slowed it down so it could be tracked or shot by another hunter.