Agreed, but most drones have less than 10 minutes battery and a very limited range ( I just checked, the one I bought has less than 100m),and even the professional model the DJI Phantom II is only 25 minutes and only 20 minutes for the Lily Camera.
http://www.dji.com/product/phantom-2-vision/feature
I have had guys in Jet skis or boats be much more annoying and almost tip my canoe and a boat went by my father in law so close it actually dumped him (in the kayak) in the lake... and just kept going, someone else came along and helped him right the kayak and gave him a tow and ride back ( I had taken the kids fishing and was in another area).
Sad so many people have little respect for others anymore, look at those willing to destroy others property worth hundreds or thousands, people will pull up 10 feet off the end of docks and fish with music cranked, wake boats with huge speakers etc. etc. but in reality for a drone to be a bother it would have to be a very expensive model and multiple batteries. They are probably are just trying it out, want to know who is out there, if you had any luck, the usual curiosity, just wave and they will probably go somewhere else in a minute to two, no reason to act like an idiot... and if you do they will have the picture or video proof. If someone wants to take a picture of me fishing (or with luck catching a fish), why not it didn't hurt me and if they are a pro they might talk to me after and give me a contact to have them send me the pics, if not it is no different than the thousands of pics taken in a year... yawn.
I do and on a trip to China in the one area of the country where there are few tourists we had teen age girls, business men (from an even more remote area) and couples wanting pictures taken with our kids. They thought the kids were cute and somewhere in China there are pictures of our kids with complete strangers, SO WHAT! We got lots of bows and thank you and thought it was fun, odd but fun and I have a life and I guess but I just don't have the insecurities and paranoia some seem to have.