A little surprised to read that, especially since about 95% of the ones I shoot are in hay fields. Granted, not all hay fields are created equal. Good drainage is critical, so if there aren't allot of rocks in with the soil, you won't find many/any groundhogs in my experience. Even in a given field, groundhogs tend to take-up residence in the same few spots that offer the best conditions. As "good" as other spots might seem to be, they never dig holes in them for some reason.
Back to the drainage issue...I suspect that is key. I find lots of holes on the sides of hills, never find them in hollows/low spots. Damp conditions no good, dry/sunny spots ideal, etc.