"you can swim there but they have sea grass"
Besides urchins,shells, broken coral or other sharp objects that can be hidden in sea grass areas IMO some people don't like how shallow the water can be until farther out, it is not their typical image of a beautiful beach with clear blue water and/or they don't like the feel/look of it.
I believe that areas of sea grass are important ecological biomes alive with a variety of sea-life. Doesn't bother me at all.
Belize it!