I had considered the doing the same thing. We took a water taxi to Caye Caulker for the day. I have to admit it was a bit too quiet for us. We were there in the off season and even had a bit of a hard time finding somewhere we wanted to eat lunch. (Rasta Pasta was closed, darn.) I'd recommend a day trip and see what you think.