For what it's worth, it sounds like you have a very solid start to a great relationship on your hands, and I think you need to try and put a more positive spin on things. It many areas, I think you're reading too much into the situation, and maybe just need to be a bit more confident and go get what you want.
I totally get how agonizing it can be to wait what seems like forever to get a response to a message, but he may not be as locked into social media as other people. He may not be constantly checking his phone, or on the flip side, he may be a bit nervous (in a good way) about responding. If he's into you (which it sounds like he is), he may be choosing his words very carefully because he doesn't want to mess things up.
The transition from teammates and pals to a real relationship is tricky, especially given the amount of shared bonding and history you two already have. However, those moment also work completely to your advantage, as there's plenty to talk and laugh about. It's really just a matter of giving things time to settle in this new light, and letting those feelings build.
It really sounds like the trip to the music store was just a typical slightly-awkward first date, and I wouldn't let it discourage you from pursuing him further. Seriously – there always has to be a first time you hang out when it's not about swimming or school, and you've made it through that. The precedent for the two of you hanging out alone is set, so it shouldn't be weird to float the idea of getting together again.
As I see it: you need to go after what you want. Call him up and make a date to go do something fun, and it should be way less awkward and tons of fun.
Read more: