If you like playing tourist, the Inner Harbor is the place. But, as others have mentioned, there are much more interesting places to go. Yes, the aquarium is worth it.
Little Italy is walking distance from the Harbor, and well worth the time. Have dinner at one of the many decent restaurants there (Da Mimmos - expensive, Caesar's Den - moderate, Amicci's - reasonable) and than head over to Vaccaros for dessert. It's a great Italian bakery and ice cream shop.
Fells Point is definitely worthwhile. It's where they filmed the Homicide TV series.
Little Greece is also interesting for the food and insular atmospherics. The Greek delis there are great.
Also, as mentioned, Camden Yards is right near all the sights and not far from the Babe Ruth house and museaum.