It's all a matter of context. You're correct about the color scheme business.
Trademark issues are not blind to context. While the phrase "Who Dat" might be in the public domain, it is possible to trademark its use in the context of the New Orleans Saints. If someone is selling shirts in Saints colors outside the stadium, well I'd say the NFL has a case.
Here's an analogy: If your last name is McDonald, and you want to open a restaurant named "McDonald's", then you will probably be sued. You can argue that your family name existed well before the fast-food restaurant McDonald's opened, but it doesn't matter. McDonald's has the name trademarked in the restaurant business. You would be free to open an electronics store and call it McDonald's if you want, as long as there is no allusion of an association to the popular restaurant.
So, it doesn't matter that "Who Dat" may have existed before the Saints - the NFL has it with regard to the Saints.
Same goes for the Fleur-de-Lis. The NFL/Saints own the trademark with regard to the Saints, and any product that uses the fleur-de-lis in a context that alludes to the football team (in a judge's opinion) is probably infringing.