They didn't go down last year, though, even according to this graph.
Most racist acts are not crimes, either. If you and your buddy are talking about Lebron and call him the n-word, that won't be included in the statistics. But my uncle who swears he isn't racist said that last week.
If you're using hate crime charts to advocate for the country not being racist, you should also realize that very few agencies are actually reporting hate crimes. Wray talked about this in depth at his last hearing. Here is the most recent graph I can find.