14th Amendment says otherwise. Representation is based on "the whole number of persons in each State, excluding Indians not taxed.,"
Of course, it also strongly implies that determining such eligibility is the right of the states', and allows for an abridgement of representation in proportion to the abridgement of voting rights. The 14th is a pretty weird amendment.