Well in the case of Clements you can only let him go if you´re seriously going to address the pass rush. Get guys that can rush the passer, situational players if you will, that can win one on one match-ups. Sure they´re hard to come by but if this defense (any defense actually) doesn´t have to send 6 guys after the QB everytime an average secondary will look like an all pro unit.
Sure address the vacancy left by letting him go but that doesn´t mean you have to draft a CB in the first round.