Google suggests you go ahead and use its Disavow Links tool if you know of bad links you have out there, even if Google has not penalized you. If you only have a couple, don't worry about it, but the more you have, the more you'll want to do it (again, according to Google).
Here's what Google's head of webspam Matt Cutts told Rae Hoffman about it on Twitter:
@sugarrae only thing I'd add is if it's 1-2 links, may not be a big deal. The more it gets close to "lots," the more worthwhile it may be.
— Matt Cutts (@mattcutts) March 13, 2014
Okay, if you "know" you have "bad" links, why not? The problem is that people often don't know which ones are actually "bad," and as we've seen in the past, people go hog-wild on getting backlinks removed just because they're afraid Google won't like them, regardless of whether or not there is evidence of this.
Cutts also said when the Disavow Links tool came out that most people shouldn't use it.
Here he is talking about when you should worry about your links. Should you spend time analyzing your links and trying to remove ones you didn't create that look spammy? The "simple answer is no."
And here's Google's "completely clear" stance on disavowing irrelevant links.
Image via YouTube