In general, people often suggest to replace them if you notice that the pads are getting holes in them, because that'll lead to button presses being detected when you're not pressing them, or buttons not being springy enough, etc. But if you don't feel like the buttons feel good on that controller, replacing the contact pads will definitely fix the problem, pending the quality of the replacement you get. There are some guys I've seen on YouTube that sell aftermarket ones for retro gamepads, but I can't attest to the quality of them. Though I do need to get some replacements for my NES pad pretty soon.