When Should I Replace the Rubber Contact Pads in Controllers?

I recently went back to playing my SNES for the heck of it and although the buttons are working fine, I don't really care for the squishiness I get when pressing down a button. Is this the right time to go find and buy some replacement contacts?

In general, people often suggest to replace them if you notice that the pads are getting holes in them, because that'll lead to button presses being detected when you're not pressing them, or buttons not being springy enough, etc. But if you don't feel like the buttons feel good on that controller, replacing the contact pads will definitely fix the problem, pending the quality of the replacement you get. There are some guys I've seen on YouTube that sell aftermarket ones for retro gamepads, but I can't attest to the quality of them. Though I do need to get some replacements for my NES pad pretty soon.