I used to be kind of scared of rudiments. I didn't understand why other drummers got all worked up about them. I saw them do crazy things, but when I sat down on a pad I didn't really get anywhere. I had a major breakthrough when I realized that every note you play is a rudiment. It's not about sounding like a Pratt piece or something, any time you hit a drum with your right stick, followed by something else with your left, you are playing single strokes. If you play two rights in a row, there's a double. Almost everything we play boils down to singles, doubles, some layered together as all sorts of flams, some multiple bounce strokes, buzzes and some other striking/special techniques that you could absolutely transcribe and connect on paper. It does not matter which limb, sound source, combination, layering, dynamic level or rhythm of the strokes. Everything you play is a rudiment, the serious question of if they need special attention depends on how much time and thought you are willing to spend organizing and focusing on your pad technique using the useful tools we have around like the standard 40 rudiments. I know it may seem like a lot of time, but if you spent 15 minutes on singles a day at one tempo, 15 on doubles at one tempo and 30 on an extra PAS rudiment that you would really take your time with and perfect over weeks or months at a time, you'd see improvement. The key is STICKING TO A PLAN. Have a method, write out the order of rudiments and tempos you want to work on, take your time, be patient and don't jump ahead before you're ready. However long it takes, I guarantee you that being able to sit with a metronome and run through the 40 rudiments at tempos you know you're comfortable with certainly will not damage your playing in any way. Dafnis Prieto has a great quote, something to the tune of "a drummer is only as good as his snare drum playing."
Good luck whatever you decide to work on!
Besides, if I ever question whether I should spend the hours on the pad, I think about all my favorite drummers and realize that all all of them spent way more time on a pad than I have up to this point and I' better get to work before I get hit by a bus or something!