Never too late! I picked up the drums seriously at 18 (I'd dabbled a bit before that, but got sick of it) and five years later... well, I'm in a band, playing regularly, posting here and loving it!
As people have already said, just practice regularly. Broaden your musical mind by listening and watching as much as you can, and learn about gear and techniques through the wonderful people here... oh, and find a group of people to play with (who are better than you and so will stretch your playing). That way you'll be a well-rounded drummer in no time!