Work hard at it all. Not just the physical.
If you want gains whether they be ones that others can see such as you getting quicker or becoming a better player for example, or ones that they can't see such as greater knowledge of how you put it all together on race or game day, you have to work hard at the basics. Some of those basics require physical effort, some of them require mental effort. They go hand in hand and without the understanding that you need to achieve on and off the track, trail, road, park, pitch, course or route, you won't get there consistently.
It's not rocket science but it is a different way of thinking. Invest in the one percents to achieve your goals and perform at a consistently higher level as a result.