Taller individuals walk faster but it is unknown whether this advantage persists at older ages. We examined the cross-sectional/longitudinal associations of height with gait speed (GS) in participants from the Dijon-Three-City cohort study (France) over 11 years. In 4011 participants (65–85 y), we measured usual/fast GS (6 m) up to five times. We examined whether the baseline height-GS association varied with age using linear regression, and whether height influenced GS change using linear mixed models. Taller participants 65 y at baseline walked faster than shorter ones (fast GS difference between top/bottom height quartiles, 0.100 m/s, P < 0.001); this association weakened with age (P-interaction = 0.02), with a 0.012 m/s (P = 0.57) difference at 80 y. Ten-year fast GS decline was 51% greater (P < 0.001) in younger participants in the top height quartile (−0.183 m/s) compared to those in the bottom quartile (−0.121 m/s), leading the GS difference between the two groups to be attenuated by 50% over the follow-up. The height-related difference in fast GS decline was not explained by time-dependent comorbidities or height shrinkage. Analyses for usual GS yielded consistent findings. The height-GS relation is more complex than previously thought, as the height related advantage in GS disappears as persons grow older due to faster decline in taller compared to shorter persons.