Previous DTI studies have reported associations between white matter integrity and performance on the Stroop interference task. The current study aimed to add to these studies of inhibitory control by investigating how the differences in age and in white matter integrity relate to Stroop performance, and to examine whether the effect of age on Stroop performance is mediated by white matter integrity. 179 healthy adults from 20 to 80 years old were recruited in the study. DTI data were processed through TRACULA and the mean fractional anisotropy (FA) of 18 major white matter tracts were extracted and used for statistical analysis. Correlation analysis showed a strong negative relationship between age and the Stroop interference score (I). Higher I indicated better inhibitory control. Simple linear regression analyses indicated that most of the tracts showed negative relationships with age, and positive relationships with I. Moderation effect of age on the relationship between FA and I was tested on tracts that significantly predicted I after multiple comparison corrections, but none of these moderations were significant. Then we tested if these tracts mediated the effect of age on I and found significant indirect effects of age on I through the FA of the left corticospinal tract and through the right inferior longitudinal fasciculus. Our results highlight the role of a number of major white matter tracts in the processes supporting the Stroop inhibitory performance and further pinpointed the lower white matter integrity of specific tracts as contributors to the decrease in inhibitory control ability associated with the Stroop test in older age.