The relationship between tau deposition and cognitive decline in cognitively healthy older adults is still unclear. The tau PET tracer 18F-MK-6240 has shown favorable imaging characteristics to identify early tau deposition in aging. We evaluated the relationship between in vivo tau levels (18F-MK-6240) and retrospective cognitive change over 5 years in episodic memory, processing speed, and reasoning. For tau quantification, a set of regions of interest (ROIs) was selected a priori based on previous literature: (1) total-ROI comprising selected areas, (2) medial temporal lobe-ROI, and (3) lateral temporal lobe-ROI and cingulate/parietal lobe-ROI. Higher tau burden in most ROIs was associated with a steeper decline in memory and speed. There were no associations between tau and reasoning change. The novelty of this finding is that tau burden may affect not only episodic memory, a well-established finding but also processing speed. Our finding reinforces the notion that early tau deposition in areas related to Alzheimer's disease is associated with cognitive decline in cognitively unimpaired individuals, even in a sample with low amyloid-β pathology.