The CHA₂DS₂-VASc score is a validated predictor of ischemic stroke in atrial fibrillation (AF) patients. However, data are limited on whether the CHA₂DS₂-VASc score is associated with subclinical brain structural changes or physical frailty among older AF patients. We assessed the relationship between CHA₂DS₂-VASc scores and brain structural changes or physical frailty in AF patients without history of stroke. Overall, 117 patients completed a comprehensive geriatric assessment for physical frailty. In brain magnetic resonance imaging sub-study (n = 49), brain volume and white matter hyperintensity lesion burden were automatically quantified using the LESIONQUANT software program. Patients with high risk of CHA₂DS₂-VASc scores (≥ 2 in men or ≥ 3 in women) tended to be older and had more comorbidities, higher frailty index, and slower gait speed. Total white matter hyperintensity lesion burden was higher in those with high risk of CHA₂DS₂-VASc score than in those with intermediate risk (score of 1 in men or 2 in women) of CHA₂DS₂-VASc score (1.67 [interquartile range: 0.70-3.45] vs. 0.64 [0.19-1.44], p = 0.036). Cognitive function was associated with brain volume, but gait speed was related with white matter hyperintensity lesion burden. In conclusion, we showed a positive relationship between CHA₂DS₂-VASc scores, white matter hyperintensity lesion burden, and physical frailty in older AF patients. Subclinical brain changes associated with high CHA₂DS₂-VASc scores may predict physical frailty risk.