Despite the importance of antigen-specific T cells in infectious disease, characterizing and tracking clonally amplified T cells during the progression of a patient's symptoms remain unclear. Here, we performed a longitudinal, in-depth single-cell multiomics analysis of samples from asymptomatic, mild, usual severe, and delayed severe patients of SARS-CoV-2 infection. Our in-depth analysis revealed that hyperactive or improper T-cell responses were more aggressive in delayed severe patients. Interestingly, tracking of antigen-specific T-cell receptor (TCR) clonotypes along the developmental trajectory indicated an attenuation in functional T cells upon severity. In addition, increased glycolysis and interleukin-6 signaling in the cytotoxic T cells were markedly distinct in delayed severe patients compared to usual severe patients, particularly in the middle and late stages of infection. Tracking B-cell receptor clonotypes also revealed distinct transitions and somatic hypermutations within B cells across different levels of disease severity. Our results suggest that single-cell TCR clonotype tracking can distinguish the severity of patients through immunological hallmarks, leading to a better understanding of the severity differences in and improving the management of infectious diseases by analyzing the dynamics of immune responses over time.