The efficacy and safety of medications can be affected by alterations in gut microbiota in human beings. Among antidiabetic medications, incretin-based therapy such as dipeptidyl peptidase 4 inhibitors might affect gut microbiomes, which are related to glucose metabolism. This was a randomized, controlled, active-competitor study that aimed to compare the effects of combinations of gemigliptin−metformin vs. glimepiride−metformin as initial therapies on gut microbiota and glucose homeostasis in drug-naïve patients with type 2 diabetes. Seventy drug-naïve patients with type 2 diabetes (mean age, 52.2 years) with a glycated hemoglobin (HbA1c) level ≥7.5% were assigned to either gemigliptin−metformin or glimepiride−metformin combination therapies for 24 weeks. Changes in gut microbiota, biomarkers linked to glucose regulation, body composition, and amino acid blood levels were investigated. Although both treatments decreased the HbA1c levels significantly, the gemigliptin−metformin group achieved HbA1c ≤ 7.0% without hypoglycemia or weight gain more effectively than did the glimepiride−metformin group (59% vs. 24%; p < 0.05). At the phylum level, the Firmicutes/Bacteroidetes ratio tended to decrease after gemigliptin−metformin therapy (p = 0.065), with a notable depletion of taxa belonging to Firmicutes, including Lactobacillus, Ruminococcus torques, and Streptococcus (all p < 0.05). However, regardless of the treatment modality, a distinct difference in the overall gut microbiome composition was noted between patients who reached the HbA1c target goal and those who did not (p < 0.001). Treatment with gemigliptin−metformin resulted in a higher achievement of the glycemic target without hypoglycemia or weight gain, better than with glimepiride−metformin; these improvements might be related to beneficial changes in gut microbiota.