Bagby reached his career high of 17 wins in each of his All-Star seasons, in 1942 and 1943, and led the American League in starts both years with 35 and 33, respectively. After that, he served much of 1944 in the US Merchant Marine and never again won more than eight games in a regular season.
As a hitter, Bagby was a better-than-average hitting pitcher in the majors, posting a .226 batting average (140-for-620) with 59 runs, 3 home runs and 56 RBI. He was used as a pinch hitter 18 times in his career.
Following his baseball career, Bagby became a professional golf player. In 1992, he was inducted posthumously into the Georgia Sports Hall of Fame.