Frazer's most notable action during the Civil War was when he surrendered the Cumberland Gap and was captured in the autumn of 1863. He was held as a prisoner of war until after hostilities ended in the spring of 1865, and then took up work in farming and business.