James Banks was born on July 27, 1900, in Columbus, Mississippi, at Bent Oaks Plantation. He came from a family with military history - his grandfather was Colonel James O. Banks who fought in the Confederate Army. As a young man, Banks attended Franklin Academy and Stephen D. Lee High School in his hometown while living at the Banks Harris Home (White Arches). He then went on to study at the Tennessee Military Academy before graduating from Annapolis in 1925.