James was born on a plantation called Laurel Hill in Patrick County, Virginia, on February 6, 1833, to Archibald Stuart, a politician and attorney, and Elizabeth Stuart. After attending school in Wytheville, Virginia, he attended Emory & Henry College from 1848 to 1850. In 1854, he entered the United States Military Academy at West Point, New York, when Robert E. Lee was its Commandant. When he graduated, he had obtained the rank of cavalry sergeant, the highest rank attainable for these cadets.
In the U.S. Army, he served as a Mounted Rifleman in Texas before joining the 1st Regiment of the U.S. Cavalry, where he was involved in several Indian conflicts and those involving the Kansas-Missouri border war, more familiarly known as Bleeding Kansas. In 1859, Stuart carried the orders for Colonel Robert E. Lee to proceed to Harpers Ferry, West Virginia, to stop John Brown’s raid on the U.S. Arsenal. Stuart was promoted to captain in April 1861 but resigned on May 14, 1861, to join the Confederate Army following the secession of Virginia.
He was commissioned as a Lieutenant Colonel of the Infantry in the Confederate Army the next month, quickly received several promotions, and became the Cavalry Commander. By July 1862, he received the rank of a Major General, fighting in several campaigns, including the Peninsula Campaign, the Second Battle of Bull Run, Antietam, Fredericksburg, Chancellorsville, and Gettysburg, before losing his life on May 12, 1864, during the Overland Campaign, at the Battle of Yellow Tavern.