During months of often intense combat, he battled only fellow Missourians, ranging from Missouri regiments of U.S. Volunteer troops, to state militia, to unarmed Unionist civilians. The single confirmed instance of his exchanging fire with Federal troops from another state occurred a month after the 1865 surrender of Confederate General Robert E. Lee, during a near-fatal encounter with Wisconsin cavalrymen. In the course of the war, James' mother and sister were arrested, his stepfather tortured, and his family banished temporarily from Missouri by state militiamen— all Unionist Missourians.[4]