The servants knew. The Confederate White House in Richmond, Virginia, was not a happy home. The coachman had heard Varina Davis, the first lady of the South, wondering aloud if the rebellion her husband led had any prayer of success. It was, he heard her say, “about played out.” Less than a year into the war, she had all but given up hope. And the president himself, Jefferson Davis, gaunt and sere, was under tremendous strain, disheartened and querulous, complaining constantly about the lack of popular support for him and his policies.

What the servants at the dinner table heard could be even more interesting: insights into policy, strategy and very private lives. They could glimpse up close the troubled emotions of Varina, who was much younger than her husband. She was in her mid-30s, he was in his mid-50s, and her energy, even her sultry beauty, were resented by many in that small society. She had a dark complexion and generous features that led at least one of her critics to describe her publically as “tawny” and suggest she looked like a mulatto.

Varina’s closest friend and ally in the cabinet was Judah P. Benjamin, the cosmopolitan Jewish secretary of war and then secretary of state. He was a frequent visitor to the Davis residence. He shaped Confederate strategy around the globe. And over port after dinner, what intimacies might have been revealed about this man, whose Louisiana Creole wife lived in self-imposed exile in Paris, and whose constant companion in Richmond was her beautiful younger brother?

As in any of the big households of yesteryear (one thinks of Downton Abbey, to take a popular example), what the servants knew about the masters was a great deal more than the masters knew about them. And in the Davis household the servants were black slaves, treated as shadows and often as something less than sentient beings. The Davises knew little of their lives, their hopes, their aspirations, and they certainly did not realize that two of them would spy for the Union.

History is almost equally oblivious. When it comes to secret agents, or servants, or slaves, all learned to tell the smooth lie that let them survive, and few kept records that endure. When it comes to the question of the spies who worked in the Confederate White House, where solid documentary evidence has failed, legend often has stepped in to fill the gaps and, to some considerable extent, to cloud the picture.

The one slave-spy we know the most about is William A. Jackson, the handsome coachman who appears to have been hired out by his owner at one point to work as a waiter in a Richmond hotel before being rented to the Davis family to drive them around the city.

In early May 1862, soon after New Orleans had fallen to the Union and as the Federal army under Gen. George McClellan was inching its way up the peninsula from Yorktown toward Richmond, the slave William Jackson crossed the lines into the Federal camp and began telling his story to the officers, who debriefed him at length, then to a handful of reporters. Over the next several weeks, tales about his revelations were printed and reprinted in papers all over the country.

Thus, one could read in the The Liberator, an abolitionist paper out of Boston, an article picked up from Horace Greeley’s Tribune in New York that was a paean to the escaped slaves making their way to Union encampments. Typically they were called “contrabands,” not yet entitled to their freedom (the Emancipation Proclamation was not announced until later that year, and did not go into effect until 1863).

“The fact cannot be questioned that the most important information we receive of the enemy’s movements reaches us through the contrabands,” the author of the Tribune article proclaimed.

When Jackson made his appearance in the Union camp, we are told, generals, colonels and majors flocked around him and the commander, Gen. Irvin McDowell, telegraphed the War Department with some of Jackson’s revelations.

If he brought useful tactical intelligence, however, it didn’t make it into the Northern newspapers, which focused on the gossip he passed along.

Jackson described Jefferson Davis as “pale and haggard,” sleeping little, eating nothing, constantly irritable and complaining about his generals: ‘He plans advances, but they execute masterly retreats,’” Jackson is quoted saying.

Varina Davis, meanwhile, had become a terror to her servants. “Mr. Davis treated me well,” said Jackson, “but Mrs. Davis is the d–––l,” the word devil considered too fraught for the paper’s readers.

Jackson seems to have spent quite a bit of time driving Varina around, and listening closely to her depressed views of the “played out” Confederacy. In part, no doubt, Jackson was telling the Union officers and press what they wanted to hear, raising their morale by talking about the declining mood in Rebel Richmond. He said not only slaves but whites were looking forward to the arrival of the Union troops. The Davises kept their bags packed and ready to go, he said, and even Mrs. Davis couldn’t pass off Confederate money.

Harper’s Weekly magazine published an engraved portrait of Jackson with his flowing signature beneath it, and a brief article filled with the kind of amazed admiration and inbuilt condescension that was common throughout the Northern press when it lionized escaped slaves: Jackson was “an extremely intelligent man, reads and writes (as his signature shows), and converses in a manner which shows that he has been used to good society.”

The Tribune’s backhanded praise, picked up by The Liberator, had been even worse. “The old plea, that a mulatto may have a soul and be intelligent on account of the white blood in his veins, while a pure negro is nothing but an overgrown monkey minus the caudal appendage, will not hold true in this instance. Jackson is as black as a Congo negro, and much more intelligent than a good many white folks.”

By the end of the summer of ’62, as McClellan’s peninsular campaign faltered and the war seemed to be stretching on endlessly, Jackson, the contrabands, and what came to be called their “black dispatches” started to catch the blame for military and political failings. The Brooklyn Daily Eagle, which billed itself as the evening paper with the highest circulation in the United States, started writing about “that arrant humbug, ‘Jeff Davis’ coachman,” who supposedly had assured the North that Union sympathizers would rise up in Richmond.

After that, Jackson faded from the scene, and it was not until after the war that stories began to circulate about another, and potentially much more effective spy in the Confederate White House.

Her name in popular history is Mary Elizabeth Bowser, but she used many different names, in fact. She was part of an extensive Union spy network run by Elizabeth Van Lew, a Richmond society woman who once owned Mary, had her educated in the North, and freed her in secret only to enlist her in the spy ring that included, it seems, Mary’s assignment as a slave-servant in the Confederate White House. Mary was of mixed blood, and she may well have been tied by that blood to Van Lew’s family, but if she worked for the Davises, as she and others claimed, she did so under a false name and false pretenses. Varina, asked about her years later, said she’d never heard of her.

The most careful and authoritative research on Bowser is to be found in Southern Lady, Yankee Spy, by Elizabeth R. Varon, a professor at the University of Virginia. As she notes, the spinsterly Van Lew never hid her Union sympathies entirely, but built a reputation for eccentricity as “Crazy Bet” that, along with her social status, afforded her considerable protection in Civil War Richmond.

As every intelligence service knows, when a woman is not taken seriously by the men around her she can work wonders as a secret agent, and that certainly was the case with Van Lew. She hid fugitive soldiers who escaped from Confederate prisons, she plotted with Union sympathizers and met clandestinely with couriers and spies from the Federal army.

By 1864, when Union Gen. Ulysses S. Grant was closing in on the Confederate capital in long, drawn out, bloody sieges, Van Lew’s network of white and black “detectives,” as spies liked to call themselves in those days, provided important, concrete military intelligence to Grant’s army. Of that there is no doubt. Grant recognized Van Lew’s valor publicly after the war, and in 1869, when he was president, had her appointed as Richmond’s postmaster (a somewhat ironic post for a former spymaster).

But what role did Mary play in Van Lew’s network?

It is safe to say (with ambiguity appropriate to espionage) a very special one.

In 1846, 15 years before the Civil War began, Van Lew had arranged the baptism of “Mary Jane, a colored child” in St. John’s church, which normally was reserved for white parishioners. It also appears from the chronologies provided by Van Lew’s biographers that Mary was not an infant at the time of the baptism. Probably she was 4 or 5 years old.

When that same little girl had barely reached her teens, Van Lew sent her to Princeton, New Jersey, “to receive an education, in order to prepare here to go to Liberia to serve as a missionary.”

Mary was only 14 when she sailed for Monrovia to teach the Gospel among slaves who had been liberated in the United States only to be sent “back to Africa” to a place among a people they never knew. Although desperately unhappy, Mary stayed for almost four years before Van Lew brought her back to Richmond. “I do love the poor creature,” Van Lew wrote to a friend in the American Colonization Society. “She was born a slave in our family—& that has made me always feel an awful responsibility.”

Interesting choice of words: “Born a slave in our family.” Later in life, Mary would talk about having “the advantage over the most of my race both in blood and intelligence,” and would tell Uncle Tom’s Cabin author Harriet Beecher Stowe, among others, that her mother was white but her father was “a Cuban-Spaniard and negro.” The rigorous and cautious Varon concludes it was much more likely that Mary’s father “was a white man, perhaps a member of the Van Lew family or their Lynchburg cousins, the Richardses.”

One might note that Mary was conceived and probably born before the death of Van Lew’s much-loved father in 1843.

The only physical description we have of Mary was after the war: “a Juno, done in somber marble … her features regular and expressive, her eyes exceedingly bright and sharp, her form and movements the perfection of grace.” (None of the photographs that purport to be her have been confirmed, according to Varon.)

Could such a woman, when she was about 20 years old, have found a place in the Davis household in Richmond? Not if she came directly from Van Lew, who was certainly no favorite of Varina Davis. Nor if she came under the name of Mary Richards, who had been arrested and jailed for not being a slave when she came back from Liberia in 1859, before the Van Lews claimed her again as part of the family’s chattel just to protect her. But under yet another alias? Very possibly.

In 1905, as a very old woman, Varina Davis felt called upon to deny that she had ever had in her employ “an educated negro ‘given or hired’ by Miss Van Lew as a spy,” and added, “My maid was an ignorant girl born and brought up on our plantation.”

But nobody had claimed Mary was Varina Davis’s maid. The executor of Van Lew’s estate had written in a biographical sketch and subsequent correspondence that Mary’s name was Mary Elizabeth Bowser (Mary Richards had married a Mr. Wilson Bowser in 1861). Then that name and general details of Mary’s life wound up in a Harper’s Monthly article in 1911:

“She was installed as waitress in the White House of the Confederacy. What she was able to learn, how long she remained behind Jefferson Davis’s dining chair, and what became of the girl ere the war ended are questions to which Time has effaced the answers.”

In fact, Varon and Lois Leveen, whose novel The Secrets of Mary Bowser tries to fill some of the gaps with fiction, did manage to find a bit more fact.

In September 1865, a woman calling herself Richmonia Richards gave a talk in New York City’s Abyssinian Baptist Church which the Anglo African, a ­newspaper there, described as “very sarcastic” and “quite humorous.” Among her anecdotes were stories of intrigue in the Confederate Senate as well as the Confederate White House. But the real substance of the intrigues is not there.

By 1867, Mary, with missionary zeal, was teaching freed slaves and their children in St. Mary’s, Georgia, on the Florida border, when Harriet Beecher Stowe, her brother the Rev. Charles Beecher, and the Rev. Crammond Kennedy of the Freedmen’s Bureau paid a visit. They were enthralled, especially by stories she told of her work as “a member of a secret organization in Richmond … a detective of Gen’l Grant.” The Rev. Kennedy thought “she could write a romance from her experience in that employment.”

Would that she had, or, if she did, that it could be found.

In fact, the last we know of Mary J.R. Richards Bowser Garvin (there was another husband after the war) aka Mary Jones aka Richmonia Richards and also known as names unknown is from her letters in 1867 to the superintendent of education for the Georgia Freedmen’s Bureau explaining why she could no longer carry on with so few resources trying to meet such enormous needs in ever more hostile and dangerous territory. The white Southerners, the old elites, meant to make it impossible to educate the former slaves and their children.

“I wish there was some law here, or some protection,” she wrote. “I know the southerners pretty well … having been in the service so long as a detective that I still find myself scrutinizing them closely. There is … that sinister expression about the eye, and the quiet but bitterly expressed feeling that I know portends evil … with a little whiskey in them, they dare do anything … Do not think I am frightened and laugh at my letter. Anyone that has spent 4 months in Richmond prison does not be so easily frightened."

What happened to Mary after that? The facts of her life fell prey to prejudice, the sinister turmoil in the Reconstruction South, and the traditions of spies who take their greatest secrets to grave. Almost 150 years later, truly, time has effaced the answers.