How Slavery Really Ended in America

On May 23, 1861, little more than a month into the Civil War, three young black men rowed across the James River in Virginia and claimed asylum in a Union-held citadel. Fort Monroe, Va., a fishhook-shaped spit of land near the mouth of the Chesapeake Bay, had been a military post since the time of the first Jamestown settlers. This spot where the slaves took refuge was also, by remarkable coincidence, the spot where slavery first took root, one summer day in 1619, when a Dutch ship landed with some 20 African captives for the fledgling Virginia Colony. 

Two and half centuries later, in the first spring of the Civil War, Fort Monroe was a lonely Union redoubt in the heart of newly Confederate territory. Its defenders stood on constant guard. Frigates and armed steamers crowded the nearby waters known as Hampton Roads, one of the world’s great natural harbors. Perspiring squads of soldiers hauled giant columbiad cannons from the fort’s wharf up to its stone parapets. Yet history would come to Fort Monroe not amid the thunder of guns and the clash of fleets, but stealthily, under cover of darkness, in a stolen boat.

Frank Baker, Shepard Mallory and James Townsend were field hands who — like hundreds of other local slaves — had been pressed into service by the Confederates, compelled to build an artillery emplacement amid the dunes across the harbor. They labored beneath the banner of the 115th Virginia Militia, a blue flag bearing a motto in golden letters: “Give me liberty or give me death.”

After a week or so of this, they learned some deeply unsettling news. Their master, a rebel colonel named Charles Mallory, was planning to send them even farther from home, to help build fortifications in North Carolina. That was when the three slaves decided to leave the Confederacy and try their luck, just across the water, with the Union.

It cannot have been an easy decision for the men. What kind of treatment would they meet with at the fort? If the federal officers sent them back, would they be punished as runaways — perhaps even as traitors? But they took their chances. Rowing toward the wharf that night in May, they hailed a guard and were admitted to the fort.

The next morning they were summoned to see the commanding general. The fugitives could not have taken this as an encouraging sign. Having lived their whole lives near the fort, they probably knew many of its peacetime officers by sight, but the man who awaited them behind a cluttered desk was someone whose face they had never seen. Worse still, as far as faces went, his was not — to put it mildly — a pleasant one. It was the face of a man whom many people, in the years ahead, would call a brute, a beast, a cold-blooded murderer. It was a face that could easily make you believe such things: a low, balding forehead, slack jowls and a tight, mean little mouth beneath a drooping mustache. It would have seemed a face of almost animal-like stupidity had it not been for the eyes. These glittered shrewdly, almost hidden amid crinkled folds of flesh. One of them had an odd sideways cast, as if its owner were always considering something else besides the thing in front of him. These were the eyes that now surveyed Baker, Mallory and Townsend.

The general began asking them questions: Who was their master? Was he a rebel or a Union man? Were they field hands or house servants? Did they have families? Why had they run away? Could they tell him anything about the Confederate fortifications they had been working on? Their response to this last question — that the battery was still far from completion — seemed to please him. At last he dismissed the three brusquely, offering no indication of their fate.

Maj. Gen. Benjamin Franklin Butler arrived at the fort only a day ahead of the fugitive slaves, greeted at the esplanade by a 13-gun salute. That morning, Butler sat down to compose an important initial report. When an adjutant interrupted to inform him of the fugitives, Butler set down his pen. The War Department could wait. The three ragged black men waiting outside were a more pressing matter. 

Butler was no abolitionist, but the three slaves presented a problem. True, the laws of the United States were clear: all fugitives must be returned to their masters. The founding fathers enshrined this in the Constitution; Congress reinforced it in 1850 with the Fugitive Slave Act; and it was still the law of the land — including, as far as the federal government was concerned, within the so-called Confederate states. The war had done nothing to change it. Most important, noninterference with slavery was the very cornerstone of the Union’s war policy. President Abraham Lincoln had begun his inaugural address by making this clear, pointedly and repeatedly. “I have no purpose, directly or indirectly, to interfere with the institution of slavery in the states where it exists,” the president said. “I believe I have no lawful right to do so, and I have no inclination to do so.”  

Yet to Fort Monroe’s new commander, the fugitives who turned up at his own front gate seemed like a novel case. The enemy had been deploying them to construct a battery aimed directly at his fort — and no doubt would put them straight back to work if recaptured, with time off only for a sound beating. They had just offered him some highly useful military intelligence. And Virginia, as of 12 or so hours ago, was officially in rebellion against the federal government, having just ratified the secession ordinance passed a month before. Butler had not invited the fugitives in or engineered their escape, but here they were, literally at his doorstep: a conundrum with political and military implications, at the very least. He could not have known — not yet — that his response that day might change the course of the national drama that was then just beginning. Yet it was not the first time, nor would it be the last, that an unanticipated bureaucratic dilemma would force the hand of history.

Despite his rank, General Butler had been a professional soldier barely four weeks. In private life, back in Massachusetts, he was a lawyer, and a very successful one — although he grew up poor, the swamp Yankee son of a widow who kept a boardinghouse in Lowell, the textile-mill town. Unable to attract clients through social connections or charm, he became an expert quibbler: a man who knew every loose thread in the great tangled skein of common law and who could unravel an opponent’s entire case with the gentlest of tugs. By his early 40s, he had also built a successful career as a state legislator and harbored larger political ambitions.

A fellow officer once called Butler “less like a major general than like a politician who is coaxing for votes.” Race-baiting was red meat to many of his working-class constituents in Lowell, and he had always been glad to toss morsels in their direction. But after barely 24 hours at Fort Monroe, the new commander had already sized up his new constituency. The garrison was made up predominantly of eager volunteers from New England, many with antislavery sympathies. How was Butler to win the confidence — or even obedience — of such men if his first act as their commander was to send three poor blacks back into bondage?

Butler’s features may have been brutish and his manners coarse, but inwardly, he nursed the outsize vanity of certain physically ugly men — vanity often manifest in a craving for approval and adulation. He also possessed a sympathetic, even occasionally sentimental, heart.

Still . . . sentiment was a fine thing; so was the admiration of one’s subordinates. Ultimately, though, his duty was to his commander in chief. With a few strokes of his pen, Lincoln had made Butler a major general; the president could just as easily unmake him, sending him back to Lowell in disgrace — and with another stroke, for that matter, send the blacks back to their master as slaves.

Whatever Butler’s decision on the three fugitives’ fate, he would have to reach it quickly. He had barely picked up his pen to finally begin that report before an adjutant interrupted with another message: a rebel officer, under flag of truce, had approached the causeway of Fort Monroe. The Virginians wanted their slaves back.

Waiting before the front gate was a man on horseback: Maj. John Baytop Cary of the 115th. With his silver gray whiskers and haughtily tilted chin, he appeared every inch the Southern cavalier.

Butler, also on horseback, went out to meet him. The men rode, side by side, off federal property and into rebel Virginia. They must have seemed an odd pair: the dumpy Yankee, unaccustomed to the saddle, slouching along like a sack of potatoes; the trim, upright Virginian, in perfect control of himself and his mount. 

Cary got down to business. “I am informed,” he said, “that three Negroes belonging to Colonel Mallory have escaped within your lines. I am Colonel Mallory’s agent and have charge of his property. What do you mean to do with those Negroes?”  

“I intend to hold them,” Butler said.

“Do you mean, then, to set aside your constitutional obligation to return them?”

Even the dour Butler must have found it hard to suppress a smile. This was, of course, a question he had expected. And he had prepared what he thought was a fairly clever answer.

“I mean to take Virginia at her word,” he said. “I am under no constitutional obligations to a foreign country, which Virginia now claims to be.”

“But you say we cannot secede,” Cary retorted, “and so you cannot consistently detain the Negroes.”

“But you say you have seceded,” Butler said, “so you cannot consistently claim them. I shall hold these Negroes as contraband of war, since they are engaged in the construction of your battery and are claimed as your property.”

Ever the diligent litigator, Butler had been reading up on his military law. In time of war, he knew, a commander had a right to seize any enemy property that was being used for hostile purposes. The three fugitive slaves, before their escape, were helping build a Confederate gun emplacement. Very well, then — if the Southerners insisted on treating blacks as property, this Yankee lawyer would treat them as property, too. Legally speaking, he had as much justification to confiscate Baker, Mallory and Townsend as to intercept a shipment of muskets or swords.

Cary, frustrated, rode back to the Confederate lines. Butler, for his part, returned to Fort Monroe feeling rather pleased with himself. Still, he knew that vanquishing the rebel officer was only a minor victory, and perhaps a momentary one if his superiors in Washington frowned on what he had done.

The following day, a Saturday, Butler picked up his pen and resumed his twice-interrupted dispatch to Washington. Certain questions had arisen, he began, “of very considerable importance both in a military and political aspect, and which I beg leave to herewith submit.”

But before this missive reached its destination, matters would become even more complicated. On Sunday morning, eight more fugitives turned up at Union lines outside the fort. On Monday, there were 47 — and not just young men now, but women, old people, entire families. There was a mother with a 3-month-old infant in her arms. There was an aged slave who had been born in the year of America’s independence.

By Wednesday, a Massachusetts soldier would write home: “Slaves are brought in here hourly.”

“What’s to Be Done With the Blacks?” asked a headline in The Chicago Tribune. That was the question now facing the Lincoln administration. Within days after the three fugitive slaves crossed the river, their exploits — and their fate — were being discussed throughout the nation. At first the newspapers played it more or less as a comic sketch in a minstrel show: a Yankee shyster outwits a drawling Southern aristocrat. But Lincoln saw things in a more serious light. The president realized he might now be forced to make a signal verdict about matters he previously tried to avoid: slavery, race and emancipation.

Lincoln and his cabinet gathered to address Butler’s decision — and ended up punting. While reminding Butler that “the business you are sent upon . . . is war, not emancipation,” they left the general to decide what to do with fugitive slaves — including whether or not to continue declaring them contraband of war. Unfortunately, no detailed account of the deliberations survives. But a letter from one cabinet secretary, Montgomery Blair, suggests they were driven by a motive as common in Washington then as it is now: “a desire to escape responsibility for acting at all at this time.” By that point, the administration had already received a second dispatch from Butler, describing the influx of women and children. With this in mind, Blair — a member of a slaveholding Maryland family — suggested one pragmatic “modification” to Butler’s policy. “You can . . . take your pick of the lot and let the rest go so as not to be required to feed unproductive laborers or indeed any that you do not require,” he urged. As to the slaves’ eventual fate, Blair wrote, of course no one was suggesting that they be set free. Perhaps at the end of the war, those who belonged to men convicted of treason could be legally confiscated and sent off to Haiti or Central America. (The New York Herald, meanwhile, proposed that the federal government should wait until the war ended and sell all the slaves back to their owners, at half-price, to finance its cost.) 

“I intend to hold them,” Butler said.

“Do you mean, then, to set aside your constitutional obligation to return them?”

Even the dour Butler must have found it hard to suppress a smile. This was, of course, a question he had expected. And he had prepared what he thought was a fairly clever answer.

“I mean to take Virginia at her word,” he said. “I am under no constitutional obligations to a foreign country, which Virginia now claims to be.”

“But you say we cannot secede,” Cary retorted, “and so you cannot consistently detain the Negroes.”

“But you say you have seceded,” Butler said, “so you cannot consistently claim them. I shall hold these Negroes as contraband of war, since they are engaged in the construction of your battery and are claimed as your property.”

Ever the diligent litigator, Butler had been reading up on his military law. In time of war, he knew, a commander had a right to seize any enemy property that was being used for hostile purposes. The three fugitive slaves, before their escape, were helping build a Confederate gun emplacement. Very well, then — if the Southerners insisted on treating blacks as property, this Yankee lawyer would treat them as property, too. Legally speaking, he had as much justification to confiscate Baker, Mallory and Townsend as to intercept a shipment of muskets or swords.

Cary, frustrated, rode back to the Confederate lines. Butler, for his part, returned to Fort Monroe feeling rather pleased with himself. Still, he knew that vanquishing the rebel officer was only a minor victory, and perhaps a momentary one if his superiors in Washington frowned on what he had done.

The following day, a Saturday, Butler picked up his pen and resumed his twice-interrupted dispatch to Washington. Certain questions had arisen, he began, “of very considerable importance both in a military and political aspect, and which I beg leave to herewith submit.”

But before this missive reached its destination, matters would become even more complicated. On Sunday morning, eight more fugitives turned up at Union lines outside the fort. On Monday, there were 47 — and not just young men now, but women, old people, entire families. There was a mother with a 3-month-old infant in her arms. There was an aged slave who had been born in the year of America’s independence.

By Wednesday, a Massachusetts soldier would write home: “Slaves are brought in here hourly.”

“What’s to Be Done With the Blacks?” asked a headline in The Chicago Tribune. That was the question now facing the Lincoln administration. Within days after the three fugitive slaves crossed the river, their exploits — and their fate — were being discussed throughout the nation. At first the newspapers played it more or less as a comic sketch in a minstrel show: a Yankee shyster outwits a drawling Southern aristocrat. But Lincoln saw things in a more serious light. The president realized he might now be forced to make a signal verdict about matters he previously tried to avoid: slavery, race and emancipation.

Lincoln and his cabinet gathered to address Butler’s decision — and ended up punting. While reminding Butler that “the business you are sent upon . . . is war, not emancipation,” they left the general to decide what to do with fugitive slaves — including whether or not to continue declaring them contraband of war. Unfortunately, no detailed account of the deliberations survives. But a letter from one cabinet secretary, Montgomery Blair, suggests they were driven by a motive as common in Washington then as it is now: “a desire to escape responsibility for acting at all at this time.” By that point, the administration had already received a second dispatch from Butler, describing the influx of women and children. With this in mind, Blair — a member of a slaveholding Maryland family — suggested one pragmatic “modification” to Butler’s policy. “You can . . . take your pick of the lot and let the rest go so as not to be required to feed unproductive laborers or indeed any that you do not require,” he urged. As to the slaves’ eventual fate, Blair wrote, of course no one was suggesting that they be set free. Perhaps at the end of the war, those who belonged to men convicted of treason could be legally confiscated and sent off to Haiti or Central America. (The New York Herald, meanwhile, proposed that the federal government should wait until the war ended and sell all the slaves back to their owners, at half-price, to finance its cost.) 

“Stampede Among the Negroes in Virginia,” proclaimed Frank Leslie’s Illustrated Newspaper, with a double-page spread of dramatic woodcuts showing black men, women and children crossing a creek under a full moon, then being welcomed heartily into the fort by General Butler himself (or rather, by the artist’s trimmer, handsomer version of him). One correspondent estimated that “this species of property under Gen. Butler’s protection [is] worth $500,000, at a fair average of $1,000 apiece in the Southern human flesh market.”

Journalists throughout the Union quipped relentlessly about the “shipments of contraband goods” or, in the words of The Times, “contraband property having legs to run away with, and intelligence to guide its flight” — until, within a week or two after Butler’s initial decision, the fugitives had a new name: contrabands. It was a perfectly composed bit of slang, a minor triumph of Yankee ingenuity.

Were these blacks people or property? Free or slave? Such questions were, as yet, unanswerable — for answering them would have raised a host of other questions that few white Americans were ready to address. Contrabands let the speaker or writer off the hook by letting the escapees be all those things at once. “Never was a word so speedily adopted by so many people in so short a time,” one Union officer wrote. Within a few weeks, the average Northern newspaper reader could scan, without blinking, a sentence like this one: “Several contrabands came into the camp of the First Connecticut Regiment today.”

As routine as the usage soon became, however, a hint of Butler’s joke remained, a slight edge of nervous laughter. A touch of racist derision, too: William Lloyd Garrison’s abolitionist newspaper, The Liberator, carped, justly enough, that it was offensive to speak of human beings that way. Yet in its very absurdity, reflecting the Alice-in-Wonderland legal reasoning behind Butler’s decision, the term also mocked the absurdity of slavery — and the willful stupidity of federal laws that, for nearly a century, had acknowledged no meaningful difference between a bushel of corn and a human being with dark skin. Eventually, even black leaders adopted it.

Back at Monroe — dubbed “the freedom fort” — fugitives continued arriving daily. Each morning, dozens lined up to pitch in with manual labor. Soon they seemed almost like members of the garrison. A Times correspondent wrote: “Their shovels and their other implements of labor, they handle and carry as soldiers do their guns. . . . I have no doubt they would make fair or even excellent soldiers.” Moreover, as the garrison’s medical chief remarked, “they are the pleasantest faces to be seen at the post.”

Many of the Union soldiers had never really spoken with a black person before; the Vermont farmboys had perhaps never even seen one before leaving home. Now they were conversing with actual men and women who had been (and perhaps still were) slaves: people who had previously figured only as a political abstraction. Some fugitives shared horrific accounts; one man described “bucking,” a practice in which a slave, before being beaten, had his wrists and ankles tied and slipped over a wooden stake. Almost all spoke of loved ones sold away; the most chilling thing was that they said it matter-of-factly, as if their wives or children had simply died.

Perhaps most surprising of all — for Northerners accustomed to Southern tales of contentedly dependent slaves — was this, in the words of one soldier: “There is a universal desire among the slaves to be free. . . . Even old men and women, with crooked backs, who could hardly walk or see, shared the same feeling.”

General Butler grew ever more adamant in the defense of “his” contrabands, to a degree that must have shocked his old associates. By July, he began pressing the Lincoln administration to admit that the contrabands were not really contraband: that they had become free. Indeed, that they were — in a legal sense — no longer things but people: “Have they not by their master’s acts, and the state of war, assumed the condition, which we hold to be the normal one, of those made in God’s image? . . . I confess that my own mind is compelled by this reasoning to look upon them as men and women.”

It would take another 14 months — and tens of thousands more Union casualties — before the Lincoln administration was ready to endorse such a view. 

“Shall we now end the war and not eradicate the cause?” the general wrote to a friend in August. “Will not God demand this of us now [that] he has taken away all excuse for not pursuing the right?” (During the rest of the war, Butler’s support for black civil rights — and harsh treatment of rebel sympathizers — made him hated throughout most of the South, where he won the nickname Beast Butler.)  

More and more people had begun to share Butler’s conviction that the fugitives at Monroe stood in the vanguard of a larger revolution. “I have watched them with deep interest, as they filed off to their work, or labored steadily through the long, hot day,” a Northern visitor to the fort wrote. “Somehow there was to my eye a weird, solemn aspect to them, as they walked slowly along, as if they, the victims, had become the judges in this awful contest, or as if they were . . . spinning, unknown to all, the destinies of the great Republic.”

Earthshaking events are sometimes set in motion by small decisions. Perhaps the most famous example was when Rosa Parks boarded a segregated bus in Montgomery, Ala. More recently, a Tunisian fruit vendor’s refusal to pay a bribe set off a revolution that continues to sweep across the Arab world. But in some ways, the moment most like the flight of fugitive slaves to Fort Monroe came two decades ago, when a minor East German bureaucratic foul-up loosed a tide of liberation across half of Europe. On the evening of Nov. 9, 1989, a tumultuous throng of people pressed against the Berlin Wall at Checkpoint Charlie, in response to an erroneous announcement that the ban on travel to the West would be lifted immediately. The captain in charge of the befuddled East German border guards dialed and redialed headquarters to find some higher-up who could give him definitive orders. None could. He put the phone down and stood still for a moment, pondering. “Perhaps he came to his own decision,” Michael Meyer of Newsweek would write. “Whatever the case, at 11:17 p.m. precisely, he shrugged his shoulders, as if to say, ‘Why not?’ . . . ‘Alles auf!’ he ordered. ‘Open ’em up,’ and the gates swung wide.”

The Iron Curtain did not unravel at that moment, but that night the possibility of cautious, incremental change ceased to exist, if it had ever really existed at all. The wall fell because of those thousands of pressing bodies, and because of that border guard’s shrug.

In the very first months of the Civil War — after Baker, Mallory and Townsend breached their own wall, and Butler shrugged — slavery’s iron curtain began falling all across the South. Lincoln’s secretaries John Hay and John Nicolay, in their biography of the president, would say of the three slaves’ escape, “Out of this incident seems to have grown one of the most sudden and important revolutions in popular thought which took place during the whole war.”

Within weeks after the first contrabands’ arrival at Fort Monroe, slaves were reported flocking to the Union lines just about anywhere there were Union lines: in Northern Virginia, on the Mississippi, in Florida. It is unclear how many of these escapees knew of Butler’s decision, but probably quite a few did. Edward Pierce, a Union soldier who worked closely with the contrabands, marveled at “the mysterious spiritual telegraph which runs through the slave population,” though he most likely exaggerated just a bit when he continued, “Proclaim an edict of emancipation in the hearing of a single slave on the Potomac, and in a few days it will be known by his brethren on the gulf.”

In August, Lincoln’s War Department tried to bring some clarity to the chaos by asking Union commanders to collect detailed information on each fugitive: not just name and physical description but “the name and the character, as loyal or disloyal, of the master” — since whether the master supported the Union or the Confederacy was, of course, essential to determining whether the particular man or woman counted as legitimate contraband. Such a system would let the federal government assure slaveholders that their “rights” were protected, and possibly return the slaves to their proper owners once the rebel states had rejoined the Union.

But how were officers supposed to tell whether a master they had never laid eyes on was loyal or disloyal — even assuming that the slave was telling the truth in identifying him? Besides, didn’t the military have more pressing business at the moment, like fighting the war? The new contraband doctrine was utterly unenforceable almost from the moment it was devised, but it became hugely influential precisely because it was so unenforceable: it did not open the floodgates in theory, but it did so in practice.

And it did so with very little political risk to the Lincoln administration. Indeed, preposterous as the contraband doctrine was as a piece of law, it was also — albeit inadvertently — a masterstroke of politics; indeed, it satisfied nearly every potential theoretical and political objection while being completely unworkable in the long run. “There is often great virtue in such technical phrases in shaping public opinion,” Pierce observed. “The venerable gentleman, who wears gold spectacles and reads a conservative daily, prefers confiscation to emancipation. He is reluctant to have slaves declared freemen but has no objection to their being declared contrabands.” 

The system was eminently practical in other terms. Regiments needed labor: extra hands to cook meals, wash clothes and dig latrines. When black men and women were willing to do these things, whites were happy not to ask inconvenient questions — not the first or the last time that the allure of cheap labor would trump political principles in America.

Blacks were contributing to the Union cause in larger ways. Not just at Fort Monroe but also throughout the South they provided Northerners with valuable intelligence and expert guidance. When Lincoln’s master spy, Allan Pinkerton, traveled undercover through the Confederacy, he wrote, “My best source of information was the colored men. . . . I mingled freely with them, and found them ever ready to answer questions and to furnish me with every fact which I desired to possess.” They were often the only friends the Yankees encountered as they groped their way anxiously through hostile territory.

Just as influential was what did not happen: the terrible moment — long feared among whites — when slaves would rise up and slaughter their masters. It soon became apparent from the behavior of the contrabands that the vast majority of slaves did not want vengeance: they simply wanted to be free and to enjoy the same rights and opportunities as other Americans. Many were even ready to share in the hardships and dangers of the war. Millions of white Americans realized they did not actually have to fear a bloodbath if the slaves were suddenly set free. This awareness in itself was a revolution.

Most important, though, was the revolution in the minds of the slaves themselves. Within little more than a year, the stream of a few hundred contrabands at Fort Monroe became a river of tens — probably even hundreds — of thousands. They “flocked in vast numbers — an army in themselves — to the camps of the Yankees,” a Union chaplain wrote. “The arrival among us of these hordes was like the oncoming of cities.”

When Lincoln finally unveiled the Emancipation Proclamation in the fall of 1862, he framed it in Butleresque terms, not as a humanitarian gesture but as a stratagem of war.On the September day of Lincoln’s edict, a Union colonel ran into William Seward, the president’s canny secretary of state, on the street in Washington and took the opportunity to congratulate him on the administration’s epochal act.

Seward snorted. “Yes,” he said, “we have let off a puff of wind over an accomplished fact.”

“What do you mean, Mr. Seward?” the officer asked.

“I mean,” the secretary replied, “that the Emancipation Proclamation was uttered in the first gun fired at Sumter, and we have been the last to hear it.”