Port Townsend Customs House.
In 1861, President Abraham Lincoln appointed Victor Smith, a newspaper editor from Cincinnati, Ohio, as Collector of Customs for the District of Puget Sound, Washington. However, when Smith arrived, he immediately disliked Port Townsend, seeing it as little more than a city of “huts” populated by uncouth Westerners.
An ambitious man, Smith began promoting the move of the Customs House from Port Townsend to Port Angeles. With the nation in the midst of the Civil War, he further supported his plan by arguing that the British intended to join forces with the Confederates and that the existing Port of Entry could be easily invaded from the British naval base on Vancouver Island in British Columbia.
Anticipating the move, Smith and four other men acquired land in Port Angeles and formed a Townsite Company to develop the new community. The townsfolk of Port Townsend soon learned of Smith’s plan, which, of course, created hostility in the burgeoning community.
When Smith’s letters to the Treasury Department failed to get immediate results, Smith traveled to Washington, D.C., to promote his plan to fortify Port Angeles and relocate the Customs House there. Promoting Port Angeles’ superior location because it was nearer the ocean, he convinced Congress to move the Customs District of Puget Sound from Port Townsend to Port Angeles. It was made official in June 1862.
Port Townsend, Washington by E.S. Glover, 1878.
However, not everything went as precisely as Smith had planned. While he celebrated his success on his return voyage to the west coast, Lieutenant Merryman, the Acting Collector of Customs in Port Townsend, audited the Customs House books. Finding the accounts short some $15,000, Merryman immediately reported the discrepancy to the Treasury Department by letter.
When Smith returned to Port Townsend on August 1, 1862, Lieutenant Merryman refused him access to the Customs House, accusing him of embezzlement and insisting that he would await official clearance from the Treasury Department before allowing Smith to return to his duties.
But Smith was not to be deterred. Returning to the ship on which he had arrived, he ordered the captain to train his three 12-pound cannons on the Customs House and the city’s commercial district. With the guns trained on the city, Merryman finally gave up the records, and the ship sailed for Port Angeles.
Incensed, Port Townsend citizens soon traveled to Olympia to protest Smith’s actions to the governor. Though a federal grand jury indicted Smith on 13 counts of embezzlement and misuse of public funds, the Treasury Department quashed the indictment after its investigation.
Port Angeles, Washington.
Meanwhile, Smith built a large building in Port Angeles that served as his family home and the Customs House. However, Port Townsend citizens were not done fighting and began building a case denouncing Smith and the transfer of the Port of Entry to Port Angeles.
After Smith completed the building in Port Angeles that served as both his home and the Customs House, he rented a portion of it to the government. However, the building was short-lived: in December 1863, a landslide in the Olympic Mountains caused the dam to burst, sending a torrent of water through the town. It leveled most of the city, swept the Customs House into the Strait of Juan de Fuca, and killed two customs employees. Smith was out of town, and his family was spared.
This disaster led to the first treasure tale. Inside the Customs House was a strongbox that allegedly held $1,500 in banknotes and numerous $20 gold coins valued at $7,500 at the time. While some say Smith stole the money during his tenure as a Customs Agent, others contend it was money the government rightfully paid him for his home in Ohio. In any event, the flood swept the strongbox away, and no trace of it has ever been found.
Finally, in January 1864, President Lincoln removed Smith as a Customs Collector. However, his “friends” at the Treasury Department gave him a new role as a special agent.
In this new job, Smith was often tasked with transporting large sums of money. In 1865, he and his family were on the East Coast, and while returning to San Francisco on the Gold Rule, Smith was tasked with transporting a large amount of payroll funds. However, on the ship’s passage from New York to San Juan, it wrecked on the Roncador Reef near the Isthmus of Panama. Though no lives were lost, the ship floundered for 11 days as the 700 passengers survived on supplies salvaged from the vessel. When two gunboats rescued them, they took the passengers to Panama. In the mass confusion following the wreck, the strongbox containing the money mysteriously disappeared. Smith remained on the reef, endeavoring to salvage the chest containing the money. The tale continues that with the aid of a diver, the chest was recovered; however, the gold was missing. Though Smith accused the captain of the theft, many thought Smith himself had absconded with the money.
San Francisco, California by Frank Marry, about 1850.
Meanwhile, Smith’s family crossed the Panama Canal by railroad and returned to San Francisco on the sailing ship America. Two weeks later, Smith followed, reaching San Francisco and transferring to the Brother Jonathan on July 28, 1865, for the trip to Port Angeles. Did the payroll funds from the East Coast end up at the bottom of the ocean near the Roncador Reef, or did they go with Smith when he finally made his way to San Francisco?
After transferring to Brother Jonathan, Smith was again accompanying a large amount of money—some $200,000—to pay the troops at Fort Vancouver, Walla Walla, and other posts in the Northwest. The ship was also thought to carry crates of $20 gold pieces for transfer to Haskins and Company and Wells Fargo. A government Indian Agent for the Northwestern Region may have also brought gold coins on board for annual treaty payments to the tribes.
As the Brother Jonathan passed through the Golden Gate and headed North, it faced a strong headwind and heavy seas that continually worsened during its journey. The next day, the ship pulled into the harbor at Crescent City to offload cargo. On its way again, the storm built, and the ship headed westward to avoid the rocks of St. George’s Reef. As the storm raged on, Captain DeWolf decided to return to Crescent City, California, to wait it out.
However, on July 30, 1865, the ship hit a rock and began to break up. Many of the passengers loaded into lifeboats were immediately thrown out as the smaller boats capsized. In the end, of the 244 people on board, only 19 people survived in a damaged boat that hobbled into the Crescent City harbor. For weeks, bodies washed ashore, but the gold and money on board were never found. Victor Smith was one of the casualties.
In 1866, the Customs Port of entry was returned to Port Townsend, to the great rejoicing of the citizens.
©Kathy Alexander/Legends of America, updated September 2026.
Also See:
John Tornow – The Wild Man of the Wynoochee
See Sources.