Despite the challenges, by the early 1870s, the agency had the world's largest collection of mug shots and a “criminal database.” During the height of its existence, the Pinkertons had more agents than the standing army of the United States of America, causing the state of Ohio to outlaw the agency, due to the possibility of its being hired out as a "private army" or militia.

Fortunes were to decrease once again for the agency when, in 1871, Chicago suffered the Great Fire which began on the evening of October 7th. Before it burned itself out three days later, the entire business district was destroyed, including the Pinkerton buildings and many of their records. When the fire was finally extinguished, martial law was declared in Chicago and guards from the Pinkerton Detective Agency were hired to prevent looting. Robert's widow, Alice Isabella Pinkerton, and his dependents were also left homeless. When she approached  Allan for assistance, he encouraged them to return to Great Britain. Offering to pay for the journey, Alice and her sons accepted his offer and sailed for Liverpool, leaving the agency entirely in the hands of Allan and his sons.

When Allan Pinkerton passed away in 1884, the agency was taken over by his sons, Robert and William. They soon became involved in the labor unrest of the late 19th century when they were hired by a number of businesses to keep strikers and suspected unionists out of their factories.

However, the rapidly expanding agency soon became known for less admirable activities as they often became the “law” in of themselves. Accused of using heavy handed tactics, such as firebombing Jesse James’ mother’s home and using intimidation against union sympathizers, the public support began to turn away from the agency.

Many labor sympathizers accused the Pinkertons of inciting riots and their reputation continued to suffer. The most notorious example of this was the Homestead Strike of 1892, when Pinkerton agents killed 11 people while enforcing strikebreaking measures.  In order to restore order, two brigades of state militia had to be called out.

Continuing their involvement against the labor movement into the 20th century, their reputation was harmed for years in the public consciousness.