For sixteen years, the castle remained unfinished. Though Snyder’s sons Robert, Jr., Leroy, and Kenneth continued the work, the pace was slow and not nearly as elaborate as the original plans. In 1922, the interior and upper floors were finally completed. During this time, Robert Snyder, Jr., lived at the castle. Plagued by health problems, he cultivated an interest in local and regional history, collecting every book on the subject that he could get his hands on. Amassing a collection of over a thousand volumes, the collection of Missouri and regional history, literature and folklore was kept at the castle until his death in 1937. Today the massive collection is housed at the University of Missouri-Kansas City.

However, by the time the castle was finally complete, the family businesses began to decline and troubles with the property were just beginning. The family faced years of adversity trying to keep the castle in the family when a long legal battle against Union Electric ensued over the waters of the Ozarks that were encroaching upon the natural spring-fed lake at the foot of Ha Ha Tonka Cliff. The family was forced to sell Snyder’s natural gas supply business to eastern interests to finance the long legal battle. By 1937, at the time of Robert Snyder, Jr’s death, the depression and the court litigations had depleted the family fortune and the surviving brothers were forced to take other financial steps with the mansion. Leasing the building to a woman named Mrs. Ellis; she operated the historic building as a hotel for the next several years.

However, in 1942, tragedy struck the castle again when sparks from one of its many fireplaces ignited the roof. Within hours, the mansion was gutted as well as the nearby carriage house. All that was left were the stark, devastated outside walls of the mansion and the nearby water tower. Again, the mansion was abandoned, sitting silently upon the cliff overlooking Ha Ha Tonka Spring and the Lake of the Ozarks for the next 36 years.