In 1934, Roy Stark passed away and his mother, Anna, died a short time later. The only residents left were Annabelle and Tony who lived in the dead town without indoor plumbing or electricity. Rarely bathing or changing clothes, they neglected the old hotel, letting the place pile with trash and discarded items, but continued to run the Home Comfort Store. The store, said to have been "sour-smelling", contained faded tins of outdated food and stale tobacco.

In 1947 when the book Stampede to Timberline was published by Muriel Sibell Wolle, which stated that St. Elmo was a ghost town, Tony and Annie were incensed claiming that it was not the tattered store or their eccentricities that drove away business, but rather Mrs. Wolle's statements in the book. Though Annabell was always said to have been kind and generous to the few who still frequented the store, the locals began to call her "Dirty Annie" because of her filthy clothing and tangled hair. She was also known to have roamed the old town, with rifle in hand, to protect her property. The town officially died on Sept. 30, 1952, when the post office closed.

Eventually, Tony and Annabelle were sent away to a mental institution, for their own safety and that of others. However, after just a few weeks, a sympathetic friend convinced the authorities that they were of no harm to anyone and they were released. Tony died a short time later and Annabelle was sent to a nursing home in 1958 where she died in 1960. Their property was left to the sympathetic friend who had helped them.

Shortly after Annabelle's death, the friend's grandchildren were said to have been playing in a room of the hotel, when suddenly all the doors in the room slammed shut and the temperature dropped nearly 20 degrees. The terrified children refused to play in the hotel again.

Another one of the grandchildren, a young woman in her twenties, decided to take on the hotel as a project, cleaning out the rooms, making minor repairs, and washing down the walls and floors. After cleaning up for the day, she and her friends would put away their tools and cleaning supplies, only to find them in the middle of the floor when they returned the next day. After this continued to occur, they started placing the items in a padlocked closet, but still they would be in the middle of the floor when they came back.