The Whaley House was the home of Thomas Whaley and his family. At various times it also housed Whaley's general store, San Diego's second county courthouse, and the first commercial theater in San Diego.The house has "witnessed more history than any other building in the city".

Shortly after the Whaley family moved in, they told the San Diego Union that they heard heavy footsteps in the house which they believed to be the ghost of James "Yankee Jim" Robinson, who had earlier been hanged on the property for stealing a boat.