Before Devonshire House was the headquarters of the Religious Society of Friends in Britain from 1794 until 1925, it was a large house called Fisher's Folly.