Sussan Ley stares down leadership speculation after a Liberal colleague says she's losing her party's support.