The fallout from anti-immigration rallies around the country is continuing, with the Victorian Premier being forced to abandon a press conference after being abused by a neo-Nazi.