Victorian police would be given extra powers to , under a proposal from the state opposition.
The Liberals also want people who publicly display Nazi symbols such as the swastika to face up to six months in jail, in line with existing serious vilification offences.
The opposition says the changes should be make through an amendment to Victoria's Racial and Religious Tolerance Act, to tackle a growing trend of the symbols being used to inspire hatred.
The proposal comes after outcry over the flying of a Nazi swastika above a home in Victoria’s north-west.
The red and black flag, which included a swastika surrounded by Iron Crosses and other Nazi-era symbols, was spotted flying over a private property in Beulah last month.
Its presence triggered several complaints to police, however homeowner Cheryl Lawdorn said she had flown the flag due to her German ancestry.
"This swastika is the most hateful symbol that we would ever imagine. It is a symbol of murder, of mass murder," Liberal MP David Southwick told reporters at Melbourne's Jewish Holocaust Centre on Sunday, after introducing his party's proposal.
"This has to stop. Today is a day to draw the line in the sand."

Liberal Member for Caulfield David Southwick made the proposal to ban Nazi symbols. Source: AAP
Under the Liberals' proposed change, Victoria Police could remove and confiscate items they reasonably believe to breach the ban, if there is an intention to incite hate or physical violence.
The change would also include exemptions for ensuring legitimate use - such as for education, research and artistic purposes - remains lawful.
If the Andrews government doesn't make such an amendment itself, the Liberals will introduce their own private members bill.
Last month, Victorian Premier Daniel Andrews called the Beulah
"If there is any decency in that household, they will take down that flag immediately," Mr Andrews said.
"It is completely unacceptable for anybody to be flying what is perhaps the ultimate symbol of hate.

Victorian Premier Daniel Andrews took aim at the homeowner flying the Nazi flag. Source: AAP
"Nothing justifies that whatsoever.”
Despite locals’ complaints to the local council and police, Yarriambiack Shire Council told SBS News, it did not have any powers to remove the flag.
Its has outraged several neighbours, who have complained to police, and has been condemned by both the state government and the opposition.
The Beulah community has stressed the local couple who flew the Nazi flag didn't represent them, after reports of it came to light.
Dozens of them gathered s as they could find to stress their support for inclusivity.
A parliamentary committee is currently reviewing Victoria's anti-vilification laws.

Dozens of people gathered in Beulah, Saturday 18 January 2020, in response to a local couple who raised a Nazi flag above their home Source: AAP
Victorian Equality Minister Martin Foley says the government is looking forward to seeing its recommendations and acting on them.
"Any expression of Nazi propaganda, whether it be swastikas and indeed any offensive, antisemitic, racist paraphernalia, has no place in Victoria," he told reporters on Sunday.
"Let's be clear, antisemitism is on the rise in this country and around the world, and we all need as a community to make sure we take firm measures to put an end to it."