Victoria Police said emergency services responded to reports of four people in difficulty in the water about 3.30 pm on 25 January.
"Three women and a man were pulled from the water unresponsive. CPR was administered on all four, but three were pronounced deceased at the scene. The woman was flown to hospital in a critical condition, where she later died,” the police told SBS Hindi.
The deceased included two women and a man in their 20s from Clyde, a southeastern suburb in Melbourne.
Another woman in her 40s was identified as an overseas visitor.
The Indian High Commission expressed condolences for the loss of lives.
Heart breaking tragedy in Australia: 4 Indians lost their lives in a drowning incident at Phillip Island, Victoria. Deepest condolences to families of the victims. @cgimelbourne team is in touch with friends of the deceased for all necessary assistance.Indian High Commission on X (formerly Twitter)
The deaths are not deemed suspicious, and the police will compile a report for the coroner.
The Victorian government's spokesperson also called the incident as an absolute tragedy.
"What unfolded yesterday afternoon at Forest Caves beach on Phillip Island is an absolute tragedy, and our hearts are with the devastated loved ones," the spokesperson said.
"This is a heartbreaking reminder to please be safe around water this summer, particularly as we come into the long weekend," the spokesperson added.
Mounil Kadakia, who established a foundation for the Indian community to promote beach safety programs following the tragic loss of his brother in a drowning incident in 2009, encouraged the community to stay alert at beaches.

Life Saving Victoria urges people to swim between the red and yellow flags whenever possible. Credit: Laurie Noble/Getty Images
A spokesperson from Life Saving Victoria said there are simple rules to ensure safety around beaches
"Swim between the red and yellow flags whenever possible, where lifesavers and lifeguards can help if you get into trouble," the spokesperson said.

Life Saving Victoria says all beaches can be dangerous so beware of beach conditions. Source: Moment RF / @ Didier Marti/Getty Images
"Learn how to spot and avoid rips and have knowledge and skills to get out of one and if you’re not sure, ask a lifesaver about the beach conditions," the spokesperson further added.
The Life Saving Victoria Drowning Report for the period from July 2022 to June 2023 highlighted that members of the Culturally and Linguistically Diverse (CALD) community face a heightened risk of drowning compared to individuals born in Australia.
Within the same period, the state registered 59 drowning fatalities, and notably, 12 of these occurred within the CALD community.