What few people still live there recall a rural lane a stone's throw from the picturesque harbour.
As another development of what will almost certainly become more second homes or holiday lets gets the green light in a coastal community, they fear that soon no villagers will remain on it.
West Norfolk Council has agreed outline permission for a bungalow on Town Lane, at Brancaster Staithe, to be demolished and replaced by seven new properties.
It came despite objections from residents swamped by holiday homes. One posted on the council's planning portal: "There will not be any residents left in 10 years at this rate."
Retired engineer Stephen Harvey, 73, who lives on the lane, said: "Less than 10pc of the houses are lived-in in Brancaster Staithe.
"When we moved here 20 years ago, there were 17 houses and two of them weren't lived-in full-time. Now there are four which are lived-in full-time."
Seven new homes have been built on Town Lane in recent years.
Unlike those in neighbouring communities, Brancaster Parish Council's neighbourhood plan does not stipulate that new developments have to be principal residences, which people live in as their main home.
Instead, it states houses should be of "a modest size" to cater for people who wish to live in the village as well as holiday lets or second homes.
READ MORE: Village's landmark vote to curb second homes
READ MORE: Norfolk has some of the highest percentages of second homes
READ MORE: Will curbs help stem second homes boom?
"The country as a whole needs to do something to limit the number of holiday lets," said Mr Harvey.
"You can't do it community by community, it doesn't work. The people who have holiday homes aren't doing anything wrong, it's the system."
Retired legal secretary Jean Loose, 85, has lived on Town Lane for 40 years.
"It was a rural lane when we came here and we knew everybody," she said. "There are only four houses that are residences now. It's really annoying, really disappointing.
"The houses opposite us have been there for two years. Before that it was a single bungalow.
"They've demolished the bungalow, there was a beautiful monkey puzzle tree, there was a lovely beech hedge."
West Norfolk Council agreed planning permission for the bungalow, called Beersheba, to be demolished and replaced with five new houses in 2019.
Developers promised a contemporary development built from traditional materials.
Russell Levy said on the council's planning portal: "The five houses that have been erected on the Beersheba site, where previously there was one house, are all being used as holiday homes.
"It is highly likely that the seven homes the Cherry Trees developer wishes to build will also be used as holiday homes."
One of the properties included in the new development must be an affordable home.
Mr Levy's wife Sarah posted: "I am pleased to note that some affordable housing will be included in the development.
"Market prices of the new build homes will inevitably be high, however, given their proximity to the wonderful local AONB [area of outstanding natural beauty], which attracts many second-homers and holiday renters there is a very strong likelihood that these homes will be purchased as holiday homes for owner enjoyment and/or rental for profit, which brings little benefit to the local community."
The neighbourhood plan admits: "There is a vast increase in population for a small number of weeks in the year. It is said that in excess of 65pc of the houses in the villages are now holiday homes or lets.
"The ratio of permanent residents to holiday residents is perceived to be reducing rapidly. "
Town Lane's dwindling number of residents would no doubt agree.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here