Villagers will be asked to vote on whether their parish council should borrow more than £1m for a new community hall.
The parish council at Heacham, near Hunstanton, wants to convert a former dairy into a new community centre, including new council offices, a library and heritage centre.
But some villagers are concerned at the size of the loan, which will be repaid via an enlarged parish precept through residents' council tax.
Others believe the site would be better used to provide more parking in the village centre.
Councillors voted on Tuesday night to hold a parish poll on whether to proceed with the project.
Planning permission for the development, on Pound Lane, was granted in 2019.
The council is waiting to hear whether its application for a £1.3 public works loan from the government to fund the work has been approved.
Terry Clay, who chairs the council working group behind the plan, said loan repayments would add £5.36 a year to the average Band D council tax bill - around 10p a week.
"It is an affordable cost for a public hall, meeting rooms and library," he said.
"The borough planners see the site as having heritage value but there are people who want a car park placed there."
Supporters say Heacham, which is one of Norfolk's largest villages with a population of more than 4,750, has outgrown its existing facilities.
The parish council says: "The community and heritage building project is a unique and forward looking opportunity to provide Heacham with a central service and focal point in the High Street."
The council is still waiting to hear whether its loan application has been approved. It says £200,000 of the £1.3m would be VAT, which could be reclaimed.
Retired businessman Dave Curtis, who describes himself as "a concerned parishioner", said he would be holding his own poll.
"The straight figures are they want to borrow £1.3m to build some new council offices, a new committee room, their meeting room and a small heritage centre," he said.
"We have a high street which lives and breathes and has a lot of shops but you cannot park in the high street, the businesses there are slowly dying.
"The site just happens to be the dead centre of the village. Wouldn't it be better if we took down the building and used it as a market place and car park?"
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here