His family have been lords of the manor there for almost a millennium.
And locals in Castle Rising seem sympathetic to the plight of Lord Greville Howard after a rebellion by "newcomers" saw him ousted from the parish council after 35 years service.
The 82-year-old peer quit as chairman of the authority with a blistering attack on recent arrivals in the village who were increasingly opposing his leadership.
The final straw was a row with fellow councillors over plans to reduce the speed limit on the picturesque hamlet's leafy lanes.
Many were keen on the restrictions, but Baron Howard of Rising - whose ancestors built the 12th century fortress Castle Rising - opposed them saying new signs would damage the area's character and charm.
Among locals we approached, there seemed some support for the ousted peer and some resentment towards the "newcomers".
"Perhaps the newcomers need to ask themselves why they chose to move here," one woman out collecting woodchip in a wheelbarrow between the sleet showers said.
"It's quintessentially English, it's such a beautiful village, some of the newcomers have come a long, long way to live here, they need to ask themselves why."
Others were more reticent, but hinted the debate was more nuanced than the peer's attack on "newcomers" had indicated.
One man walking near the church said: "I live in the village and it's been the same for generations, I need to be careful what I say.
"It's the most beautiful village but it's been divided, it's a 60mph limit through here, that's what's divided the village."
READ MORE: Plans for 600 homes get go-ahead near Castle Rising
READ MORE: Bungalow blighting historic castle set to knocked down
Another said: "The parish council should make a statement. When the motion for the speed limit was passed by the council, five to two, only one of those five has been in the village for less than a decade."
Others declined to comment. "I have an opinion but I'd rather keep it to myself," one woman said.
Another man said: "I'm trying to keep out of it."
Parish council chair Jon Taylor said "I won't discuss anything with the press until the full council has had the chance to discuss it.
"The minutes of all the meetings are available, there's a web page, they're the public record."
Things went sour in the village - which is home to only around 200 people - after May's local elections, which saw new faces elected to sit alongside Lord Howard on the council.
They pushed for new policies including lowering the speed limit through the village, introducing a code of conduct and raising the precept - the share of residents' council tax bills charged by the parish.
In his resignation letter, Lord Howard - who helped in the Tory leadership campaigns of both Boris Johnson and Liz Truss - said: "There has always been a very friendly and amicable relationship at the council but recently this has changed," he wrote.
"I felt the council was going in a direction that was wrong for the village.
"People have moved into Castle Rising with different views on how they wanted to do things and the only correct thing for me to do was to resign.
"It has been a privilege to serve on the parish council but sadly, as it is today, this parish council has become something I do not wish to be associated with.
"There have been complaints about the contributions I have made over the years to the well-being of the village.
"To avoid giving further offence I will in future treat the parish council at arms length."
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here