His ancestors have been lords of the manor for almost 1,000 years.
But a rebellion by villagers has now ousted Lord Greville Howard from his role at the head of his local parish council.
The 82-year-old Tory peer has quit as chairman of Castle Rising parish council - following 35 years in the post - with an attack on newcomers to the village who had increasingly opposed his leadership.
His departure comes amid growing tension between members on the tiny authority.
The grandee, who once served as an aide to Enoch Powell, has found his position on the council embattled since May's election, which changed the make-up of the authority. Since then, other members have complained about a lack of democracy.
In his resignation letter, Lord Howard, whose ancestor William d'Aubigny, 1st Earl of Arundel, built the 12th century fortress Castle Rising is known for, said the character of the council had changed as a result of new arrivals.
He continues to live in his ancestral home in the village, which is one of the most exclusive enclaves in the county with a population of 200 people and is only a short distance from the royal family's Sandringham estate.
READ MORE: Mutiny in Mundesley leaves parish council in disarray
"There has always been a very friendly and amicable relationship at the council but recently this has changed," he wrote.
"I felt the council was going in a direction that was wrong for the village.
"People have moved into Castle Rising with different views on how they wanted to do things and the only correct thing for me to do was to resign.
"After 35 years, it is sad and I have done an awful lot for the village."
Things first soured during the run-up to the May council election.
A former clerk, who had served for 14 years in the voluntary role, was accused of breaking election laws by helping Lord Howard with his election campaign - it is illegal for a public servant to be involved with such campaigns.
The matter was referred to the police, but Lord Howard says the matter was dismissed as she had quit her role as clerk before the election.
"It was truly shocking behaviour. This is someone who has given 14 years of voluntary service to the village and her work was always above board," he said.
READ MORE: Extraordinary row over village website leaves parish council in chaos
Following the election - in which Lord Howard received the largest share of the vote - the new members pushed for new policies.
This included introducing a code of conduct, lowering the speed limit through the village and raising the precept - the share of council tax given to the parish council.
"The newcomers wanted to make the council more bureaucratic through a code of conduct and to increase taxation. People should not be taxed unnecessarily.
"All of this was taking away from the village atmosphere of the council."
Baron Howard of Rising - who owns much of the land in the village - was staunch in his opposition to lowering the speed limit to 30mph.
He argued that putting 30mph signs would damage the character of the picturesque hamlet - which is in a conservation area - and said people naturally slow down due to the road configuration.
READ MORE: The 24 aristocrats who own 10 per cent of Norfolk's land
But Dr Clare Smith, a fellow parish councillor, argued the new measures were wanted by the local community, who had been calling for the changes for years.
Tensions came to a head at a meeting earlier this month.
Accusations were made against Lord Howard that he had heavily edited the draft minutes of the previous meeting but this complaint was later retracted
Following this, the Tory peer who is one of the wealthiest landowners in the county, announced his resignation.
In his statement, he said: "It has been a privilege to serve on the parish council but sadly, as it is today, this parish council has become something I do not wish to be associated with.
"There have been complaints about the contributions I have made over the years to the well-being of the village.
"To avoid giving further offence I will in future treat the Parish Council at arms length."
Attempts have been made to contact the remaining five members of the parish council but there has been no response.
LORD OF THE MANOR
Born into nobility at the height of the Second World War on April 22, 1941, Lord Greville Howard comes from a long line of aristocrats.
He is a descendant of William d'Aubigny, 1st Earl of Arundel, who built the castle itself in the 12th century.
The Howard family have played a prominent role throughout British history and during the reign of Richard III, John Howard was named as the first Duke of Norfolk before he was later slain along with the king during the Battle of Bosworth, which ended the War of the Roses.
The Howards continued to have close ties with royalty: both Henry VIII's beheaded wives, Anne Boleyn and Catherine Howard, were members of the family.
Other Howards fought in other major battles, including as an admiral of the English fleet that defeated the Spanish Armada.
Lord Howard also forged a strong relationship with the royal family and was lifelong friends with Prince Phillip, the Duke of Edinburgh.
After studying at Eton College, Lord Howard entered the world of politics and between 1968 and 1970, he was private secretary to Enoch Powell - a senior Tory who was blamed for stoking up violence against immigrants following his 'Rivers of Blood' speech.
Lord Howard continued to serve in various ministerial positions in the Conservative Party and he was made a life peer in 2004.
As well as serving as a member of the House of Lords, he continued to be involved in politics at a local level, serving as a West Norfolk councillor between 2015 and May 2023, as well as being a parish councillor for Castle Rising for 35 years.
He has also remained involved in Westminster politics and his London townhouse was the base for last year's campaign to install Liz Truss as prime minister and has previously been used as campaign headquarters by Boris Johnson in 2019 and Michael Portillo in 1995.
Comments: Our rules
We want our comments to be a lively and valuable part of our community - a place where readers can debate and engage with the most important local issues. The ability to comment on our stories is a privilege, not a right, however, and that privilege may be withdrawn if it is abused or misused.
Please report any comments that break our rules.
Read the rules here