Dunster is dominated by its Victorian manor house (built on the ruins of a mediaeval castle), which rises up on a hill above the town, commanding the landscape for miles around. It also lends the village its name - 'dun' meaning fortified settlement, and 'tor' meaning hill.

The history of Dunster dates back at least as far as Saxon times. It's believed to have been settled around 700AD, although prehistoric earthworks around the uplands suggest still earlier activity at the site. The real growth of Dunster began after the Norman invasion, when William de Mohun built the original medieval castle. It was also at this time that the church settled here, founding Dunster's priory at William's invitation, around 1097AD.

In 1539AD, Dunster Priory was dissolved. The priory church fell into decline, but was restored in 1875–77AD by George Edmund Street, and has remained in use since then as the parish church of St George.