Lamphey was allegedly the seat of the last native Welsh bishops. The current ruins date back to the late 13th Century, when the Normans established the site as a clerical retreat. The bulk of the surviving ruins are the work of Henry de Gower, bishop of St David's from 1328AD to 1347AD, who greatly expanded upon the earlier structure.

Following the Reformation, Lamphey Palace fell into secular hands, and was used for a while as a country residence for the Earls of Essex. During the 17th Century, the site fell into disuse and decay.

Today, Lamphey Palace survives as a surprisingly complete set of ruins, benefiting from a picturesque and secluded location. The boundary wall is mostly in tact, and a distinctive bell tower dominates the central enclosure. Other notable remains include the great hall, chapel and gatehouses.

The location of Lamphey Palace seems to deter tourists from visiting in large numbers. Consequently, the site is very peaceful, offering a pleasant ambience in which to explore the ruins.