According to legend an Irish bishop had put a curse on William Marshal saying that all his sons would die childless. The castle thus passed into the hands of William de Valence, a half-brother of Henry III who became Lord of Pembroke through his marriage to Joan, granddaughter of William Marshal.
Valence family held the castle for 70 years, strengthening it by building the walls and towers around the outer ward. They also fortified the town, creating a ring of walls with three main gates and a postern.On the death of Aymer, William de Valence's son, the castle passed through marriage into the hands of the Hastings family. In 1389 the death of John Hastings at the age of 17 ended a line of inheritance stretching back for over a quarter of a millennium.