The Castle is thought to have been built by Sir William de Mortimer, one of the many immigrant knights of Norman-French ancestry who arrived in Scotland in the twelfth century. de Mortimer acquired Aberdour as a barony in the 1150s or 1160s and is remembered for his quarrel with the monks of nearby Inchcolm Abbey.

By the 1180s, the barony had passed to Robert of London, the illegitimate son of the Scottish king, William the Lion. From the 1230s, it was in the hands of Sir Richard Siward, an Anglo-Welsh knight renowned as a brilliant soldier and who fought against corrupt English officials. Aberdour was acquired by the nephew of Robert the Bruce, Sir Thomas Randolph (d 1332), the Earl of Moray, who was famed for his role at the Battle of Bannockburn and for storming Edinburgh Castle.

In the 16th century, Aberdour was owned by Regent Morton, 4th Earl of Morton. Morton governed Scotland in the 1570s before being executed on his own guillotine – the ‘Maiden’. He was an enthusiast for French architecture and landscape gardening and had a spiral staircase which led directly from the upper terrace to the dining room, the central space of his Lodgings. At the far end of the terraces is the dovecot shaped as a beehive while steps in the angle of the terraces lead down to the orchard.

In the 17th century, Morton’s descendents updated the Castle and added a long gallery which had stairs linking it to the gardens. The Walled Garden which lies beyond the forecourt was probably built by the 7th Earl in 1630s. In the south-west corner of the gardens is the vestry which is probably on the site of a 17th century summerhouse; in the south-east corner are the ruins of another summerhouse. There are also the remains of a bridge which led over the lane to the kitchen garden, now a bungalow.

Similar to Aberdour Castle is Edzell Castle. This also has a walled garden created in 1604 with a summerhouse and (ruinous) bath house at two corners.