In 1562, Osterley was bought by Sir Thomas Gresham who rebuilt the house and developed the garden. The house passed through several hands until 1752 when, on inheriting the house from his father, Sir Francis Child commissioned Robert Adam to redesign the house.
In 1547, the Duke of Somerset was given Syon Monastery and the parkland. He built a new house with raised terraces to look out over the Thames. However the terrace was misconstrued as the first stage of building fortifications, and the Duke was charged with felony and executed in 1552.