The manor house of the former lords of Wiltz was located in the lower town at the site of the present decanal church.

As this castle could no longer withstand the constant attacks of the enemy during the Norman invasion, the Lords of Wiltz built another fortified castle towards the end of the 12th century at the eastern end of a strip of rock that rises steeply on the banks of the River Wiltz. In the 14th century, the fortifications were demolished and a new, larger enclosure was built.

In 1388, the castle was destroyed by the soldiers of the King of France during an expedition against the Duchy of Juliers.

At the beginning of the 15th century, the town and the castle were burnt down by the Count of Nassau. In 1453, barely out of its ruins, Wiltz was destroyed again under Philip of Burgundy by de Croy, governor of Luxembourg.

On 23 May 1631, John VI of Wiltz had a new castle built at the same site. Of the original castle, only the old witch tower and the square tower, which is included in the north-western buildings, remain. This tower once housed the main entrance which could be reached by a drawbridge.