The castle was built around the end of the 13th century, in written sources it has been appearing firstly in the year 1318 in the mark of nobility of Ojir from Kosumberk. To this oldest phase of a small centre in a northwestern part of the present grounds belongs a robust oval rampart of an egg-shaped ground plan supported at the southwestern side by massive abutments. From the best accessible southeast forward side of a peripheral wall sticks out a massive oval portal tower, which protected the castle from the most threatened direction. The both walls forming its east corner run into towering width of 3.8 meters, thus, they are almost twice thicker than remaining two. High above surrounding terrain level in south-east entry side exists so far an apparent space for a drop bridge, on which was entered probably by means of a wooden ramp going from patio. Along a perimeter of peripheral wall was adjacent inner built-up area, which was protected by the wall thick against a contingency attack. Also from this reason, buildings in the centre probably did not overtop the peripheral rampart. At this phase was Kosumberk a typical castle with peripheral wall, which by its appearance markedly resembled the fenced centre of the Rabi castle.

The castle during the 14th century changed several times owners until in the year 1372 it was bought from a Divis of Popovec by brothers Divis, Mstislav, Slavata and Jesek from Chlum, members of the old-Bohemian aristocratic family, later known as the Slavata of Chlum and Kosumberk. In this way Kosumberk was become Slavata's generic mansion for three centuries. On the castle had their residence the Slavata's sons from who Jan reputedly escorted Jan Hus into Kostnice. Perhaps first Slavata's owners or still before them somewhen in during of the 14th century the center was completely rebuilt and into its heart was built in square tower with a rounded corner. By this, the Kosumberk superficially more approximated to the castle Rabi. In late Gothic, the castle centre was walled and a rampart was made thicker with small cylindrical and squared towers and bastions. At present, a squared tower in the south corner and basements of two cylindrical bastions in the south and north-west direction from the centre preserved only. From the view of the castle on a background of the picture from the first half of the 18th century saved in castle's museum were around the castle in total six cylindrical bastions. In addition, the entry side was made thicker with a triagonal gate which east corner occupied burgrave's building.

Before half of the 16th century the castle was again rebuilt which is manifested by the year 1544 on the gate from the fore-gate in the southeast corner of a courtyard. This is possible to join with reconstruction of burgrave's building in Renaissance style. The building received new facades with today's partially preserved letter-like graffito. In the year 1547 Divis Slavata was punished of the sovereign for attendance at unsuccessful revolt against Ferdinand (he was discharging the function of one from twelve directors) - he lost a part of his possession, Chlum and Kosumberk retained in his vassalage. A component of the punishment was also his in-house detention on the Kosumberk. Soon he gained forgiveness and still before his death in the year 1575 Kosumberk was from vassalage set free. After half the 16th century he found on the castle Bohemian Brethren school, at which except others studied Albrecht of Valdstejn and defenestration proconsul Vilem Slavata (from the year 1623 the domain owner).

Gothic appearance of the castle obviously did not fit higher demands of aristocracy and already in the year 1560 Divis had referred in one of his letters about plan rebuild Kosumberk according the castle in Jindrichuv Hradec. The last motivation to realize a rebuilding was fire caused by lightning in July in the year 1573. Old centre obtained renascence exterior culminated with dormers, the entry tower was shorten on level others buildings. Northeast from the centre was built cavernous two-wing renascence castle, which whelmed part of original fortification. On the courtyard side was two-floor arcade loggia on round smooth pillar column of eight arched fields. From arcades preserved one supporting pillar (newly erected in front of courtyard face of north wing), imprints vaults and dripstone consoles in the brickwork of both wings. On the south corner was joined to this palace slim tower wit clockwork standing on the way oriel on cylindrical leg. Facades of the castle have made letters graffito rotated with strips of stylized wavelets.

The castle has become the best-looking ones in the Chrudim region. Despite of these adaptations it held character of a fort and therefore in the uncertain time before the fight on the Bila Mountain noblemen and the surrounding inhabitants saved here the valuable things and important documents. Divis' grandson Divis Lacembok was for his attendance in aristocracy revolt in the years 1618-20 punished in the year 1623 by forfeiture of half possession. His relative Vilem Slavata of Chlum and Kosumberk bought Kosumberk. According to the juridical survey was located in the castle at this sale pictorial generic gallery, blaster weapons, equipment and outfit for 18 horse and 20 infantry divisions, 7 horses for the lord, the mare for the lady, 7 horses for the carriage, bassinet, etc. Vilem Slavata sold the Kosumberk Lacembok's son Jindrich Vilem in the year 1634. In the period of the Thirty-years' war Kosumberk went to ruins, martial expenses exhausted the establishment and in the years 1639, 1643, 1645 and 1648 Swede soldieries plundered the land.

In the year 1654 Jindrich Vilem, the last male descendant of this family, died and Kosumberk came into his daughter Johanka Barbora and after her death in three years later by way of her mother Marie Maxmiliana (born Zdarska from Zdar) the domain came into her second husband Frantisek Krystof Hieserl of Chod. He died in the year 1666 and Marie run the domain herself. In the year 1684, she forever moved in a farmstead below pilgrim chapel on Chlumek. By this, the Kosumberk lost its residential function, which marked the beginning of its gradual fall. After her death in the year 1690 a part of the domain and Kosumberk was bequeathed to the Jesuits. They had their residence by new pilgrim church on Chlumek and did not used Kosumberk. In the year 1769, they mentioned the first floor as inhabitable. After cancelling of the Jesuitical order by the emperor Josef II in the year 1773 the Kosumberk was administered by the Church Foundation and at the beginning of the 19th century the domain was bought by the Belgian aristocrat Leopold de Laing and in the year 1826 he sold it to the Thurn-Taxis family. In the year 1920, the remains of the castle got the Society for Kosumberk renewal, which conserved ruins and arranged castle's museum in the burgrave's building. From the year 1953, the castle was the state property, in the ninetieth years the castle has been in the property of the municipality of Luze.