Watching over the town from the highest point of Carovigno is the Dentice di Frasso Castle, which has a triangular plan with mighty towers at each corner. Founded by the Normans, the castle was enlarged and fortified in the 15th and 16th centuries, adapting it to the new and different strategies of attack and defense dictated by the weapons in use in those historic times. The emblematic Almond Tower, located on the northeast corner of the castle, was built in the 16th century by the Loffreda family, the feudal lords of Carovigno. The tower is particularly interesting and reflects the stylistic influence of the Sienese architect Francesco di Giorgio Martini, engaged at that time in supervising the construction of fortresses in Taranto, Otranto, Gallipoli and Brindisi. During the early 1900s, the castle became the home of the Dentice di Frasso family, who carried out new restoration work, attempting to restore the old castle to its original appearance and deciding to live within it.
Today the Castle is the home of the Municipal Library and the Consortium for the Management of the Torre Guaceto State Nature Reserve, and soon it will be the home of a Civic Archaeological Museum.