the castle was held by the counts who attempted to expand their power in the simmental (simme valley). at some point, probably in the 14th century, they built unterer mannenberg castle as an expansion of the oberer castle. in 1339 the counts of gruyères supported the city of fribourg against bern in the battle of laupen. the bernese victory and their expanding power in the simmental created tension between the counts and the surrounding farmers. in 1350 bernese troops attacked and partially destroyed oberer mannenberg castle. it was apparently rebuilt, because in 1356 the counts of gruyères sold the castles to the fribourg supporting düdingen family. the düdingens already held nearby blankenburg castle, so mannenberg reinforced their power in the area. however, their hold was short lived. in 1376 the villagers of the surrounding communities rose up against the fribourg nobility, forcing the düdingens to sell mannenberg and blankenburg to the city of fribourg.