the castle changed hands several times in the following centuries. it was owned by the "von hürnheim" family from 1256 to 1368. then the "von westerstetten" family held the castle from 1368 until 1442. the hospital in ulm and the "von westernach" family held it from 1512 until 1538.