through the years , especially since charles had taken over the reins of the family 's business interests , their holdings had diminished to the one building bearing the sign that now swayed in the wind .