While it makes no sense to worry that you are living well at y'all's grandchildren's expense, you might legitimately worry that someone else is living well at y'all's grandchildren's expense.
They are childless without an heir.