While it makes no sense to worry that you are living well at y'all's grandchildren's expense, you might legitimately worry that someone else is living well at y'all's grandchildren's expense.
They have children of their own.