text: Title: [BUG-REPORT] Group By memory Issue
  Body: Hello,
  
  I have a project running on vaex v4.0.0, I also have it wrapped around flask to have API's running off it. I was hopping to get some help related to memory.
  
  I do face memory leak issues while using group by here's an example.
  
  df.groupby(['rooms_count'], agg={vx.agg.mean('price_per_meter'),vx.agg.min('price_per_meter'),vx.agg.max('price_per_meter'),vx.agg.count('price_per_meter')})
  
  My issue is not with the amount of memory being used. But after the API call is executed the memory used is not released back to the OS. Scale it to multiple API requests and soon I am out of memory on server. I have tried using garbage collection but still the memory isn't released back to the OS.
  
  I was asked to help replicate the issue. You can find the code and steps to replicate over there 
  [https://github.com/MHK107/vaex-groupby-memory-issue/tree/main](Link to the repo)
  
  Please let me know if I can help in any way possible to replicate and resolve this 