and its typically accessed much faster than normal CPU memory.
Performance tuning:
stage3_max_live_parameters: 1e9
stage3_max_reuse_distance: 1e9
If hitting OOM reduce stage3_max_live_parameters and stage3_max_reuse_distance. They should have minimal impact
on performance unless you are doing activation checkpointing. 1e9 would consume ~2GB. The memory is shared by
stage3_max_live_parameters and stage3_max_reuse_distance, so its not additive, its just 2GB total.
stage3_max_live_parameters is the upper limit on how many full parameters you want to keep on the GPU at any given
time. "reuse distance" is a metric we are using to figure out when will a parameter be used again in the future, and we
use the stage3_max_reuse_distance to decide whether to throw away the parameter or to keep it. If a parameter is
