New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
text
New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
new_line
new_line
Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
text
Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
new_line
new_line
In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
text
In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
new_line
new_line
🔍Project page: 
text
🔍Project page: 
https://research.nvidia.com/labs/toronto-ai/FMS/
https://research.nvidia.com/labs/toronto-ai/FMS/
link
new_line
👨‍💻 Code for reproduction: 
text
👨‍💻 Code for reproduction: 
https://github.com/NVlabs/forecasting-model-search
https://github.com/NVlabs/forecasting-model-search
link
new_line
📄 Full Paper: 
text
📄 Full Paper: 
https://arxiv.org/abs/2406.18630
https://arxiv.org/abs/2406.18630
link
text
new_line
new_line
Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
text
Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
new_line
new_line
Check out more work from Toronto AI Lab here: 
text
Check out more work from Toronto AI Lab here: 
https://research.nvidia.com/labs/toronto-ai/
https://research.nvidia.com/labs/toronto-ai/
link
text
new_line
new_line
You can view the TAO toolkit here: 
text
You can view the TAO toolkit here: 
https://developer.nvidia.com/tao-toolkit
https://developer.nvidia.com/tao-toolkit
link
New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights

Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.

In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.

🔍Project page: https://research.nvidia.com/labs/toronto-ai/FMS/
👨‍💻 Code for reproduction: https://github.com/NVlabs/forecasting-model-search
📄 Full Paper: https://arxiv.org/abs/2406.18630 

Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.

Check out more work from Toronto AI Lab here: https://research.nvidia.com/labs/toronto-ai/ 

You can view the TAO toolkit here: https://developer.nvidia.com/tao-toolkit
🔥
yifei01 osanseviero Joseph717171 ioda-idg atasoglu lorraine2
🤗
yifei01 Joseph717171 John6666 lorraine2
🚀
yifei01 Joseph717171 lorraine2