content.raw: New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
content.type: text
content.value: New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
content.type: new_line
content.type: new_line
content.raw: Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
content.type: text
content.value: Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
content.type: new_line
content.type: new_line
content.raw: In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
content.type: text
content.value: In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
content.type: new_line
content.type: new_line
content.raw: 🔍Project page: 
content.type: text
content.value: 🔍Project page: 
content.href: https://research.nvidia.com/labs/toronto-ai/FMS/
content.raw: https://research.nvidia.com/labs/toronto-ai/FMS/
content.type: link
content.type: new_line
content.raw: 👨‍💻 Code for reproduction: 
content.type: text
content.value: 👨‍💻 Code for reproduction: 
content.href: https://github.com/NVlabs/forecasting-model-search
content.raw: https://github.com/NVlabs/forecasting-model-search
content.type: link
content.type: new_line
content.raw: 📄 Full Paper: 
content.type: text
content.value: 📄 Full Paper: 
content.href: https://arxiv.org/abs/2406.18630
content.raw: https://arxiv.org/abs/2406.18630
content.type: link
content.type: text
content.type: new_line
content.type: new_line
content.raw: Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
content.type: text
content.value: Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
content.type: new_line
content.type: new_line
content.raw: Check out more work from Toronto AI Lab here: 
content.type: text
content.value: Check out more work from Toronto AI Lab here: 
content.href: https://research.nvidia.com/labs/toronto-ai/
content.raw: https://research.nvidia.com/labs/toronto-ai/
content.type: link
content.type: text
content.type: new_line
content.type: new_line
content.raw: You can view the TAO toolkit here: 
content.type: text
content.value: You can view the TAO toolkit here: 
content.href: https://developer.nvidia.com/tao-toolkit
content.raw: https://developer.nvidia.com/tao-toolkit
content.type: link
rawContent: New #NVIDIA paper: Improving Hyperparameter Optimization with Checkpointed Model Weights
  
  Hyperparameter optimization often dominates the cost of model design.  So, we want cheap surrogate functions that approximate model performance to guide our search.  Existing methods can train on optimization metadata – like a trajectory of losses – to build these surrogates.
  
  In our work, we add the ability to train our hyperparameter optimization surrogates on checkpointed model weights with a graph metanetwork.  This allows us to leverage a large, pre-existing source of information that can featurize the architecture, dataset, losses, and optimization procedure.
  
  🔍Project page: https://research.nvidia.com/labs/toronto-ai/FMS/
  👨‍💻 Code for reproduction: https://github.com/NVlabs/forecasting-model-search
  📄 Full Paper: https://arxiv.org/abs/2406.18630 
  
  Our project was a collaboration between NVIDIA’s Toronto AI Lab and the TAO team.
  
  Check out more work from Toronto AI Lab here: https://research.nvidia.com/labs/toronto-ai/ 
  
  You can view the TAO toolkit here: https://developer.nvidia.com/tao-toolkit
reactions.reaction: 🔥
reactions.users: yifei01 osanseviero Joseph717171 ioda-idg atasoglu lorraine2
reactions.reaction: 🤗
reactions.users: yifei01 Joseph717171 John6666 lorraine2
reactions.reaction: 🚀
reactions.users: yifei01 Joseph717171 lorraine2