title: Survey of Active Learning Hyperparameters: Insights from a Large-Scale
    Experimental Grid
summary: Annotating data is a time-consuming and costly task, but it is inherently
  required for supervised machine learning. Active Learning (AL) is an
  established method that minimizes human labeling effort by iteratively
  selecting the most informative unlabeled samples for expert annotation, thereby
  improving the overall classification performance. Even though AL has been known
  for decades, AL is still rarely used in real-world applications. As indicated
  in the two community web surveys among the NLP community about AL, two main
  reasons continue to hold practitioners back from using AL: first, the
  complexity of setting AL up, and second, a lack of trust in its effectiveness.
  We hypothesize that both reasons share the same culprit: the large
  hyperparameter space of AL. This mostly unexplored hyperparameter space often
  leads to misleading and irreproducible AL experiment results. In this study, we
  first compiled a large hyperparameter grid of over 4.6 million hyperparameter
  combinations, second, recorded the performance of all combinations in the
  so-far biggest conducted AL study, and third, analyzed the impact of each
  hyperparameter in the experiment results. In the end, we give recommendations
  about the influence of each hyperparameter, demonstrate the surprising
  influence of the concrete AL strategy implementation, and outline an
  experimental study design for reproducible AL experiments with minimal
  computational effort, thus contributing to more reproducible and trustworthy AL
  research in the future.
paper_authors._id: 68415ee454d7c6b3f9786deb
paper_authors.name: Julius Gonsior
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:58:30.615Z
paper_authors._id: 637638fa1f0421002b42facb
paper_authors.avatarUrl: /avatars/ed1e3024cd2ed7284b437db4bbeb2668.svg
paper_authors.fullname: Julius Gonsior
paper_authors.type: user
paper_authors.user: jgonsior
paper_authors._id: 68415ee454d7c6b3f9786dec
paper_authors.name: Tim Rieß
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:58:21.628Z
paper_authors._id: 6728a69e4d0fe229c96eea4c
paper_authors.avatarUrl: /avatars/0c6ee4d8dd0752654ca5d86f27bebaea.svg
paper_authors.fullname: Tim Rieß
paper_authors.type: user
paper_authors.user: Timischaf
paper_authors._id: 68415ee454d7c6b3f9786ded
paper_authors.name: Anja Reusch
paper_authors._id: 68415ee454d7c6b3f9786dee
paper_authors.name: Claudio Hartmann
paper_authors._id: 68415ee454d7c6b3f9786def
paper_authors.name: Maik Thiele
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-06-05T12:58:04.471Z
paper_authors._id: 65f5af9309cf7381af5e261b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65f5af9309cf7381af5e261b/2ciaq2fzRwkpf7jk0k9JZ.png
paper_authors.fullname: Maik Thiele
paper_authors.type: user
paper_authors.user: DILHTWD
paper_authors._id: 68415ee454d7c6b3f9786df0
paper_authors.name: Wolfgang Lehner
paper_title: Survey of Active Learning Hyperparameters: Insights from a Large-Scale
    Experimental Grid
paper_summary: Annotating data is a time-consuming and costly task, but it is inherently
  required for supervised machine learning. Active Learning (AL) is an
  established method that minimizes human labeling effort by iteratively
  selecting the most informative unlabeled samples for expert annotation, thereby
  improving the overall classification performance. Even though AL has been known
  for decades, AL is still rarely used in real-world applications. As indicated
  in the two community web surveys among the NLP community about AL, two main
  reasons continue to hold practitioners back from using AL: first, the
  complexity of setting AL up, and second, a lack of trust in its effectiveness.
  We hypothesize that both reasons share the same culprit: the large
  hyperparameter space of AL. This mostly unexplored hyperparameter space often
  leads to misleading and irreproducible AL experiment results. In this study, we
  first compiled a large hyperparameter grid of over 4.6 million hyperparameter
  combinations, second, recorded the performance of all combinations in the
  so-far biggest conducted AL study, and third, analyzed the impact of each
  hyperparameter in the experiment results. In the end, we give recommendations
  about the influence of each hyperparameter, demonstrate the surprising
  influence of the concrete AL strategy implementation, and outline an
  experimental study design for reproducible AL experiments with minimal
  computational effort, thus contributing to more reproducible and trustworthy AL
  research in the future.
paper_ai_summary: The study investigates the impact of hyperparameters on Active Learning performance, providing insights to improve its practical application and reproducibility.
paper_ai_keywords: Active Learning hyperparameter space hyperparameter grid experimental study design reproducibility trustworthiness