Create dataset by sampling from near-boundary of binary classifier to improve accuracy Say I have some binary classifier $f: X \to [0, 1]$. I think the following bi-stage training method is straightforward to reduce error.
Step1. Sample uniformly from $X$ and create dataset with relatively smaller number of dataset. Then train the model $f'$.
Step2. Taking advantage of the trained $f'$, sampling from $X$ such that more datapoint will be sampled from the decision boundary of $f'$. Then using such dataset, train the model $f$.
I think, the above method is trivial and must have been investigated and improved much by researchers. Could you tell me the name for the relevant technique, if it has name?

Such problem setting is called batch-mode active learning. Standard active learning setting which only samples single data point per iteration cannot be usually applied to deep-learning. This is mainly because 1) single data point hardly change the network weight 2) training the entire network per each active learning iteration take really long cost. Thus, batch-mode active learning have been studied primarily for image classification.
The related paper can be found in

Hoi, Steven CH, et al. "Batch mode active learning and its application
to medical image classification." Proceedings of the 23rd
international conference on Machine learning. 2006.
Wei, Kai, Rishabh Iyer, and Jeff Bilmes. "Submodularity in data subset
selection and active learning." International conference on machine
learning. PMLR, 2015.
Sener, Ozan, and Silvio Savarese. "Active learning for convolutional
neural networks: A core-set approach." arXiv preprint arXiv:1708.00489
(2017).

