  Many cases exist in which a black-box function $f$ with high evaluation cost
depends on two types of variables $\bm x$ and $\bm w$, where $\bm x$ is a
controllable \emph{design} variable and $\bm w$ are uncontrollable
\emph{environmental} variables that have random variation following a certain
distribution $P$. In such cases, an important task is to find the range of
design variables $\bm x$ such that the function $f(\bm x, \bm w)$ has the
desired properties by incorporating the random variation of the environmental
variables $\bm w$. A natural measure of robustness is the probability that
$f(\bm x, \bm w)$ exceeds a given threshold $h$, which is known as the
\emph{probability threshold robustness} (PTR) measure in the literature on
robust optimization. However, this robustness measure cannot be correctly
evaluated when the distribution $P$ is unknown. In this study, we addressed
this problem by considering the \textit{distributionally robust PTR} (DRPTR)
measure, which considers the worst-case PTR within given candidate
distributions. Specifically, we studied the problem of efficiently identifying
a reliable set $H$, which is defined as a region in which the DRPTR measure
exceeds a certain desired probability $\alpha$, which can be interpreted as a
level set estimation (LSE) problem for DRPTR. We propose a theoretically
grounded and computationally efficient active learning method for this problem.
We show that the proposed method has theoretical guarantees on convergence and
accuracy, and confirmed through numerical experiments that the proposed method
outperforms existing methods.

Machine Learning;Machine Learning