Question: I am looking for a way to test whether a boundary threshold exists in a physiological response – a sample of the data is plotted below. My hypothesis is that the X-variable imposes a physiological constraint on Y-values, thus producing a boundary 'ceiling' for maximum Y-values that decreases at higher X-values (indicated by the red line on figure). I assume any Y-values below the boundary are limited by some other factor not included in this model.

Essentially, my goal is to determine if the boundary exists and if so to derive a confidence interval for the boundary line model – similar to a linear regression model, but describing the upper bound of the Y-values, rather than the center of mass.

I'm sure something like this exists, but I haven't come across it before. Also, I would appreciate any suggestions on a better title or tags for this post – I assume there are more accurate terms for what I'm describing that would help folks find this post.

![threshold](https://i.stack.imgur.com/1BQBk.jpg)

Answer: 
You can use a permutation based test for such threshold. 

[Permutation-based test](http://www.hydrol-earth-syst-sci.net/16/1255/2012/hess-16-1255-2012.pdf)

It tests the hypothesis whether a "data-sparse" region above the threshold line is due to a random chance or not.

In brief:

The basic idea behind is to calculate the area of the "data-sparse" region and use it as a statistic. The next step is to randomly permute the X-coordinates of the scatter-plot and repeat the calculation of the area of "data-sparse" region. 

Probability p is the proportion of times the calculated area exceeded the original area. If p is sufficiently small the "data-sparse" region deemed to be significant.
My intention on how this problem might be solved is:

1. Calculate **linear model** to receive the regression line $r$.
2. Calculate the **normal vector** $v$ to the resulting regression line.
3. Shift $r$ by $v$ till all data points are under $r$.

To optimize $r$, you might rotate it by some angle $\alpha$ and stop for the best $\alpha$ you found, maybe using the [Residual Sum of Squares](http://en.wikipedia.org/wiki/Residual_sum_of_squares) as reference term.

Like I tried to show in this figure:

![enter image description here](https://i.stack.imgur.com/Dw615.png) 

Another approach could be to use [Support Vector machines](http://de.wikipedia.org/wiki/Support_Vector_Machine). I don't know if this is possible with your data, but maybe you can produce some dummy points located above your data and split them from your original points using a SVM. This is just some idea I came up with. Though, I would prefer the first method.