Question: I am looking for a way to test whether a boundary threshold exists in a physiological response – a sample of the data is plotted below. My hypothesis is that the X-variable imposes a physiological constraint on Y-values, thus producing a boundary 'ceiling' for maximum Y-values that decreases at higher X-values (indicated by the red line on figure). I assume any Y-values below the boundary are limited by some other factor not included in this model.

Essentially, my goal is to determine if the boundary exists and if so to derive a confidence interval for the boundary line model – similar to a linear regression model, but describing the upper bound of the Y-values, rather than the center of mass.

I'm sure something like this exists, but I haven't come across it before. Also, I would appreciate any suggestions on a better title or tags for this post – I assume there are more accurate terms for what I'm describing that would help folks find this post.

![threshold](https://i.stack.imgur.com/1BQBk.jpg)

Answer: 
You can use a permutation based test for such threshold. 

[Permutation-based test](http://www.hydrol-earth-syst-sci.net/16/1255/2012/hess-16-1255-2012.pdf)

It tests the hypothesis whether a "data-sparse" region above the threshold line is due to a random chance or not.

In brief:

The basic idea behind is to calculate the area of the "data-sparse" region and use it as a statistic. The next step is to randomly permute the X-coordinates of the scatter-plot and repeat the calculation of the area of "data-sparse" region. 

Probability p is the proportion of times the calculated area exceeded the original area. If p is sufficiently small the "data-sparse" region deemed to be significant.
This is potentially not the most robust solution. But you may be able to seriously improve the quality of the envelope using something along these lines:

1. break down your data into n intervals (where the number of intervals depends on the density of your data)
2. find the max of you data within each interval
3. Pass linear regression model though the selected maximum data points.