<h4 dir="auto">Description</h4>
<p dir="auto">Output of SVC.decision_function for one observation depends on the other observations --&gt; it is not independant.</p>
<h4 dir="auto">Steps/Code to Reproduce</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import numpy
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data[:, :3]
y = iris.target
model = SVC()
model.fit(X, y)

X1 = X[:1] * 0.10
X2 = X[1:2]
X3 = X[2:3]
X1X2 = numpy.vstack([X1, X2])
X2X3 = numpy.vstack([X2, X3])
X1X3 = numpy.vstack([X1, X3])
X1X2X3 = numpy.vstack([X1, X2, X3])

print('X1')
print(model.decision_function(X1))
print('X2')
print(model.decision_function(X2))
print('X3')
print('')
print('X1 X2')
print(model.decision_function(X1X2))
print('X2 X3')
print(model.decision_function(X2X3))
print('')
print('X1 X2 X3')
print(model.decision_function(X1X2X3))"><pre class="notranslate"><code class="notranslate">import numpy
from sklearn.svm import SVC
from sklearn.datasets import load_iris

iris = load_iris()
X = iris.data[:, :3]
y = iris.target
model = SVC()
model.fit(X, y)

X1 = X[:1] * 0.10
X2 = X[1:2]
X3 = X[2:3]
X1X2 = numpy.vstack([X1, X2])
X2X3 = numpy.vstack([X2, X3])
X1X3 = numpy.vstack([X1, X3])
X1X2X3 = numpy.vstack([X1, X2, X3])

print('X1')
print(model.decision_function(X1))
print('X2')
print(model.decision_function(X2))
print('X3')
print('')
print('X1 X2')
print(model.decision_function(X1X2))
print('X2 X3')
print(model.decision_function(X2X3))
print('')
print('X1 X2 X3')
print(model.decision_function(X1X2X3))
</code></pre></div>
<h4 dir="auto">Expected Results</h4>
<p dir="auto">decision_function should verify:</p>
<p dir="auto">decision_function(X1) == decision_function(X1X2)[:1]</p>
<h4 dir="auto">Actual Results</h4>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="X1
[[-0.49153396  0.99153396  2.5       ]]
X2
[[ 2.5         0.77197248 -0.27197248]]
X3

X1 X2
[[-0.04698652  0.99919072  2.0477958 ]
 [ 2.5         0.77197248 -0.27197248]]
X2 X3
[[ 2.48614274  0.77829215 -0.26443489]
 [ 2.5         0.75104163 -0.25104163]]

X1 X2 X3
[[-0.04568431  0.99921315  2.04647116]
 [ 2.48614274  0.77829215 -0.26443489]
 [ 2.5         0.75104163 -0.25104163]]"><pre class="notranslate"><code class="notranslate">X1
[[-0.49153396  0.99153396  2.5       ]]
X2
[[ 2.5         0.77197248 -0.27197248]]
X3

X1 X2
[[-0.04698652  0.99919072  2.0477958 ]
 [ 2.5         0.77197248 -0.27197248]]
X2 X3
[[ 2.48614274  0.77829215 -0.26443489]
 [ 2.5         0.75104163 -0.25104163]]

X1 X2 X3
[[-0.04568431  0.99921315  2.04647116]
 [ 2.48614274  0.77829215 -0.26443489]
 [ 2.5         0.75104163 -0.25104163]]
</code></pre></div>
<h4 dir="auto">Reasons</h4>
<p dir="auto">decision_function calls _ovr_decision_function which normalizes outputs based on the whole set of observations.</p>
<p dir="auto"><a href="https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/multiclass.py#L432">https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/utils/multiclass.py#L432</a></p>
<h4 dir="auto">Versions</h4>
<p dir="auto">Windows-10-10.0.17134-SP0<br>
Python 3.7.0 (v3.7.0:1bf9cc5093, Jun 27 2018, 04:59:51) [MSC v.1914 64 bit (AMD64)]<br>
NumPy 1.15.3<br>
SciPy 1.1.0<br>
Scikit-Learn 0.20.0</p>
<h3 dir="auto">Describe the workflow you want to enable</h3>
<p dir="auto">In my dreams, a new <code class="notranslate">InferenceData</code> class would simplify training and prediction to look more like</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="import ... as learner

data = InferenceData(
    df=..., # a data frame 
    meta=Meta(
        y_cols=..., # name(s) of output variable
        ... # additional metadata fields
    )
)
train_data, test_data = data.split(train_fraction=0.7, ...)
learner.fit(train_data)
predictions = learner.predict(test_data.x)"><pre class="notranslate"><code class="notranslate">import ... as learner

data = InferenceData(
    df=..., # a data frame 
    meta=Meta(
        y_cols=..., # name(s) of output variable
        ... # additional metadata fields
    )
)
train_data, test_data = data.split(train_fraction=0.7, ...)
learner.fit(train_data)
predictions = learner.predict(test_data.x)
</code></pre></div>
<p dir="auto">Note that this introduces just two data variables <code class="notranslate">train_data</code> and <code class="notranslate">test_data</code> instead of the current standard four (<code class="notranslate">X_train, X_test, y_train, y_test</code>).</p>
<p dir="auto">In addition, <code class="notranslate">InferenceData</code> could easily be extended to allow the above pipeline to handle related metadata such as feature weights, replacing a step like <code class="notranslate">learner.fit(train_data)</code> by <code class="notranslate">learner.fit(train_data, weights=train_data.row_weights)</code>, for example.</p>
<h3 dir="auto">Describe your proposed solution</h3>
<p dir="auto">A solution could look something like this:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="@dataclass
class Meta:
    &quot;&quot;&quot;Metadata for a Data class.&quot;&quot;&quot;
    y_cols: Optional[list[str]] = None
    row_weights_col: Optional[str] = None

    @property
    def y(self) -&gt; list[str]:
        &quot;&quot;&quot;Output variable column names.&quot;&quot;&quot;
        if not self.y_cols:
            return []
        return self.y_cols
    
    @property
    def columns(self) -&gt; set[str]:
        &quot;&quot;&quot;All metadata columns.&quot;&quot;&quot;
        cols = set(self.y)
        if self.row_weights_col:
            cols.add(row_weights_col)
        return cols


@dataclass
class InferenceData:
    &quot;&quot;&quot;A data frame container that includes metadata relevant for machine learning and inference.&quot;&quot;&quot;
   
    df: DataFrame
    meta: Meta = Meta()
        
    def __post_init__(self) -&gt; None:
        &quot;&quot;&quot;Parameter validation.&quot;&quot;&quot;
        if (self.y is not None) and (self.n_rows != len(self.y)):
            raise ValueError(&quot;Expected y to have the same number of data points as x has.&quot;)
        # TODO: also validate that all columns referenced in self.meta exist in self.df etc
    
    @property
    def x(self) -&gt; DataFrame:
        &quot;&quot;&quot;The data frame of predictor variables, excluding output variables and weights, etc.&quot;&quot;&quot;
        non_metadata_cols = [col for col in self.df if col not in self.meta.columns]
        return self.df[non_metadata_cols]
    
    @property
    def y(self) -&gt; None | np.ndarray:
        &quot;&quot;&quot;Output features.&quot;&quot;&quot;
        if not self.meta.y_cols:
            return None
        if len(self.meta.y_cols) == 1:
            y_col = self.meta.y_cols[0]
            return self.df[y_col].to_numpy()
        return self.df[self.meta.y_cols].to_numpy()

    @property
    def w(self) -&gt; None | np.ndarray:
        &quot;&quot;&quot;The data frame of predictor variables, excluding output variables and weights, etc.&quot;&quot;&quot;
        return self.df[row_weights_col]

    def iloc(self, positional_indexes: Iterable) -&gt; &quot;InferenceData&quot;:
        &quot;&quot;&quot;Return the subset of the data in self corresponding to the specificied positional indices.&quot;&quot;&quot;
        return InferenceData(df=df.iloc[positional_indexes], meta=self.meta)
    
    @property
    def n_rows(self) -&gt; int:
        &quot;&quot;&quot;Number of rows of data.&quot;&quot;&quot;
        return self.df.shape[0]"><pre class="notranslate"><code class="notranslate">@dataclass
class Meta:
    """Metadata for a Data class."""
    y_cols: Optional[list[str]] = None
    row_weights_col: Optional[str] = None

    @property
    def y(self) -&gt; list[str]:
        """Output variable column names."""
        if not self.y_cols:
            return []
        return self.y_cols
    
    @property
    def columns(self) -&gt; set[str]:
        """All metadata columns."""
        cols = set(self.y)
        if self.row_weights_col:
            cols.add(row_weights_col)
        return cols


@dataclass
class InferenceData:
    """A data frame container that includes metadata relevant for machine learning and inference."""
   
    df: DataFrame
    meta: Meta = Meta()
        
    def __post_init__(self) -&gt; None:
        """Parameter validation."""
        if (self.y is not None) and (self.n_rows != len(self.y)):
            raise ValueError("Expected y to have the same number of data points as x has.")
        # TODO: also validate that all columns referenced in self.meta exist in self.df etc
    
    @property
    def x(self) -&gt; DataFrame:
        """The data frame of predictor variables, excluding output variables and weights, etc."""
        non_metadata_cols = [col for col in self.df if col not in self.meta.columns]
        return self.df[non_metadata_cols]
    
    @property
    def y(self) -&gt; None | np.ndarray:
        """Output features."""
        if not self.meta.y_cols:
            return None
        if len(self.meta.y_cols) == 1:
            y_col = self.meta.y_cols[0]
            return self.df[y_col].to_numpy()
        return self.df[self.meta.y_cols].to_numpy()

    @property
    def w(self) -&gt; None | np.ndarray:
        """The data frame of predictor variables, excluding output variables and weights, etc."""
        return self.df[row_weights_col]

    def iloc(self, positional_indexes: Iterable) -&gt; "InferenceData":
        """Return the subset of the data in self corresponding to the specificied positional indices."""
        return InferenceData(df=df.iloc[positional_indexes], meta=self.meta)
    
    @property
    def n_rows(self) -&gt; int:
        """Number of rows of data."""
        return self.df.shape[0]
</code></pre></div>
<p dir="auto">Of course, the usefulness of this solution will depend on wrapping existing machine learning algorithms to accept an InferenceData class as input.</p>
<h3 dir="auto">Describe alternatives you've considered, if relevant</h3>
<p dir="auto"><em>No response</em></p>
<h3 dir="auto">Additional context</h3>
<p dir="auto"><em>No response</em></p>