title: Re-fitting a saved scikit-learn model without some features not used - "ValueError: A given column is not a column of the dataframe"
body: <p>I'd need to re-fit a scikit-learn pipeline using a smaller dataset, without some features that are actually not used by the model.</p>
  <p>(The actual situation is that I'm saving it through joblib and loading it in another file where I need to re-fit is since it contains some custom transformers I made, but adding all features would be a pain since it's a different kind of model. However this is not important since the same error happens also if I re-fit the model before saving it in the same file where I first trained it).</p>
  <p>This is my custom transformer:</p>
  <pre class="lang-py prettyprint-override"><code>class TransformAdoptionFeatures(BaseEstimator, TransformerMixin):
      def __init__(self):
          pass
  
      def fit(self, X, y=None):
          return self
  
      def transform(self, X):
          adoption_features = X.columns
          feats_munic = [feat for feat in adoption_features if '_munic' in feat]
          feats_adj_neigh = [feat for feat in adoption_features
                             if '_adj' in feat]
          feats_port = [feat for feat in adoption_features if '_port' in feat]
  
          feats_to_keep_all = feats_munic + feats_adj_neigh + feats_port
          feats_to_keep = [feat for feat in feats_to_keep_all
                           if 'tot_cumul' not in feat]
          
          return X[feats_to_keep]
  </code></pre>
  <p>And this is my pipeline:</p>
  <pre class="lang-py prettyprint-override"><code>full_pipeline = Pipeline([
      ('transformer', TransformAdoptionFeatures()),
      ('scaler', StandardScaler())
  ])
  
  model = Pipeline([
      (&quot;preparation&quot;, full_pipeline),
      (&quot;regressor&quot;, ml_model)
  ])
  </code></pre>
  <p>Where <code>ml_model</code> is whichever scikit-learn machine learning model. Both the <code>full_pipeline</code> and the <code>ml_model</code> are already fitted when saving the <code>model</code>. (In the actual model there is a <code>ColumnTransformer</code> intermediate step that represent the actual <code>full_pipeline</code>, since I need to have different transformers for different columns, but I copied only the important one for brevity).</p>
  <p><strong>Issue:</strong> I reduced the number of features of the dataset I already used to fit everything, removing some features that are not considered in <code>TransformAdoptionFeatures()</code> (they do not get into the features to keep). Then, I tried to re-fit the model to the new dataset with reduced features and I got this error:</p>
  <pre class="lang-py prettyprint-override"><code>Traceback (most recent call last):
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\pandas\core\indexes\base.py&quot;, line 2889, in get_loc
      return self._engine.get_loc(casted_key)
  
    File &quot;pandas\_libs\index.pyx&quot;, line 70, in pandas._libs.index.IndexEngine.get_loc
  
    File &quot;pandas\_libs\index.pyx&quot;, line 97, in pandas._libs.index.IndexEngine.get_loc
  
    File &quot;pandas\_libs\hashtable_class_helper.pxi&quot;, line 1675, in pandas._libs.hashtable.PyObjectHashTable.get_item
  
    File &quot;pandas\_libs\hashtable_class_helper.pxi&quot;, line 1683, in pandas._libs.hashtable.PyObjectHashTable.get_item
  
  KeyError: 'tot_cumul_adoption_pr_y_munic'
  
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\utils\__init__.py&quot;, line 447, in _get_column_indices
      col_idx = all_columns.get_loc(col)
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\pandas\core\indexes\base.py&quot;, line 2891, in get_loc
      raise KeyError(key) from err
  
  KeyError: 'tot_cumul_adoption_pr_y_munic'
  
  
  The above exception was the direct cause of the following exception:
  
  Traceback (most recent call last):
  
    File &quot;C:\Users\giaco\sbp-abm\municipalities_abm\test.py&quot;, line 15, in &lt;module&gt;
      modelSBP = model.SBPAdoption(initial_year=start_year)
  
    File &quot;C:\Users\giaco\sbp-abm\municipalities_abm\municipalities_abm\model.py&quot;, line 103, in __init__
      self._upload_ml_models(ml_clsf_folder, ml_regr_folder)
  
    File &quot;C:\Users\giaco\sbp-abm\municipalities_abm\municipalities_abm\model.py&quot;, line 183, in _upload_ml_models
      self._ml_clsf.fit(clsf_dataset.drop('adoption_in_year', axis=1),
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\pipeline.py&quot;, line 330, in fit
      Xt = self._fit(X, y, **fit_params_steps)
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\pipeline.py&quot;, line 292, in _fit
      X, fitted_transformer = fit_transform_one_cached(
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\joblib\memory.py&quot;, line 352, in __call__
      return self.func(*args, **kwargs)
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\pipeline.py&quot;, line 740, in _fit_transform_one
      res = transformer.fit_transform(X, y, **fit_params)
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\compose\_column_transformer.py&quot;, line 529, in fit_transform
      self._validate_remainder(X)
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\compose\_column_transformer.py&quot;, line 327, in _validate_remainder
      cols.extend(_get_column_indices(X, columns))
  
    File &quot;C:\Users\giaco\anaconda3\envs\mesa_geo_ml\lib\site-packages\sklearn\utils\__init__.py&quot;, line 454, in _get_column_indices
      raise ValueError(
  
  ValueError: A given column is not a column of the dataframe
  </code></pre>
  <p>I do not understand what does this error is due to, I thought scikit-learn was not storing the name of the columns that I pass.</p>