diff --git a/CHANGELOG.md b/CHANGELOG.md
index b16ee732..45bb97c9 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -15,6 +15,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add support for multi-node training.
 [\#103](https://github.com/mllam/neural-lam/pull/103) @simonkamuk @sadamov

+- Add option to clamp output prediction using limits specified in config file [\#92](https://github.com/mllam/neural-lam/pull/92) @SimonKamuk
+
 ### Fixed
 - Only print on rank 0 to avoid duplicates of all print statements.
 [\#103](https://github.com/mllam/neural-lam/pull/103) @simonkamuk @sadamov
diff --git a/README.md b/README.md
index d3e7379c..dfe9a137 100644
--- a/README.md
+++ b/README.md
@@ -148,12 +148,23 @@ training:
     weights:
       u100m: 1.0
       v100m: 1.0
-```
-
-For now the neural-lam config only defines two things: 1) the kind of data
-store and the path to its config, and 2) the weighting of different features in
-the loss function. If you don't define the state feature weighting it will default
-to weighting all features equally.
+      t2m: 1.0
+      r2m: 1.0
+  output_clamping:
+    lower:
+      t2m: 0.0
+      r2m: 0
+    upper:
+      r2m: 1.0
+```
+
+For now the neural-lam config only defines few things:
+
+1. The kind of datastore and the path to its config
+2. The weighting of different features in
+the loss function. If you don't define the state feature weighting it will default to
+weighting all features equally.
+3. Valid numerical range for output of each feature.The numerical range of all features default to $]-\infty, \infty[$.

 (This example is taken from the `tests/datastore_examples/mdp` directory.)

diff --git a/neural_lam/config.py b/neural_lam/config.py
index d3e09697..9deb15d6 100644
--- a/neural_lam/config.py
+++ b/neural_lam/config.py
@@ -68,6 +68,23 @@ class UniformFeatureWeighting:
     pass

+@dataclasses.dataclass
+class OutputClamping:
+    """
+    Configuration for clamping the output of the model.
+
+    Attributes
+    ----------
+    lower : Dict[str, float]
+        The minimum value to clamp each output feature to.
+    upper : Dict[str, float]
+        The maximum value to clamp each output feature to.
+    """
+
+    lower: Dict[str, float] = dataclasses.field(default_factory=dict)
+    upper: Dict[str, float] = dataclasses.field(default_factory=dict)
+
+
 @dataclasses.dataclass
 class TrainingConfig:
     """
@@ -86,6 +103,10 @@ class TrainingConfig:
         ManualStateFeatureWeighting, UniformFeatureWeighting
     ] = dataclasses.field(default_factory=UniformFeatureWeighting)

+    output_clamping: OutputClamping = dataclasses.field(
+        default_factory=OutputClamping
+    )
+

 @dataclasses.dataclass
 class NeuralLAMConfig(dataclass_wizard.JSONWizard, dataclass_wizard.YAMLWizard):
diff --git a/neural_lam/models/base_graph_model.py b/neural_lam/models/base_graph_model.py
index 728a9489..35b1ab12 100644
--- a/neural_lam/models/base_graph_model.py
+++ b/neural_lam/models/base_graph_model.py
@@ -79,6 +79,192 @@ def __init__(self, args, config: NeuralLAMConfig, datastore: BaseDatastore):
             layer_norm=False,
         )  # No layer norm on this one

+        # Compute indices and define clamping functions
+        self.prepare_clamping_params(config, datastore)
+
+    def prepare_clamping_params(
+        self, config: NeuralLAMConfig, datastore: BaseDatastore
+    ):
+        """
+        Prepare parameters for clamping predicted values to valid range
+        """
+
+        # Read configs
+        state_feature_names = datastore.get_vars_names(category="state")
+        lower_lims = config.training.output_clamping.lower
+        upper_lims = config.training.output_clamping.upper
+
+        # Check that limits in config are for valid features
+        unknown_features_lower = set(lower_lims.keys()) - set(
+            state_feature_names
+        )
+        unknown_features_upper = set(upper_lims.keys()) - set(
+            state_feature_names
+        )
+        if unknown_features_lower or unknown_features_upper:
+            raise ValueError(
+                "State feature limits were provided for unknown features: "
+                f"{unknown_features_lower.union(unknown_features_upper)}"
+            )
+
+        # Constant parameters for clamping
+        sigmoid_sharpness = 1
+        softplus_sharpness = 1
+        sigmoid_center = 0
+        softplus_center = 0
+
+        normalize_clamping_lim = (
+            lambda x, feature_idx: (x - self.state_mean[feature_idx])
+            / self.state_std[feature_idx]
+        )
+
+        # Check which clamping functions to use for each feature
+        sigmoid_lower_upper_idx = []
+        sigmoid_lower_lims = []
+        sigmoid_upper_lims = []
+
+        softplus_lower_idx = []
+        softplus_lower_lims = []
+
+        softplus_upper_idx = []
+        softplus_upper_lims = []
+
+        for feature_idx, feature in enumerate(state_feature_names):
+            if feature in lower_lims and feature in upper_lims:
+                assert (
+                    lower_lims[feature] < upper_lims[feature]
+                ), f'Invalid clamping limits for feature "{feature}",\
+                     lower: {lower_lims[feature]}, larger than\
+                     upper: {upper_lims[feature]}'
+                sigmoid_lower_upper_idx.append(feature_idx)
+                sigmoid_lower_lims.append(
+                    normalize_clamping_lim(lower_lims[feature], feature_idx)
+                )
+                sigmoid_upper_lims.append(
+                    normalize_clamping_lim(upper_lims[feature], feature_idx)
+                )
+            elif feature in lower_lims and feature not in upper_lims:
+                softplus_lower_idx.append(feature_idx)
+                softplus_lower_lims.append(
+                    normalize_clamping_lim(lower_lims[feature], feature_idx)
+                )
+            elif feature not in lower_lims and feature in upper_lims:
+                softplus_upper_idx.append(feature_idx)
+                softplus_upper_lims.append(
+                    normalize_clamping_lim(upper_lims[feature], feature_idx)
+                )
+
+        self.register_buffer(
+            "sigmoid_lower_lims", torch.tensor(sigmoid_lower_lims)
+        )
+        self.register_buffer(
+            "sigmoid_upper_lims", torch.tensor(sigmoid_upper_lims)
+        )
+        self.register_buffer(
+            "softplus_lower_lims", torch.tensor(softplus_lower_lims)
+        )
+        self.register_buffer(
+            "softplus_upper_lims", torch.tensor(softplus_upper_lims)
+        )
+
+        self.register_buffer(
+            "clamp_lower_upper_idx", torch.tensor(sigmoid_lower_upper_idx)
+        )
+        self.register_buffer(
+            "clamp_lower_idx", torch.tensor(softplus_lower_idx)
+        )
+        self.register_buffer(
+            "clamp_upper_idx", torch.tensor(softplus_upper_idx)
+        )
+
+        # Define clamping functions
+        self.clamp_lower_upper = lambda x: (
+            self.sigmoid_lower_lims
+            + (self.sigmoid_upper_lims - self.sigmoid_lower_lims)
+            * torch.sigmoid(sigmoid_sharpness * (x - sigmoid_center))
+        )
+        self.clamp_lower = lambda x: (
+            self.softplus_lower_lims
+            + torch.nn.functional.softplus(
+                x - softplus_center, beta=softplus_sharpness
+            )
+        )
+        self.clamp_upper = lambda x: (
+            self.softplus_upper_lims
+            - torch.nn.functional.softplus(
+                softplus_center - x, beta=softplus_sharpness
+            )
+        )
+
+        self.inverse_clamp_lower_upper = lambda x: (
+            sigmoid_center
+            + utils.inverse_sigmoid(
+                (x - self.sigmoid_lower_lims)
+                / (self.sigmoid_upper_lims - self.sigmoid_lower_lims)
+            )
+            / sigmoid_sharpness
+        )
+        self.inverse_clamp_lower = lambda x: (
+            utils.inverse_softplus(
+                x - self.softplus_lower_lims, beta=softplus_sharpness
+            )
+            + softplus_center
+        )
+        self.inverse_clamp_upper = lambda x: (
+            -utils.inverse_softplus(
+                self.softplus_upper_lims - x, beta=softplus_sharpness
+            )
+            + softplus_center
+        )
+
+    def get_clamped_new_state(self, state_delta, prev_state):
+        """
+        Clamp prediction to valid range supplied in config
+        Returns the clamped new state after adding delta to original state
+
+        Instead of the new state being computed as
+        $X_{t+1} = X_t + \\delta = X_t + model(\\{X_t,X_{t-1},...\\}, forcing)$
+        The clamped values will be
+        $f(f^{-1}(X_t) + model(\\{X_t, X_{t-1},... \\}, forcing))$
+        Which means the model will learn to output values in the range of the
+        inverse clamping function
+
+        state_delta: (B, num_grid_nodes, feature_dim)
+        prev_state: (B, num_grid_nodes, feature_dim)
+        """
+
+        # Assign new state, but overwrite clamped values of each type later
+        new_state = prev_state + state_delta
+
+        # Sigmoid/logistic clamps between ]a,b[
+        if self.clamp_lower_upper_idx.numel() > 0:
+            idx = self.clamp_lower_upper_idx
+
+            new_state[:, :, idx] = self.clamp_lower_upper(
+                self.inverse_clamp_lower_upper(prev_state[:, :, idx])
+                + state_delta[:, :, idx]
+            )
+
+        # Softplus clamps between ]a,infty[
+        if self.clamp_lower_idx.numel() > 0:
+            idx = self.clamp_lower_idx
+
+            new_state[:, :, idx] = self.clamp_lower(
+                self.inverse_clamp_lower(prev_state[:, :, idx])
+                + state_delta[:, :, idx]
+            )
+
+        # Softplus clamps between ]-infty,b[
+        if self.clamp_upper_idx.numel() > 0:
+            idx = self.clamp_upper_idx
+
+            new_state[:, :, idx] = self.clamp_upper(
+                self.inverse_clamp_upper(prev_state[:, :, idx])
+                + state_delta[:, :, idx]
+            )
+
+        return new_state
+
     def get_num_mesh(self):
         """
         Compute number of mesh nodes from loaded features,
@@ -173,5 +359,7 @@ def predict_step(self, prev_state, prev_prev_state, forcing):
         # Rescale with one-step difference statistics
         rescaled_delta_mean = pred_delta_mean * self.diff_std + self.diff_mean

-        # Residual connection for full state
-        return prev_state + rescaled_delta_mean, pred_std
+        # Clamp values to valid range (also add the delta to the previous state)
+        new_state = self.get_clamped_new_state(rescaled_delta_mean, prev_state)
+
+        return new_state, pred_std
diff --git a/neural_lam/utils.py b/neural_lam/utils.py
index ab213453..b1a277ea 100644
--- a/neural_lam/utils.py
+++ b/neural_lam/utils.py
@@ -307,3 +307,36 @@ def setup_training_logger(datastore, args, run_name):
         )

     return logger
+
+
+def inverse_softplus(x, beta=1, threshold=20):
+    """
+    Inverse of torch.nn.functional.softplus
+
+    For x*beta above threshold, returns linear function for numerical
+    stability.
+
+    Input is clamped to x > ln(1+1e-6)/beta which is approximately positive
+    values of x.
+    Note that this torch.clamp_min will make gradients 0, but this is not a
+    problem as values of x that are this close to 0 have gradients of 0 anyhow.
+    """
+    non_linear_part = (
+        torch.log(torch.clamp_min(torch.expm1(x * beta), 1e-6)) / beta
+    )
+    x = torch.where(x * beta <= threshold, non_linear_part, x)
+
+    return x
+
+
+def inverse_sigmoid(x):
+    """
+    Inverse of torch.sigmoid
+
+    Sigmoid output takes values in [0,1], this makes sure input is just within
+    this interval.
+    Note that this torch.clamp will make gradients 0, but this is not a problem
+    as values of x that are this close to 0 or 1 have gradients of 0 anyhow.
+    """
+    x_clamped = torch.clamp(x, min=1e-6, max=1 - 1e-6)
+    return torch.log(x_clamped / (1 - x_clamped))

diff --git a/tests/datastore_examples/mdp/danra_100m_winds/config.yaml b/tests/datastore_examples/mdp/danra_100m_winds/config.yaml
index 0bb5c5ec..8b3362e0 100644
--- a/tests/datastore_examples/mdp/danra_100m_winds/config.yaml
+++ b/tests/datastore_examples/mdp/danra_100m_winds/config.yaml
@@ -7,3 +7,12 @@ training:
     weights:
       u100m: 1.0
       v100m: 1.0
+      t2m: 1.0
+      r2m: 1.0
+  output_clamping:
+    lower:
+      t2m: 0.0
+      r2m: 0
+    upper:
+      r2m: 1.0
+      u100m: 100.0
diff --git a/tests/datastore_examples/mdp/danra_100m_winds/danra.datastore.yaml b/tests/datastore_examples/mdp/danra_100m_winds/danra.datastore.yaml
index 3f9bafac..876b1405 100644
--- a/tests/datastore_examples/mdp/danra_100m_winds/danra.datastore.yaml
+++ b/tests/datastore_examples/mdp/danra_100m_winds/danra.datastore.yaml
@@ -55,7 +55,7 @@ inputs:
         dims: [x, y]
     target_output_variable: state

-  danra_surface:
+  danra_surface_forcing:
     path: https://object-store.os-api.cci1.ecmwf.int/mllam-testdata/danra_cropped/v0.2.0/single_levels.zarr
     dims: [time, x, y]
     variables:
@@ -73,6 +73,24 @@ inputs:
         name_format: "{var_name}"
     target_output_variable: forcing

+  danra_surface:
+    path: https://object-store.os-api.cci1.ecmwf.int/mllam-testdata/danra_cropped/v0.2.0/single_levels.zarr
+    dims: [time, x, y]
+    variables:
+      - r2m
+      - t2m
+    dim_mapping:
+      time:
+        method: rename
+        dim: time
+      grid_index:
+        method: stack
+        dims: [x, y]
+      state_feature:
+        method: stack_variables_by_var_name
+        name_format: "{var_name}"
+    target_output_variable: state
+
   danra_static:
     path: https://object-store.os-api.cci1.ecmwf.int/mllam-testdata/danra_cropped/v0.2.0/single_levels.zarr
     dims: [x, y]
diff --git a/tests/test_clamping.py b/tests/test_clamping.py
new file mode 100644
index 00000000..f3f9365d
--- /dev/null
+++ b/tests/test_clamping.py
@@ -0,0 +1,283 @@
+# Standard library
+from pathlib import Path
+
+# Third-party
+import torch
+
+# First-party
+from neural_lam import config as nlconfig
+from neural_lam.create_graph import create_graph_from_datastore
+from neural_lam.datastore.mdp import MDPDatastore
+from neural_lam.models.graph_lam import GraphLAM
+from tests.conftest import init_datastore_example
+
+
+def test_clamping():
+    datastore = init_datastore_example(MDPDatastore.SHORT_NAME)
+
+    graph_name = "1level"
+
+    graph_dir_path = Path(datastore.root_path) / "graph" / graph_name
+
+    if not graph_dir_path.exists():
+        create_graph_from_datastore(
+            datastore=datastore,
+            output_root_path=str(graph_dir_path),
+            n_max_levels=1,
+        )
+
+    class ModelArgs:
+        output_std = False
+        loss = "mse"
+        restore_opt = False
+        n_example_pred = 1
+        graph = graph_name
+        hidden_dim = 4
+        hidden_layers = 1
+        processor_layers = 2
+        mesh_aggr = "sum"
+        lr = 1.0e-3
+        val_steps_to_log = [1, 3]
+        metrics_watch = []
+        num_past_forcing_steps = 1
+        num_future_forcing_steps = 1
+
+    model_args = ModelArgs()
+
+    config = nlconfig.NeuralLAMConfig(
+        datastore=nlconfig.DatastoreSelection(
+            kind=datastore.SHORT_NAME, config_path=datastore.root_path
+        ),
+        training=nlconfig.TrainingConfig(
+            output_clamping=nlconfig.OutputClamping(
+                lower={"t2m": 0.0, "r2m": 0.0},
+                upper={"r2m": 1.0, "u100m": 100.0},
+            )
+        ),
+    )
+
+    model = GraphLAM(
+        args=model_args,
+        datastore=datastore,
+        config=config,
+    )
+
+    features = datastore.get_vars_names(category="state")
+    original_state = torch.zeros(1, 1, len(features))
+    zero_delta = original_state.clone()
+
+    # Get a state well within the bounds
+    original_state[:, :, model.clamp_lower_upper_idx] = (
+        model.sigmoid_lower_lims + model.sigmoid_upper_lims
+    ) / 2
+    original_state[:, :, model.clamp_lower_idx] = model.softplus_lower_lims + 10
+    original_state[:, :, model.clamp_upper_idx] = model.softplus_upper_lims - 10
+
+    # Get a delta that tries to push the state out of bounds
+    delta = torch.ones_like(zero_delta)
+    delta[:, :, model.clamp_lower_upper_idx] = (
+        model.sigmoid_upper_lims - model.sigmoid_lower_lims
+    ) / 3
+    delta[:, :, model.clamp_lower_idx] = -5
+    delta[:, :, model.clamp_upper_idx] = 5
+
+    # Check that a delta of 0 gives unchanged state
+    zero_prediction = model.get_clamped_new_state(zero_delta, original_state)
+    assert (abs(original_state - zero_prediction) < 1e-6).all().item()
+
+    # Make predictions towards bounds for each feature
+    prediction = zero_prediction.clone()
+    n_loops = 100
+    for i in range(n_loops):
+        prediction = model.get_clamped_new_state(delta, prediction)
+
+    # check that unclamped states are as expected
+    # delta is 1, so they should be 1*n_loops
+    assert (
+        (
+            abs(
+                prediction[
+                    :,
+                    :,
+                    list(
+                        set(range(len(features)))
+                        - set(model.clamp_lower_upper_idx.tolist())
+                        - set(model.clamp_lower_idx.tolist())
+                        - set(model.clamp_upper_idx.tolist())
+                    ),
+                ]
+                - n_loops
+            )
+            < 1e-6
+        )
+        .all()
+        .item()
+    )
+
+    # Check that clamped states are within bounds
+    # they should not be at the bounds but allow it due to numerical precision
+    assert (
+        (
+            model.sigmoid_lower_lims
+            <= prediction[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .all()
+        .item()
+    )
+    assert (
+        (model.softplus_lower_lims <= prediction[:, :, model.clamp_lower_idx])
+        .all()
+        .item()
+    )
+    assert (
+        (prediction[:, :, model.clamp_upper_idx] <= model.softplus_upper_lims)
+        .all()
+        .item()
+    )
+
+    # Check that prediction is within bounds in original non-normalized space
+    unscaled_prediction = prediction * model.state_std + model.state_mean
+    features_idx = {f: i for i, f in enumerate(features)}
+    lower_lims = {
+        features_idx[f]: lim
+        for f, lim in config.training.output_clamping.lower.items()
+    }
+    upper_lims = {
+        features_idx[f]: lim
+        for f, lim in config.training.output_clamping.upper.items()
+    }
+    assert (
+        (
+            torch.tensor(list(lower_lims.values()))
+            <= unscaled_prediction[:, :, list(lower_lims.keys())]
+        )
+        .all()
+        .item()
+    )
+    assert (
+        (
+            unscaled_prediction[:, :, list(upper_lims.keys())]
+            <= torch.tensor(list(upper_lims.values()))
+        )
+        .all()
+        .item()
+    )
+
+    # Check that a prediction from a state starting outside the bounds is also
+    # pushed within bounds. 3 delta should be enough to give an initial state
+    # out of bounds so 5 is well outside
+    invalid_state = original_state + 5 * delta
+    assert (
+        not (
+            model.sigmoid_lower_lims
+            <= invalid_state[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .any()
+        .item()
+    )
+    assert (
+        not (
+            model.softplus_lower_lims
+            <= invalid_state[:, :, model.clamp_lower_idx]
+        )
+        .any()
+        .item()
+    )
+    assert (
+        not (
+            invalid_state[:, :, model.clamp_upper_idx]
+            <= model.softplus_upper_lims
+        )
+        .any()
+        .item()
+    )
+    invalid_prediction = model.get_clamped_new_state(zero_delta, invalid_state)
+    assert (
+        (
+            model.sigmoid_lower_lims
+            <= invalid_prediction[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .all()
+        .item()
+    )
+    assert (
+        (
+            model.softplus_lower_lims
+            <= invalid_prediction[:, :, model.clamp_lower_idx]
+        )
+        .all()
+        .item()
+    )
+    assert (
+        (
+            invalid_prediction[:, :, model.clamp_upper_idx]
+            <= model.softplus_upper_lims
+        )
+        .all()
+        .item()
+    )
+
+    # Above tests only check the upper sigmoid limit.
+    # Repeat to check lower sigmoid limit
+
+    # Make predictions towards bounds for each feature
+    prediction = zero_prediction.clone()
+    n_loops = 100
+    for i in range(n_loops):
+        prediction = model.get_clamped_new_state(-delta, prediction)
+
+    # Check that clamped states are within bounds
+    assert (
+        (
+            model.sigmoid_lower_lims
+            <= prediction[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .all()
+        .item()
+    )
+
+    # Check that prediction is within bounds in original non-normalized space
+    assert (
+        (
+            torch.tensor(list(lower_lims.values()))
+            <= unscaled_prediction[:, :, list(lower_lims.keys())]
+        )
+        .all()
+        .item()
+    )
+    assert (
+        (
+            unscaled_prediction[:, :, list(upper_lims.keys())]
+            <= torch.tensor(list(upper_lims.values()))
+        )
+        .all()
+        .item()
+    )
+
+    # Check that a prediction from a state starting outside the bounds is also
+    # pushed within bounds. 3 delta should be enough to give an initial state
+    # out of bounds so 5 is well outside
+    invalid_state = original_state - 5 * delta
+    assert (
+        not (
+            model.sigmoid_lower_lims
+            <= invalid_state[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .any()
+        .item()
+    )
+    invalid_prediction = model.get_clamped_new_state(zero_delta, invalid_state)
+    assert (
+        (
+            model.sigmoid_lower_lims
+            <= invalid_prediction[:, :, model.clamp_lower_upper_idx]
+            <= model.sigmoid_upper_lims
+        )
+        .all()
+        .item()
+    )

Feature Request: Add Functionality to Apply Constraints to Predictions
I am proposing the addition of a new method to our model class, designed to apply constraints to predictions to ensure that the values fall within specified bounds. This functionality would be useful for maintaining the integrity of our model's predictions in scenarios where certain variables have predefined limits.

Proposed Method:

```
def apply_constraints(self, prediction):
    """
    Apply constraints to prediction to ensure values are within the
    specified bounds
    """
    for param, (min_val, max_val) in constants.PARAM_CONSTRAINTS.items():
        indices = self.variable_indices[param]
        for index in indices:
            # Apply clamping to ensure values are within the specified
            # bounds
            prediction[:, :, :, index] = torch.clamp(
                prediction[:, :, :, index],
                min=min_val,
                max=max_val if max_val is not None else float("inf"),
            )
    return prediction
```
Rationale:

Data Integrity: Ensuring that predictions adhere to real-world constraints is essential for the reliability of our model's outputs. This method would allow us to enforce these constraints directly on the prediction tensor.
Flexibility: By dynamically applying constraints based on the variable indices, we can maintain a high degree of flexibility in how we handle different variables with varying constraints.

The method could be added to the ARModel class, which is our primary model class.
The constants.PARAM_CONSTRAINTS dictionary, which maps variable names to their minimum and maximum values, should be used to determine the constraints for each variable.

```
PARAM_CONSTRAINTS = {
    "RELHUM": (0, 100),
    "CLCT": (0, 100),
    "TOT_PREC": (0, None),
}
```

This feature is closely related to #18

I would want to do this by constraining the model output itself, also for what is used during training (a clamp does not give gradient). Specifically, variables with constraints should be handled by rescaling a sigmoid output or a softplus (for > 0). As this applies to all models, it would be nice with it included on a high level, so it does not have to be handled separately in each model.

One option would be to have a function `constrain_outputs` in the `ARModel` class that applies these activation functions. Then everything that inherits from this can just apply this to the state after making a prediction.
@joeloskarsson @sadamov
I agree with Joel that simply clipping values would impact model training/performance, and that using activation functions is the solution. In the development meeting it sounded like no one was really working on this, so if you want I can make a PR to implement this?
I agree with both of you, and that would be highly appreciated!
That would be wonderful @SimonKamuk! I had a though about this: As the model currently predicts residuals, it's not obvious how to do this in that case. Since the predicted variables are $X^{t} + \text{model}(\dots)$ we can't just throw e.g. a sigmoid on top of that One solution to this is to just stop predicting residuals and predicting full states. But that's quite a big change and maybe something we would want to experiment with before. Maybe you can come up with some other solution.

Another consideration here is that the variable constraints apply in the original scale of the data, not for the standardized data. So e.g. a constraint that a variable should be in $[0,1]$ does not mean that the corresponding model output should be in that range, but some other range $[a,b]$.

Happy to discuss this more!


diff --git a/cuda_core/cuda/core/experimental/_module.py b/cuda_core/cuda/core/experimental/_module.py
index 36178f5d..0274b300 100644
--- a/cuda_core/cuda/core/experimental/_module.py
+++ b/cuda_core/cuda/core/experimental/_module.py
@@ -3,6 +3,8 @@
 # SPDX-License-Identifier: LicenseRef-NVIDIA-SOFTWARE-LICENSE

+from warnings import warn
+
 from cuda.core.experimental._utils import driver, get_binding_version, handle_return, precondition

 _backend = {
@@ -10,6 +12,7 @@
         "file": driver.cuModuleLoad,
         "data": driver.cuModuleLoadDataEx,
         "kernel": driver.cuModuleGetFunction,
+        "attribute": driver.cuFuncGetAttribute,
     },
 }

@@ -34,6 +37,7 @@ def _lazy_init():
             "file": driver.cuLibraryLoadFromFile,
             "data": driver.cuLibraryLoadData,
             "kernel": driver.cuLibraryGetKernel,
+            "attribute": driver.cuKernelGetAttribute,
         }
         _kernel_ctypes = (driver.CUfunction, driver.CUkernel)
     else:
@@ -42,6 +46,136 @@ def _lazy_init():
     _inited = True

+class KernelAttributes:
+    def __init__(self):
+        raise RuntimeError("KernelAttributes should not be instantiated directly")
+
+    slots = ("_handle", "_cache", "_backend_version", "_loader")
+
+    def _init(handle):
+        self = KernelAttributes.__new__(KernelAttributes)
+        self._handle = handle
+        self._cache = {}
+
+        self._backend_version = "new" if (_py_major_ver >= 12 and _driver_ver >= 12000) else "old"
+        self._loader = _backend[self._backend_version]
+        return self
+
+    def _get_cached_attribute(self, device_id: int, attribute: driver.CUfunction_attribute) -> int:
+        """Helper function to get a cached attribute or fetch and cache it if not present."""
+        if device_id in self._cache and attribute in self._cache[device_id]:
+            return self._cache[device_id][attribute]
+        if self._backend_version == "new":
+            result = handle_return(self._loader["attribute"](attribute, self._handle, device_id))
+        else:  # "old" backend
+            warn(
+                "Device ID argument is ignored when getting attribute from kernel when cuda version < 12. ",
+                RuntimeWarning,
+                stacklevel=2,
+            )
+            result = handle_return(self._loader["attribute"](attribute, self._handle))
+        if device_id not in self._cache:
+            self._cache[device_id] = {}
+        self._cache[device_id][attribute] = result
+        return result
+
+    def max_threads_per_block(self, device_id: int = None) -> int:
+        """int : The maximum number of threads per block.
+        This attribute is read-only."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_MAX_THREADS_PER_BLOCK
+        )
+
+    def shared_size_bytes(self, device_id: int = None) -> int:
+        """int : The size in bytes of statically-allocated shared memory required by this function.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_SHARED_SIZE_BYTES)
+
+    def const_size_bytes(self, device_id: int = None) -> int:
+        """int : The size in bytes of user-allocated constant memory required by this function.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_CONST_SIZE_BYTES)
+
+    def local_size_bytes(self, device_id: int = None) -> int:
+        """int : The size in bytes of local memory used by each thread of this function.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_LOCAL_SIZE_BYTES)
+
+    def num_regs(self, device_id: int = None) -> int:
+        """int : The number of registers used by each thread of this function.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_NUM_REGS)
+
+    def ptx_version(self, device_id: int = None) -> int:
+        """int : The PTX virtual architecture version for which the function was compiled.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_PTX_VERSION)
+
+    def binary_version(self, device_id: int = None) -> int:
+        """int : The binary architecture version for which the function was compiled.
+        This attribute is read-only."""
+        return self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_BINARY_VERSION)
+
+    def cache_mode_ca(self, device_id: int = None) -> bool:
+        """bool : Whether the function has been compiled with user specified option "-Xptxas --dlcm=ca" set.
+        This attribute is read-only."""
+        return bool(self._get_cached_attribute(device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_CACHE_MODE_CA))
+
+    def max_dynamic_shared_size_bytes(self, device_id: int = None) -> int:
+        """int : The maximum size in bytes of dynamically-allocated shared memory that can be used
+        by this function."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_MAX_DYNAMIC_SHARED_SIZE_BYTES
+        )
+
+    def preferred_shared_memory_carveout(self, device_id: int = None) -> int:
+        """int : The shared memory carveout preference, in percent of the total shared memory."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_PREFERRED_SHARED_MEMORY_CARVEOUT
+        )
+
+    def cluster_size_must_be_set(self, device_id: int = None) -> bool:
+        """bool : The kernel must launch with a valid cluster size specified.
+        This attribute is read-only."""
+        return bool(
+            self._get_cached_attribute(
+                device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_CLUSTER_SIZE_MUST_BE_SET
+            )
+        )
+
+    def required_cluster_width(self, device_id: int = None) -> int:
+        """int : The required cluster width in blocks."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_REQUIRED_CLUSTER_WIDTH
+        )
+
+    def required_cluster_height(self, device_id: int = None) -> int:
+        """int : The required cluster height in blocks."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_REQUIRED_CLUSTER_HEIGHT
+        )
+
+    def required_cluster_depth(self, device_id: int = None) -> int:
+        """int : The required cluster depth in blocks."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_REQUIRED_CLUSTER_DEPTH
+        )
+
+    def non_portable_cluster_size_allowed(self, device_id: int = None) -> bool:
+        """bool : Whether the function can be launched with non-portable cluster size."""
+        return bool(
+            self._get_cached_attribute(
+                device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_NON_PORTABLE_CLUSTER_SIZE_ALLOWED
+            )
+        )
+
+    def cluster_scheduling_policy_preference(self, device_id: int = None) -> int:
+        """int : The block scheduling policy of a function."""
+        return self._get_cached_attribute(
+            device_id, driver.CUfunction_attribute.CU_FUNC_ATTRIBUTE_CLUSTER_SCHEDULING_POLICY_PREFERENCE
+        )
+
+
 class Kernel:
     """Represent a compiled kernel that had been loaded onto the device.

@@ -53,13 +187,10 @@ class Kernel:

     """

-    __slots__ = (
-        "_handle",
-        "_module",
-    )
+    __slots__ = ("_handle", "_module", "_attributes")

     def __init__(self):
-        raise NotImplementedError("directly constructing a Kernel instance is not supported")
+        raise RuntimeError("directly constructing a Kernel instance is not supported")

     @staticmethod
     def _from_obj(obj, mod):
@@ -68,8 +199,16 @@ def _from_obj(obj, mod):
         ker = Kernel.__new__(Kernel)
         ker._handle = obj
         ker._module = mod
+        ker._attributes = None
         return ker

+    @property
+    def attributes(self):
+        """Get the read-only attributes of this kernel."""
+        if self._attributes is None:
+            self._attributes = KernelAttributes._init(self._handle)
+        return self._attributes
+
     # TODO: implement from_handle()

diff --git a/cuda_core/docs/source/release/0.2.0-notes.rst b/cuda_core/docs/source/release/0.2.0-notes.rst
index 0a34f825..73db65e3 100644
--- a/cuda_core/docs/source/release/0.2.0-notes.rst
+++ b/cuda_core/docs/source/release/0.2.0-notes.rst
@@ -9,6 +9,7 @@ Highlights
 ----------

 - Add :class:`~ProgramOptions` to facilitate the passing of runtime compile options to :obj:`~Program`.
+- Add kernel attributes to :class:`~_module.Kernel`

 Limitations
 -----------

diff --git a/cuda_core/tests/test_module.py b/cuda_core/tests/test_module.py
index 7db017f1..9f126fa1 100644
--- a/cuda_core/tests/test_module.py
+++ b/cuda_core/tests/test_module.py
@@ -10,14 +10,75 @@
 import pytest
 from conftest import can_load_generated_ptx

-from cuda.core.experimental import Program, ProgramOptions
+from cuda.core.experimental import Program, ProgramOptions, system
+
+
+@pytest.fixture(scope="function")
+def get_saxpy_kernel(init_cuda):
+    code = """
+    template<typename T>
+    __global__ void saxpy(const T a,
+                        const T* x,
+                        const T* y,
+                        T* out,
+                        size_t N) {
+        const unsigned int tid = threadIdx.x + blockIdx.x * blockDim.x;
+        for (size_t i=tid; i<N; i+=gridDim.x*blockDim.x) {
+            out[tid] = a * x[tid] + y[tid];
+        }
+    }
+    """
+
+    # prepare program
+    prog = Program(code, code_type="c++")
+    mod = prog.compile(
+        "cubin",
+        name_expressions=("saxpy<float>", "saxpy<double>"),
+    )
+
+    # run in single precision
+    return mod.get_kernel("saxpy<float>")

 @pytest.mark.xfail(not can_load_generated_ptx(), reason="PTX version too new")
-def test_get_kernel():
+def test_get_kernel(init_cuda):
     kernel = """extern "C" __global__ void ABC() { }"""
     object_code = Program(kernel, "c++", options=ProgramOptions(relocatable_device_code=True)).compile("ptx")
     assert object_code._handle is None
     kernel = object_code.get_kernel("ABC")
     assert object_code._handle is not None
     assert kernel._handle is not None
+
+
+@pytest.mark.parametrize(
+    "attr, expected_type",
+    [
+        ("max_threads_per_block", int),
+        ("shared_size_bytes", int),
+        ("const_size_bytes", int),
+        ("local_size_bytes", int),
+        ("num_regs", int),
+        ("ptx_version", int),
+        ("binary_version", int),
+        ("cache_mode_ca", bool),
+        ("cluster_size_must_be_set", bool),
+        ("max_dynamic_shared_size_bytes", int),
+        ("preferred_shared_memory_carveout", int),
+        ("required_cluster_width", int),
+        ("required_cluster_height", int),
+        ("required_cluster_depth", int),
+        ("non_portable_cluster_size_allowed", bool),
+        ("cluster_scheduling_policy_preference", int),
+    ],
+)
+def test_read_only_kernel_attributes(get_saxpy_kernel, attr, expected_type):
+    kernel = get_saxpy_kernel
+    method = getattr(kernel.attributes, attr)
+    # get the value without providing a device ordinal
+    value = method()
+    assert value is not None
+
+    # get the value for each device on the system
+    for device in system.devices:
+        value = method(device.device_id)
+    assert isinstance(value, expected_type), f"Expected {attr} to be of type {expected_type}, but got {type(value)}"

Add `Kernel` attribute getter/setter
Need a nice way to get/set `CUfunction_attribute`. Make them maybe nice Python properties, instead of using the C enumerator names literally.

To check:
- For attributes that can be set either through launch config (ex: #204) or kernel attribute, what's the right way to go?

User request (to CuPy) for setting `cudaFuncAttributeNonPortableClusterSizeAllowed`: https://github.com/cupy/cupy/issues/8851
I think we should provide both avenues for specifying clusters or other configs which can be done via the kernel attributes of launch configs. There are 3 reasons for this:

1) Runtime vs compile time means they're not exactly the same

2) A user coming from c++ will expect both options, and a new user doesn't need to understand both options at first in order to get started. The possibility of being confused by two options is outweighed by the potential confusion of learning about both options via existing cuda c++ documentation or looking at the source code, but not being able to use them.

3) For us to provide both requires almost no effort, as we are leveraging the existing backend. Isolating things like clusters to one of either lunch configs, or kernel attributes, would require us to hack away a subset of perfectly valid attributes/configs on either launches or kernels.


diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index 2e7dcba82e846..243c63ab0c7e2 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -646,6 +646,7 @@ Kernels:
    :template: class.rst

    impute.SimpleImputer
+   impute.MICEImputer

 .. _kernel_approximation_ref:

diff --git a/doc/modules/impute.rst b/doc/modules/impute.rst
index e806cc2fd5b4a..c4e8a3395025c 100644
--- a/doc/modules/impute.rst
+++ b/doc/modules/impute.rst
@@ -15,6 +15,10 @@ values. However, this comes at the price of losing data which may be valuable
 (even though incomplete). A better strategy is to impute the missing values,
 i.e., to infer them from the known part of the data.

+
+Univariate feature imputation
+=============================
+
 The :class:`SimpleImputer` class provides basic strategies for imputing missing
 values, either using the mean, the median or the most frequent value of
 the row or column in which the missing values are located. This class
@@ -52,5 +56,34 @@ Note that, here, missing values are encoded by 0 and are thus implicitly stored
 in the matrix. This format is thus suitable when there are many more missing
 values than observed values.

-:class:`SimpleImputer` can be used in a Pipeline as a way to build a composite
-estimator that supports imputation. See :ref:`sphx_glr_auto_examples_plot_missing_values.py`.
+
+Multivariate feature imputation
+===============================
+
+A more sophisticated approach is to use the :class:`MICEImputer` class, which
+implements the Multivariate Imputation by Chained Equations technique. MICE
+models each feature with missing values as a function of other features, and
+uses that estimate for imputation. It does so in a round-robin fashion: at
+each step, a feature column is designated as output `y` and the other feature
+columns are treated as inputs `X`. A regressor is fit on `(X, y)` for known `y`.
+Then, the regressor is used to predict the unknown values of `y`. This is
+repeated for each feature, and then is done for a number of imputation rounds.
+Here is an example snippet::
+
+    >>> import numpy as np
+    >>> from sklearn.impute import MICEImputer
+    >>> imp = MICEImputer(n_imputations=10, random_state=0)
+    >>> imp.fit([[1, 2], [np.nan, 3], [7, np.nan]])
+    MICEImputer(imputation_order='ascending', initial_strategy='mean',
+          max_value=None, min_value=None, missing_values='NaN', n_burn_in=10,
+          n_imputations=10, n_nearest_features=None, predictor=None,
+          random_state=0, verbose=False)
+    >>> X_test = [[np.nan, 2], [6, np.nan], [np.nan, 6]]
+    >>> print(np.round(imp.transform(X_test)))
+    [[ 1.  2.]
+     [ 6.  4.]
+     [13.  6.]]
+
+Both :class:`SimpleImputer` and :class:`MICEImputer` can be used in a Pipeline
+as a way to build a composite estimator that supports imputation.
+See :ref:`sphx_glr_auto_examples_plot_missing_values.py`.
diff --git a/examples/plot_missing_values.py b/examples/plot_missing_values.py
index 9cc07b75d2542..e32c19fae0846 100644
--- a/examples/plot_missing_values.py
+++ b/examples/plot_missing_values.py
@@ -1,72 +1,127 @@
 """
-======================================================
+====================================================
 Imputing missing values before building an estimator
-======================================================
-
-This example shows that imputing the missing values can give better
-results than discarding the samples containing any missing value.
-Imputing does not always improve the predictions, so please check via
-cross-validation.  Sometimes dropping rows or using marker values is
-more effective.
+====================================================

 Missing values can be replaced by the mean, the median or the most frequent
-value using the ``strategy`` hyper-parameter.
+value using the basic ``SimpleImputer``.
 The median is a more robust estimator for data with high magnitude variables
 which could dominate results (otherwise known as a 'long tail').

-Script output::
-
-  Score with the entire dataset = 0.56
-  Score without the samples containing missing values = 0.48
-  Score after imputation of the missing values = 0.55
-
-In this case, imputing helps the classifier get close to the original score.
-
+Another option is the MICE imputer. This uses round-robin linear regression,
+treating every variable as an output in turn. The version implemented assumes
+Gaussian (output) variables. If your features are obviously non-Normal,
+consider transforming them to look more Normal so as to improve performance.
 """
+
 import numpy as np
+import matplotlib.pyplot as plt

+from sklearn.datasets import load_diabetes
 from sklearn.datasets import load_boston
 from sklearn.ensemble import RandomForestRegressor
 from sklearn.pipeline import Pipeline
-from sklearn.impute import SimpleImputer
+from sklearn.impute import SimpleImputer, MICEImputer
 from sklearn.model_selection import cross_val_score

 rng = np.random.RandomState(0)

-dataset = load_boston()
-X_full, y_full = dataset.data, dataset.target
-n_samples = X_full.shape[0]
-n_features = X_full.shape[1]
-
-# Estimate the score on the entire dataset, with no missing values
-estimator = RandomForestRegressor(random_state=0, n_estimators=100)
-score = cross_val_score(estimator, X_full, y_full).mean()
-print("Score with the entire dataset = %.2f" % score)
-
-# Add missing values in 75% of the lines
-missing_rate = 0.75
-n_missing_samples = int(np.floor(n_samples * missing_rate))
-missing_samples = np.hstack((np.zeros(n_samples - n_missing_samples,
-                                      dtype=np.bool),
-                             np.ones(n_missing_samples,
-                                     dtype=np.bool)))
-rng.shuffle(missing_samples)
-missing_features = rng.randint(0, n_features, n_missing_samples)
-
-# Estimate the score without the lines containing missing values
-X_filtered = X_full[~missing_samples, :]
-y_filtered = y_full[~missing_samples]
-estimator = RandomForestRegressor(random_state=0, n_estimators=100)
-score = cross_val_score(estimator, X_filtered, y_filtered).mean()
-print("Score without the samples containing missing values = %.2f" % score)
-
-# Estimate the score after imputation of the missing values
-X_missing = X_full.copy()
-X_missing[np.where(missing_samples)[0], missing_features] = 0
-y_missing = y_full.copy()
-estimator = Pipeline([("imputer", SimpleImputer(missing_values=0,
-                                                strategy="mean")),
-                      ("forest", RandomForestRegressor(random_state=0,
-                                                       n_estimators=100))])
-score = cross_val_score(estimator, X_missing, y_missing).mean()
-print("Score after imputation of the missing values = %.2f" % score)
+
+def get_results(dataset):
+    X_full, y_full = dataset.data, dataset.target
+    n_samples = X_full.shape[0]
+    n_features = X_full.shape[1]
+
+    # Estimate the score on the entire dataset, with no missing values
+    estimator = RandomForestRegressor(random_state=0, n_estimators=100)
+    full_scores = cross_val_score(estimator, X_full, y_full,
+                                  scoring='neg_mean_squared_error')
+
+    # Add missing values in 75% of the lines
+    missing_rate = 0.75
+    n_missing_samples = int(np.floor(n_samples * missing_rate))
+    missing_samples = np.hstack((np.zeros(n_samples - n_missing_samples,
+                                          dtype=np.bool),
+                                 np.ones(n_missing_samples,
+                                         dtype=np.bool)))
+    rng.shuffle(missing_samples)
+    missing_features = rng.randint(0, n_features, n_missing_samples)
+
+    # Estimate the score after replacing missing values by 0
+    X_missing = X_full.copy()
+    X_missing[np.where(missing_samples)[0], missing_features] = 0
+    y_missing = y_full.copy()
+    estimator = RandomForestRegressor(random_state=0, n_estimators=100)
+    zero_impute_scores = cross_val_score(estimator, X_missing, y_missing,
+                                         scoring='neg_mean_squared_error')
+
+    # Estimate the score after imputation (mean strategy) of the missing values
+    X_missing = X_full.copy()
+    X_missing[np.where(missing_samples)[0], missing_features] = 0
+    y_missing = y_full.copy()
+    estimator = Pipeline([("imputer", SimpleImputer(missing_values=0,
+                                                    strategy="mean")),
+                          ("forest", RandomForestRegressor(random_state=0,
+                                                           n_estimators=100))])
+    mean_impute_scores = cross_val_score(estimator, X_missing, y_missing,
+                                         scoring='neg_mean_squared_error')
+
+    # Estimate the score after imputation (MICE strategy) of the missing values
+    estimator = Pipeline([("imputer", MICEImputer(missing_values=0,
+                                                  random_state=0)),
+                          ("forest", RandomForestRegressor(random_state=0,
+                                                           n_estimators=100))])
+    mice_impute_scores = cross_val_score(estimator, X_missing, y_missing,
+                                         scoring='neg_mean_squared_error')
+
+    return ((full_scores.mean(), full_scores.std()),
+            (zero_impute_scores.mean(), zero_impute_scores.std()),
+            (mean_impute_scores.mean(), mean_impute_scores.std()),
+            (mice_impute_scores.mean(), mice_impute_scores.std()))
+
+
+results_diabetes = np.array(get_results(load_diabetes()))
+mses_diabetes = results_diabetes[:, 0] * -1
+stds_diabetes = results_diabetes[:, 1]
+
+results_boston = np.array(get_results(load_boston()))
+mses_boston = results_boston[:, 0] * -1
+stds_boston = results_boston[:, 1]
+
+n_bars = len(mses_diabetes)
+xval = np.arange(n_bars)
+
+x_labels = ['Full data',
+            'Zero imputation',
+            'Mean Imputation',
+            'MICE Imputation']
+colors = ['r', 'g', 'b', 'orange']
+
+# plot diabetes results
+plt.figure(figsize=(12, 6))
+ax1 = plt.subplot(121)
+for j in xval:
+    ax1.barh(j, mses_diabetes[j], xerr=stds_diabetes[j],
+             color=colors[j], alpha=0.6, align='center')
+
+ax1.set_title('Feature Selection Techniques with Diabetes Data')
+ax1.set_xlim(left=np.min(mses_diabetes) * 0.9,
+             right=np.max(mses_diabetes) * 1.1)
+ax1.set_yticks(xval)
+ax1.set_xlabel('MSE')
+ax1.invert_yaxis()
+ax1.set_yticklabels(x_labels)
+
+# plot boston results
+ax2 = plt.subplot(122)
+for j in xval:
+    ax2.barh(j, mses_boston[j], xerr=stds_boston[j],
+             color=colors[j], alpha=0.6, align='center')
+
+ax2.set_title('Feature Selection Techniques with Boston Data')
+ax2.set_yticks(xval)
+ax2.set_xlabel('MSE')
+ax2.invert_yaxis()
+ax2.set_yticklabels([''] * n_bars)
+
+plt.show()
diff --git a/sklearn/impute.py b/sklearn/impute.py
index 2420e02560e42..f395c634ccced 100644
--- a/sklearn/impute.py
+++ b/sklearn/impute.py
@@ -1,16 +1,23 @@
 """Transformers for missing value imputation"""
 # Authors: Nicolas Tresegnie <nicolas.tresegnie@gmail.com>
+#          Sergey Feldman <sergeyfeldman@gmail.com>
 # License: BSD 3 clause

+from __future__ import division
+
 import warnings
+from time import time

 import numpy as np
 import numpy.ma as ma
 from scipy import sparse
 from scipy import stats
+from collections import namedtuple

 from .base import BaseEstimator, TransformerMixin
-from .utils import check_array
+from .base import clone
+from .preprocessing import normalize
+from .utils import check_array, check_random_state, safe_indexing
 from .utils.sparsefuncs import _get_median
 from .utils.validation import check_is_fitted
 from .utils.validation import FLOAT_DTYPES
@@ -20,8 +27,13 @@
 zip = six.moves.zip
 map = six.moves.map

+MICETriplet = namedtuple('MICETriplet', ['feat_idx',
+                                         'neighbor_feat_idx',
+                                         'predictor'])
+
 __all__ = [
     'SimpleImputer',
+    'MICEImputer',
 ]

@@ -321,3 +333,541 @@ def transform(self, X):
             X[coordinates] = values

         return X
+
+
+class MICEImputer(BaseEstimator, TransformerMixin):
+    """MICE transformer to impute missing values.
+
+    Basic implementation of MICE (Multivariate Imputations by Chained
+    Equations) package from R. This version assumes all of the features are
+    Gaussian.
+
+    Parameters
+    ----------
+    missing_values : int or "NaN", optional (default="NaN")
+        The placeholder for the missing values. All occurrences of
+        ``missing_values`` will be imputed. For missing values encoded as
+        np.nan, use the string value "NaN".
+
+    imputation_order : str, optional (default="ascending")
+        The order in which the features will be imputed. Possible values:
+
+        "ascending"
+            From features with fewest missing values to most.
+        "descending"
+            From features with most missing values to fewest.
+        "roman"
+            Left to right.
+        "arabic"
+            Right to left.
+        "random"
+            A random order for each round.
+
+    n_imputations : int, optional (default=100)
+        Number of MICE rounds to perform, the results of which will be
+        used in the final average.
+
+    n_burn_in : int, optional (default=10)
+        Number of initial MICE rounds to perform the results of which
+        will not be returned.
+
+    predictor : estimator object, default=BayesianRidge()
+        The predictor to use at each step of the round-robin imputation.
+        It must support ``return_std`` in its ``predict`` method.
+
+    n_nearest_features : int, optional (default=None)
+        Number of other features to use to estimate the missing values of
+        the each feature column. Nearness between features is measured using
+        the absolute correlation coefficient between each feature pair (after
+        initial imputation). Can provide significant speed-up when the number
+        of features is huge. If ``None``, all features will be used.
+
+    initial_strategy : str, optional (default="mean")
+        Which strategy to use to initialize the missing values. Same as the
+        ``strategy`` parameter in :class:`sklearn.preprocessing.Imputer`
+        Valid values: {"mean", "median", or "most_frequent"}.
+
+    min_value : float, optional (default=None)
+        Minimum possible imputed value. Default of ``None`` will set minimum
+        to negative infinity.
+
+    max_value : float, optional (default=None)
+        Maximum possible imputed value. Default of ``None`` will set maximum
+        to positive infinity.
+
+    verbose : int, optional (default=0)
+        Verbosity flag, controls the debug messages that are issued
+        as functions are evaluated. The higher, the more verbose. Can be 0, 1,
+        or 2.
+
+    random_state : int, RandomState instance or None, optional (default=None)
+        The seed of the pseudo random number generator to use when shuffling
+        the data.  If int, random_state is the seed used by the random number
+        generator; If RandomState instance, random_state is the random number
+        generator; If None, the random number generator is the RandomState
+        instance used by ``np.random``.
+
+    Attributes
+    ----------
+    initial_imputer_ : object of class :class:`sklearn.preprocessing.Imputer`'
+        The imputer used to initialize the missing values.
+
+    imputation_sequence_ : list of tuples
+        Each tuple has ``(feat_idx, neighbor_feat_idx, predictor)``, where
+        ``feat_idx`` is the current feature to be imputed,
+        ``neighbor_feat_idx`` is the array of other features used to impute the
+        current feature, and ``predictor`` is the trained predictor used for
+        the imputation.
+
+    Notes
+    -----
+    The R version of MICE does not have inductive functionality, i.e. first
+    fitting on ``X_train`` and then transforming any ``X_test`` without
+    additional fitting. We do this by storing each feature's predictor during
+    the round-robin ``fit`` phase, and predicting without refitting (in order)
+    during the ``transform`` phase.
+
+    Features which contain all missing values at ``fit`` are discarded upon
+    ``transform``.
+
+    Features with missing values in transform which did not have any missing
+    values in fit will be imputed with the initial imputation method only.
+
+    References
+    ----------
+    .. [1] `Stef van Buuren, Karin Groothuis-Oudshoorn (2011). "mice:
+        Multivariate Imputation by Chained Equations in R". Journal of
+        Statistical Software 45: 1-67.
+        <https://www.jstatsoft.org/article/view/v045i03>`_
+    """
+
+    def __init__(self,
+                 missing_values='NaN',
+                 imputation_order='ascending',
+                 n_imputations=100,
+                 n_burn_in=10,
+                 predictor=None,
+                 n_nearest_features=None,
+                 initial_strategy="mean",
+                 min_value=None,
+                 max_value=None,
+                 verbose=False,
+                 random_state=None):
+
+        self.missing_values = missing_values
+        self.imputation_order = imputation_order
+        self.n_imputations = n_imputations
+        self.n_burn_in = n_burn_in
+        self.predictor = predictor
+        self.n_nearest_features = n_nearest_features
+        self.initial_strategy = initial_strategy
+        self.min_value = min_value
+        self.max_value = max_value
+        self.verbose = verbose
+        self.random_state = random_state
+
+    def _impute_one_feature(self,
+                            X_filled,
+                            mask_missing_values,
+                            feat_idx,
+                            neighbor_feat_idx,
+                            predictor=None,
+                            fit_mode=True):
+        """Impute a single feature from the others provided.
+
+        This function predicts the missing values of one of the features using
+        the current estimates of all the other features. The ``predictor`` must
+        support ``return_std=True`` in its ``predict`` method for this function
+        to work.
+
+        Parameters
+        ----------
+        X_filled : ndarray
+            Input data with the most recent imputations.
+
+        mask_missing_values : ndarray
+            Input data's missing indicator matrix.
+
+        feat_idx : int
+            Index of the feature currently being imputed.
+
+        neighbor_feat_idx : ndarray
+            Indices of the features to be used in imputing ``feat_idx``.
+
+        predictor : object
+            The predictor to use at this step of the round-robin imputation.
+            It must support ``return_std`` in its ``predict`` method.
+            If None, it will be cloned from self._predictor.
+
+        fit_mode : boolean, default=True
+            Whether to fit and predict with the predictor or just predict.
+
+        Returns
+        -------
+        X_filled : ndarray
+            Input data with ``X_filled[missing_row_mask, feat_idx]`` updated.
+
+        predictor : predictor with sklearn API
+            The fitted predictor used to impute
+            ``X_filled[missing_row_mask, feat_idx]``.
+        """
+
+        # if nothing is missing, just return the default
+        # (should not happen at fit time because feat_ids would be excluded)
+        missing_row_mask = mask_missing_values[:, feat_idx]
+        if not np.any(missing_row_mask):
+            return X_filled, predictor
+
+        if predictor is None and fit_mode is False:
+            raise ValueError("If fit_mode is False, then an already-fitted "
+                             "predictor should be passed in.")
+
+        if predictor is None:
+            predictor = clone(self._predictor)
+
+        if fit_mode:
+            X_train = safe_indexing(X_filled[:, neighbor_feat_idx],
+                                    ~missing_row_mask)
+            y_train = safe_indexing(X_filled[:, feat_idx],
+                                    ~missing_row_mask)
+            predictor.fit(X_train, y_train)
+
+        # get posterior samples
+        X_test = safe_indexing(X_filled[:, neighbor_feat_idx],
+                               missing_row_mask)
+        mus, sigmas = predictor.predict(X_test, return_std=True)
+        good_sigmas = sigmas > 0
+        imputed_values = np.zeros(mus.shape, dtype=X_filled.dtype)
+        imputed_values[~good_sigmas] = mus[~good_sigmas]
+        imputed_values[good_sigmas] = self.random_state_.normal(
+            loc=mus[good_sigmas], scale=sigmas[good_sigmas])
+
+        # clip the values
+        imputed_values = np.clip(imputed_values,
+                                 self._min_value,
+                                 self._max_value)
+
+        # update the feature
+        X_filled[missing_row_mask, feat_idx] = imputed_values
+        return X_filled, predictor
+
+    def _get_neighbor_feat_idx(self,
+                               n_features,
+                               feat_idx,
+                               abs_corr_mat):
+        """Get a list of other features to predict ``feat_idx``.
+
+        If self.n_nearest_features is less than or equal to the total
+        number of features, then use a probability proportional to the absolute
+        correlation between ``feat_idx`` and each other feature to randomly
+        choose a subsample of the other features (without replacement).
+
+        Parameters
+        ----------
+        n_features : int
+            Number of features in ``X``.
+
+        feat_idx : int
+            Index of the feature currently being imputed.
+
+        abs_corr_mat : ndarray, shape (n_features, n_features)
+            Absolute correlation matrix of ``X``. The diagonal has been zeroed
+            out and each feature has been normalized to sum to 1. Can be None.
+
+        Returns
+        -------
+        neighbor_feat_idx : array-like
+            The features to use to impute ``feat_idx``.
+        """
+        if (self.n_nearest_features is not None and
+                self.n_nearest_features < n_features):
+            p = abs_corr_mat[:, feat_idx]
+            neighbor_feat_idx = self.random_state_.choice(
+                np.arange(n_features), self.n_nearest_features, replace=False,
+                p=p)
+        else:
+            inds_left = np.arange(feat_idx)
+            inds_right = np.arange(feat_idx + 1, n_features)
+            neighbor_feat_idx = np.concatenate((inds_left, inds_right))
+        return neighbor_feat_idx
+
+    def _get_ordered_idx(self, mask_missing_values):
+        """Decide in what order we will update the features.
+
+        As a homage to the MICE R package, we will have 4 main options of
+        how to order the updates, and use a random order if anything else
+        is specified.
+
+        Also, this function skips features which have no missing values.
+
+        Parameters
+        ----------
+        mask_missing_values : array-like, shape (n_samples, n_features)
+            Input data's missing indicator matrix, where "n_samples" is the
+            number of samples and "n_features" is the number of features.
+
+        Returns
+        -------
+        ordered_idx : ndarray, shape (n_features,)
+            The order in which to impute the features.
+        """
+        frac_of_missing_values = mask_missing_values.mean(axis=0)
+        missing_values_idx = np.nonzero(frac_of_missing_values)[0]
+        if self.imputation_order == 'roman':
+            ordered_idx = missing_values_idx
+        elif self.imputation_order == 'arabic':
+            ordered_idx = missing_values_idx[::-1]
+        elif self.imputation_order == 'ascending':
+            n = len(frac_of_missing_values) - len(missing_values_idx)
+            ordered_idx = np.argsort(frac_of_missing_values,
+                                     kind='mergesort')[n:][::-1]
+        elif self.imputation_order == 'descending':
+            n = len(frac_of_missing_values) - len(missing_values_idx)
+            ordered_idx = np.argsort(frac_of_missing_values,
+                                     kind='mergesort')[n:]
+        elif self.imputation_order == 'random':
+            ordered_idx = missing_values_idx
+            self.random_state_.shuffle(ordered_idx)
+        else:
+            raise ValueError("Got an invalid imputation order: '{0}'. It must "
+                             "be one of the following: 'roman', 'arabic', "
+                             "'ascending', 'descending', or "
+                             "'random'.".format(self.imputation_order))
+        return ordered_idx
+
+    def _get_abs_corr_mat(self, X_filled, tolerance=1e-6):
+        """Get absolute correlation matrix between features.
+
+        Parameters
+        ----------
+        X_filled : ndarray, shape (n_samples, n_features)
+            Input data with the most recent imputations.
+
+        tolerance : float, optional (default=1e-6)
+            ``abs_corr_mat`` can have nans, which will be replaced
+            with ``tolerance``.
+
+        Returns
+        -------
+        abs_corr_mat : ndarray, shape (n_features, n_features)
+            Absolute correlation matrix of ``X`` at the beginning of the
+            current round. The diagonal has been zeroed out and each feature's
+            absolute correlations with all others have been normalized to sum
+            to 1.
+        """
+        n_features = X_filled.shape[1]
+        if (self.n_nearest_features is None or
+                self.n_nearest_features >= n_features):
+            return None
+        abs_corr_mat = np.abs(np.corrcoef(X_filled.T))
+        # np.corrcoef is not defined for features with zero std
+        abs_corr_mat[np.isnan(abs_corr_mat)] = tolerance
+        # ensures exploration, i.e. at least some probability of sampling
+        abs_corr_mat[abs_corr_mat < tolerance] = tolerance
+        # features are not their own neighbors
+        np.fill_diagonal(abs_corr_mat, 0)
+        # needs to sum to 1 for np.random.choice sampling
+        abs_corr_mat = normalize(abs_corr_mat, norm='l1', axis=0, copy=False)
+        return abs_corr_mat
+
+    def _initial_imputation(self, X):
+        """Perform initial imputation for input X.
+
+        Parameters
+        ----------
+        X : ndarray, shape (n_samples, n_features)
+            Input data, where "n_samples" is the number of samples and
+            "n_features" is the number of features.
+
+        Returns
+        -------
+        Xt : ndarray, shape (n_samples, n_features)
+            Input data, where "n_samples" is the number of samples and
+            "n_features" is the number of features.
+
+        X_filled : ndarray, shape (n_samples, n_features)
+            Input data with the most recent imputations.
+
+        mask_missing_values : ndarray, shape (n_samples, n_features)
+            Input data's missing indicator matrix, where "n_samples" is the
+            number of samples and "n_features" is the number of features.
+        """
+        X = check_array(X, dtype=FLOAT_DTYPES, order="F",
+                        force_all_finite='allow-nan'
+                        if self.missing_values == 'NaN'
+                        or np.isnan(self.missing_values) else True)
+
+        mask_missing_values = _get_mask(X, self.missing_values)
+        if self.initial_imputer_ is None:
+            self.initial_imputer_ = SimpleImputer(
+                                            missing_values=self.missing_values,
+                                            strategy=self.initial_strategy)
+            X_filled = self.initial_imputer_.fit_transform(X)
+        else:
+            X_filled = self.initial_imputer_.transform(X)
+
+        valid_mask = np.flatnonzero(np.logical_not(
+            np.isnan(self.initial_imputer_.statistics_)))
+        Xt = X[:, valid_mask]
+        mask_missing_values = mask_missing_values[:, valid_mask]
+
+        return Xt, X_filled, mask_missing_values
+
+    def fit_transform(self, X, y=None):
+        """Fits the imputer on X and return the transformed X.
+
+        Parameters
+        ----------
+        X : array-like, shape (n_samples, n_features)
+            Input data, where "n_samples" is the number of samples and
+            "n_features" is the number of features.
+
+        y : ignored.
+
+        Returns
+        -------
+        Xt : array-like, shape (n_samples, n_features)
+             The imputed input data.
+        """
+        self.random_state_ = getattr(self, "random_state_",
+                                     check_random_state(self.random_state))
+
+        if self.predictor is None:
+            from .linear_model import BayesianRidge
+            self._predictor = BayesianRidge()
+        else:
+            self._predictor = clone(self.predictor)
+
+        self._min_value = np.nan if self.min_value is None else self.min_value
+        self._max_value = np.nan if self.max_value is None else self.max_value
+
+        self.initial_imputer_ = None
+        X, X_filled, mask_missing_values = self._initial_imputation(X)
+
+        # edge case: in case the user specifies 0 for n_imputations,
+        # then there is no need to do burn in and the result should be
+        # just the initial imputation (before clipping)
+        if self.n_imputations < 1:
+            return X_filled
+
+        X_filled = np.clip(X_filled, self._min_value, self._max_value)
+
+        # order in which to impute
+        # note this is probably too slow for large feature data (d > 100000)
+        # and a better way would be good.
+        # see: https://goo.gl/KyCNwj and subsequent comments
+        ordered_idx = self._get_ordered_idx(mask_missing_values)
+
+        abs_corr_mat = self._get_abs_corr_mat(X_filled)
+
+        # impute data
+        n_rounds = self.n_burn_in + self.n_imputations
+        n_samples, n_features = X_filled.shape
+        Xt = np.zeros((n_samples, n_features), dtype=X.dtype)
+        self.imputation_sequence_ = []
+        if self.verbose > 0:
+            print("[MICE] Completing matrix with shape %s" % (X.shape,))
+        start_t = time()
+        for i_rnd in range(n_rounds):
+            if self.imputation_order == 'random':
+                ordered_idx = self._get_ordered_idx(mask_missing_values)
+
+            for feat_idx in ordered_idx:
+                neighbor_feat_idx = self._get_neighbor_feat_idx(n_features,
+                                                                feat_idx,
+                                                                abs_corr_mat)
+                X_filled, predictor = self._impute_one_feature(
+                    X_filled, mask_missing_values, feat_idx, neighbor_feat_idx,
+                    predictor=None, fit_mode=True)
+                predictor_triplet = MICETriplet(feat_idx,
+                                                neighbor_feat_idx,
+                                                predictor)
+                self.imputation_sequence_.append(predictor_triplet)
+
+            if i_rnd >= self.n_burn_in:
+                Xt += X_filled
+            if self.verbose > 0:
+                print('[MICE] Ending imputation round '
+                      '%d/%d, elapsed time %0.2f'
+                      % (i_rnd + 1, n_rounds, time() - start_t))
+
+        Xt /= self.n_imputations
+        Xt[~mask_missing_values] = X[~mask_missing_values]
+        return Xt
+
+    def transform(self, X):
+        """Imputes all missing values in X.
+
+        Note that this is stochastic, and that if random_state is not fixed,
+        repeated calls, or permuted input, will yield different results.
+
+        Parameters
+        ----------
+        X : array-like, shape = [n_samples, n_features]
+            The input data to complete.
+
+        Returns
+        -------
+        Xt : array-like, shape (n_samples, n_features)
+             The imputed input data.
+        """
+        check_is_fitted(self, 'initial_imputer_')
+
+        X, X_filled, mask_missing_values = self._initial_imputation(X)
+
+        # edge case: in case the user specifies 0 for n_imputations,
+        # then there is no need to do burn in and the result should be
+        # just the initial imputation (before clipping)
+        if self.n_imputations < 1:
+            return X_filled
+
+        X_filled = np.clip(X_filled, self._min_value, self._max_value)
+
+        n_rounds = self.n_burn_in + self.n_imputations
+        n_imputations = len(self.imputation_sequence_)
+        imputations_per_round = n_imputations // n_rounds
+        i_rnd = 0
+        Xt = np.zeros(X.shape, dtype=X.dtype)
+        if self.verbose > 0:
+            print("[MICE] Completing matrix with shape %s" % (X.shape,))
+        start_t = time()
+        for it, predictor_triplet in enumerate(self.imputation_sequence_):
+            X_filled, _ = self._impute_one_feature(
+                X_filled,
+                mask_missing_values,
+                predictor_triplet.feat_idx,
+                predictor_triplet.neighbor_feat_idx,
+                predictor=predictor_triplet.predictor,
+                fit_mode=False
+            )
+            if not (it + 1) % imputations_per_round:
+                if i_rnd >= self.n_burn_in:
+                    Xt += X_filled
+                if self.verbose > 1:
+                    print('[MICE] Ending imputation round '
+                          '%d/%d, elapsed time %0.2f'
+                          % (i_rnd + 1, n_rounds, time() - start_t))
+                i_rnd += 1
+
+        Xt /= self.n_imputations
+        Xt[~mask_missing_values] = X[~mask_missing_values]
+        return Xt
+
+    def fit(self, X, y=None):
+        """Fits the imputer on X and return self.
+
+        Parameters
+        ----------
+        X : array-like, shape (n_samples, n_features)
+            Input data, where "n_samples" is the number of samples and
+            "n_features" is the number of features.
+
+        y : ignored
+
+        Returns
+        -------
+        self : object
+            Returns self.
+        """
+        self.fit_transform(X)
+        return self
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 9a321e914b238..8508e166cd8f9 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -72,7 +72,7 @@
                 'OrthogonalMatchingPursuit', 'PLSCanonical', 'PLSRegression',
                 'RANSACRegressor', 'RadiusNeighborsRegressor',
                 'RandomForestRegressor', 'Ridge', 'RidgeCV']
-
+ALLOW_NAN = ['Imputer', 'SimpleImputer', 'MICEImputer']

 def _yield_non_meta_checks(name, estimator):
     yield check_estimators_dtypes
@@ -93,7 +93,7 @@ def _yield_non_meta_checks(name, estimator):
         # cross-decomposition's "transform" returns X and Y
         yield check_pipeline_consistency

-    if name not in ['SimpleImputer', 'Imputer']:
+    if name not in ALLOW_NAN:
         # Test that all estimators check their input for NaN's and infs
         yield check_estimators_nan_inf

diff --git a/sklearn/tests/test_impute.py b/sklearn/tests/test_impute.py
index f2bf5912e2213..954a016a835bb 100644
--- a/sklearn/tests/test_impute.py
+++ b/sklearn/tests/test_impute.py
@@ -1,14 +1,19 @@
+from __future__ import division
+
+import pytest

 import numpy as np
 from scipy import sparse

-from sklearn.utils.testing import assert_equal
+from sklearn.utils.testing import assert_allclose
 from sklearn.utils.testing import assert_array_equal
 from sklearn.utils.testing import assert_array_almost_equal
 from sklearn.utils.testing import assert_raises
 from sklearn.utils.testing import assert_false

-from sklearn.impute import SimpleImputer
+from sklearn.impute import SimpleImputer, MICEImputer
+from sklearn.dummy import DummyRegressor
+from sklearn.linear_model import BayesianRidge, ARDRegression
 from sklearn.pipeline import Pipeline
 from sklearn.model_selection import GridSearchCV
 from sklearn import tree
@@ -61,10 +66,14 @@ def test_imputation_shape():

     for strategy in ['mean', 'median', 'most_frequent']:
         imputer = SimpleImputer(strategy=strategy)
-        X_imputed = imputer.fit_transform(X)
-        assert_equal(X_imputed.shape, (10, 2))
         X_imputed = imputer.fit_transform(sparse.csr_matrix(X))
-        assert_equal(X_imputed.shape, (10, 2))
+        assert X_imputed.shape == (10, 2)
+        X_imputed = imputer.fit_transform(X)
+        assert X_imputed.shape == (10, 2)
+
+        mice_imputer = MICEImputer(initial_strategy=strategy)
+        X_imputed = mice_imputer.fit_transform(X)
+        assert X_imputed.shape == (10, 2)

 def safe_median(arr, *args, **kwargs):
@@ -257,3 +266,227 @@ def test_imputation_copy():
     Xt = imputer.fit(X).transform(X)
     Xt.data[0] = -1
     assert_false(np.all(X.data == Xt.data))
+
+    # Note: If X is sparse and if missing_values=0, then a (dense) copy of X is
+    # made, even if copy=False.
+
+
+def test_mice_rank_one():
+    rng = np.random.RandomState(0)
+    d = 100
+    A = rng.rand(d, 1)
+    B = rng.rand(1, d)
+    X = np.dot(A, B)
+    nan_mask = rng.rand(d, d) < 0.5
+    X_missing = X.copy()
+    X_missing[nan_mask] = np.nan
+
+    imputer = MICEImputer(n_imputations=5,
+                          n_burn_in=5,
+                          verbose=True,
+                          random_state=rng)
+    X_filled = imputer.fit_transform(X_missing)
+    assert_allclose(X_filled, X, atol=0.001)
+
+
+@pytest.mark.parametrize(
+    "imputation_order",
+    ['random', 'roman', 'ascending', 'descending', 'arabic']
+)
+def test_mice_imputation_order(imputation_order):
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 10
+    X = sparse_random_matrix(n, d, density=0.10, random_state=rng).toarray()
+    X[:, 0] = 1  # this column should not be discarded by MICEImputer
+
+    imputer = MICEImputer(missing_values=0,
+                          n_imputations=1,
+                          n_burn_in=1,
+                          n_nearest_features=5,
+                          min_value=0,
+                          max_value=1,
+                          verbose=False,
+                          imputation_order=imputation_order,
+                          random_state=rng)
+    imputer.fit_transform(X)
+    ordered_idx = [i.feat_idx for i in imputer.imputation_sequence_]
+    if imputation_order == 'roman':
+        assert np.all(ordered_idx[:d-1] == np.arange(1, d))
+    elif imputation_order == 'arabic':
+        assert np.all(ordered_idx[:d-1] == np.arange(d-1, 0, -1))
+    elif imputation_order == 'random':
+        ordered_idx_round_1 = ordered_idx[:d-1]
+        ordered_idx_round_2 = ordered_idx[d-1:]
+        assert ordered_idx_round_1 != ordered_idx_round_2
+    elif 'ending' in imputation_order:
+        assert len(ordered_idx) == 2 * (d - 1)
+
+
+@pytest.mark.parametrize(
+    "predictor",
+    [DummyRegressor(), BayesianRidge(), ARDRegression()]
+)
+def test_mice_predictors(predictor):
+    rng = np.random.RandomState(0)
+
+    n = 100
+    d = 10
+    X = sparse_random_matrix(n, d, density=0.10, random_state=rng).toarray()
+
+    imputer = MICEImputer(missing_values=0,
+                          n_imputations=1,
+                          n_burn_in=1,
+                          predictor=predictor,
+                          random_state=rng)
+    imputer.fit_transform(X)
+
+    # check that types are correct for predictors
+    hashes = []
+    for triplet in imputer.imputation_sequence_:
+        assert triplet.predictor
+        hashes.append(id(triplet.predictor))
+
+    # check that each predictor is unique
+    assert len(set(hashes)) == len(hashes)
+
+
+def test_mice_clip():
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 10
+    X = sparse_random_matrix(n, d, density=0.10,
+                             random_state=rng).toarray()
+
+    imputer = MICEImputer(missing_values=0,
+                          n_imputations=1,
+                          n_burn_in=1,
+                          min_value=0.1,
+                          max_value=0.2,
+                          random_state=rng)
+
+    Xt = imputer.fit_transform(X)
+    assert_allclose(np.min(Xt[X == 0]), 0.1)
+    assert_allclose(np.max(Xt[X == 0]), 0.2)
+    assert_allclose(Xt[X != 0], X[X != 0])
+
+
+@pytest.mark.parametrize(
+    "strategy",
+    ["mean", "median", "most_frequent"]
+)
+def test_mice_missing_at_transform(strategy):
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 10
+    X_train = rng.randint(low=0, high=3, size=(n, d))
+    X_test = rng.randint(low=0, high=3, size=(n, d))
+
+    X_train[:, 0] = 1  # definitely no missing values in 0th column
+    X_test[0, 0] = 0  # definitely missing value in 0th column
+
+    mice = MICEImputer(missing_values=0,
+                       n_imputations=1,
+                       n_burn_in=1,
+                       initial_strategy=strategy,
+                       random_state=rng).fit(X_train)
+    initial_imputer = SimpleImputer(missing_values=0,
+                                    strategy=strategy).fit(X_train)
+
+    # if there were no missing values at time of fit, then mice will
+    # only use the initial imputer for that feature at transform
+    assert np.all(mice.transform(X_test)[:, 0] ==
+                  initial_imputer.transform(X_test)[:, 0])
+
+
+def test_mice_transform_stochasticity():
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 10
+    X = sparse_random_matrix(n, d, density=0.10,
+                             random_state=rng).toarray()
+
+    imputer = MICEImputer(missing_values=0,
+                          n_imputations=1,
+                          n_burn_in=1,
+                          random_state=rng)
+    imputer.fit(X)
+
+    X_fitted_1 = imputer.transform(X)
+    X_fitted_2 = imputer.transform(X)
+
+    # sufficient to assert that the means are not the same
+    assert np.mean(X_fitted_1) != pytest.approx(np.mean(X_fitted_2))
+
+
+def test_mice_no_missing():
+    rng = np.random.RandomState(0)
+    X = rng.rand(100, 100)
+    X[:, 0] = np.nan
+    m1 = MICEImputer(n_imputations=10, random_state=rng)
+    m2 = MICEImputer(n_imputations=10, random_state=rng)
+    pred1 = m1.fit(X).transform(X)
+    pred2 = m2.fit_transform(X)
+    # should exclude the first column entirely
+    assert_allclose(X[:, 1:], pred1)
+    # fit and fit_transform should both be identical
+    assert_allclose(pred1, pred2)
+
+
+@pytest.mark.parametrize(
+    "rank",
+    [3, 5]
+)
+def test_mice_transform_recovery(rank):
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 100
+    A = rng.rand(n, rank)
+    B = rng.rand(rank, d)
+    X_filled = np.dot(A, B)
+    # half is randomly missing
+    nan_mask = rng.rand(n, d) < 0.5
+    X_missing = X_filled.copy()
+    X_missing[nan_mask] = np.nan
+
+    # split up data in half
+    n = n // 2
+    X_train = X_missing[:n]
+    X_test_filled = X_filled[n:]
+    X_test = X_missing[n:]
+
+    imputer = MICEImputer(n_imputations=10,
+                          n_burn_in=10,
+                          verbose=True,
+                          random_state=rng).fit(X_train)
+    X_test_est = imputer.transform(X_test)
+    assert_allclose(X_test_filled, X_test_est, rtol=1e-5, atol=0.1)
+
+
+def test_mice_additive_matrix():
+    rng = np.random.RandomState(0)
+    n = 100
+    d = 10
+    A = rng.randn(n, d)
+    B = rng.randn(n, d)
+    X_filled = np.zeros(A.shape)
+    for i in range(d):
+        for j in range(d):
+            X_filled[:, (i+j) % d] += (A[:, i] + B[:, j]) / 2
+    # a quarter is randomly missing
+    nan_mask = rng.rand(n, d) < 0.25
+    X_missing = X_filled.copy()
+    X_missing[nan_mask] = np.nan
+
+    # split up data
+    n = n // 2
+    X_train = X_missing[:n]
+    X_test_filled = X_filled[n:]
+    X_test = X_missing[n:]
+
+    imputer = MICEImputer(n_imputations=25,
+                          n_burn_in=10,
+                          verbose=True,
+                          random_state=rng).fit(X_train)
+    X_test_est = imputer.transform(X_test)
+    assert_allclose(X_test_filled, X_test_est, atol=0.01)

MICE imputer
Proceeding from https://github.com/scikit-learn/scikit-learn/pull/4844#issuecomment-253043238 is a suggestion that MICE imputation be included in scikit-learn. @sergeyf has implemented it [here](https://github.com/hammerlab/fancyimpute/blob/master/fancyimpute/mice.py).

Here we will discuss issues related to its design and inclusion.

@sergeyf [wrote](https://github.com/scikit-learn/scikit-learn/pull/4844#issuecomment-259055975):

> @amueller and @jnothman do either of you have a sense of what the right structure for MICE would be? It's pretty involved, so I don't want to just add a new option to `Imputer`. What about adding a new `MICE` class to `imputation.py`?
>
> Also, the `fit` method doesn't really make sense for MICE. It can pretty much only do `fit_transform` or `transform`: you provide some `X_train` that has holes in it, and MICE fills them in. If you then have an `X_test`, it doesn't use any of the info from `X_train` to fill it in. The simplest thing to do here would be to have `fit` do nothing, have `transform` do all the work, and have `fit_transform` alias to `transform`.

Yes, please create a new `MICEImputer` class.

I think we need an API that allows people to use this in a standard supervised prediction pipeline. Two issues with your proposal then:
1. the data being transformed is often much smaller (and therefore less informative) than the training data. To solve this, we could store the training data (pre-imputed, IMO) and perform the imputation rounds on the concatenation of training and transform data. However:
2. a downstream estimator will operate best when the training transformation and testing transformation is identical. To solve this, we could instead store the model constructed for each column in each of the `n_imputations` performed at train time. (In the case of PMM, we would draw the K nearest neighbors only from the training data as imputed at that point in time.) This also ensures that each row is transformed deterministically given a fitted model.

Could you please comment on how this relates to what's possible in the R interface?

@jnothman It is certainly often true that the data being transformed is smaller than the training data. I like the idea of storing each model for each iteration that was done during the fit phase (as well as the order the imputations were done in). So this would be `fit` or `fit_transform`?

Also, I wasn't going to implement PMM for this pull request: it's much slower and did not work nearly as well in most of my experiments. If someone requests it later, I can add it on without too much trouble.

As for MICE in R: I'm not too familiar with it, but I think MICE just fills in matrices. There is no notion of training and test - that is left up to the user.

One key point is that MICE provides the last `n_last` imputations, and doesn't average them for you. The intent is to then take the multiple imputations and use them all in parallel downstream, averaging only as the final step (after, say, training and prediction of `n_last` classifiers). We could return both an averaged version, and optionally return the last `n_last` imputations. What do you think?

`fit` and `fit_transform` are meant to be functionally identical in terms of the model that is stored. Both would run the full imputation process on the training data and store the sufficient information to run the imputations on new data in `transform`. I suspect in this case little would differ between their implementations but the return value.

OK, sounds good, thanks.

I think I have a sense of what to do. I'll get started once the `return_std` is accepted.

Though you're right, we don't _need_ any ability to transform new data (just as we do not provide this in TSNE), as the technique is unsupervised. So first implement `fit_transform`, then we'll decide whether it's worth storing the model and implementing `transform`.

Well I already implemented `transform` as it was fairly straight-forward. I'll leave it in and we can trim it later if that seems like the wiser path.