diff --git a/doc/computing/scaling_strategies.rst b/doc/computing/scaling_strategies.rst
index 5eee5728e4b9a..277d499f4cc13 100644
--- a/doc/computing/scaling_strategies.rst
+++ b/doc/computing/scaling_strategies.rst
@@ -80,6 +80,7 @@ Here is a list of incremental estimators for different tasks:
       + :class:`sklearn.decomposition.MiniBatchDictionaryLearning`
       + :class:`sklearn.decomposition.IncrementalPCA`
       + :class:`sklearn.decomposition.LatentDirichletAllocation`
+      + :class:`sklearn.decomposition.MiniBatchNMF`
   - Preprocessing
       + :class:`sklearn.preprocessing.StandardScaler`
       + :class:`sklearn.preprocessing.MinMaxScaler`
diff --git a/doc/modules/classes.rst b/doc/modules/classes.rst
index 9640f176925c5..a4217ad44cd76 100644
--- a/doc/modules/classes.rst
+++ b/doc/modules/classes.rst
@@ -319,6 +319,7 @@ Samples generator
    decomposition.MiniBatchDictionaryLearning
    decomposition.MiniBatchSparsePCA
    decomposition.NMF
+   decomposition.MiniBatchNMF
    decomposition.PCA
    decomposition.SparsePCA
    decomposition.SparseCoder
diff --git a/doc/modules/decomposition.rst b/doc/modules/decomposition.rst
index dd8fac301ceed..4f6a889473f13 100644
--- a/doc/modules/decomposition.rst
+++ b/doc/modules/decomposition.rst
@@ -921,6 +921,29 @@ stored components::
     * :ref:`sphx_glr_auto_examples_applications_plot_topics_extraction_with_nmf_lda.py`
     * :ref:`sphx_glr_auto_examples_decomposition_plot_beta_divergence.py`

+.. _MiniBatchNMF:
+
+Mini-batch Non Negative Matrix Factorization
+--------------------------------------------
+
+:class:`MiniBatchNMF` [7]_ implements a faster, but less accurate version of the
+non negative matrix factorization (i.e. :class:`~sklearn.decomposition.NMF`),
+better suited for large datasets.
+
+By default, :class:`MiniBatchNMF` divides the data into mini-batches and
+optimizes the NMF model in an online manner by cycling over the mini-batches
+for the specified number of iterations. The ``batch_size`` parameter controls
+the size of the batches.
+
+In order to speed up the mini-batch algorithm it is also possible to scale
+past batches, giving them less importance than newer batches. This is done
+introducing a so-called forgetting factor controlled by the ``forget_factor``
+parameter.
+
+The estimator also implements ``partial_fit``, which updates ``H`` by iterating
+only once over a mini-batch. This can be used for online learning when the data
+is not readily available from the start, or when the data does not fit into memory.
+
 .. topic:: References:

     .. [1] `"Learning the parts of objects by non-negative matrix factorization"
@@ -945,6 +968,9 @@ stored components::
            the beta-divergence" <1010.1763>`
            C. Fevotte, J. Idier, 2011

+    .. [7] :arxiv:`"Online algorithms for nonnegative matrix factorization with the
+       Itakura-Saito divergence" <1106.4198>`
+       A. Lefevre, F. Bach, C. Fevotte, 2011

 .. _LatentDirichletAllocation:

diff --git a/doc/whats_new/v1.1.rst b/doc/whats_new/v1.1.rst
index 3dc5799c228ad..0acd0612dd0c7 100644
--- a/doc/whats_new/v1.1.rst
+++ b/doc/whats_new/v1.1.rst
@@ -288,6 +288,11 @@ Changelog
 :mod:`sklearn.decomposition`
 ............................

+- |MajorFeature| Added a new estimator :class:`decomposition.MiniBatchNMF`. It is a
+  faster but less accurate version of non-negative matrix factorization, better suited
+  for large datasets. :pr:`16948` by :user:`Chiara Marmo <cmarmo>`,
+  :user:`Patricio Cerda <pcerda>` and :user:`Jérémie du Boisberranger <jeremiedbb>`.
+
 - |Enhancement| :class:`decomposition.PCA` exposes a parameter `n_oversamples` to tune
   :func:`sklearn.decomposition.randomized_svd` and
   get accurate results when the number of features is large.
diff --git a/examples/applications/plot_topics_extraction_with_nmf_lda.py b/examples/applications/plot_topics_extraction_with_nmf_lda.py
index 48b69d710226b..38945241ab68b 100644
--- a/examples/applications/plot_topics_extraction_with_nmf_lda.py
+++ b/examples/applications/plot_topics_extraction_with_nmf_lda.py
@@ -30,13 +30,15 @@
 import matplotlib.pyplot as plt

 from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer
-from sklearn.decomposition import NMF, LatentDirichletAllocation
+from sklearn.decomposition import NMF, MiniBatchNMF, LatentDirichletAllocation
 from sklearn.datasets import fetch_20newsgroups

 n_samples = 2000
 n_features = 1000
 n_components = 10
 n_top_words = 20
+batch_size = 128
+init = "nndsvda"

 def plot_top_words(model, feature_names, n_top_words, title):
@@ -101,7 +103,15 @@ def plot_top_words(model, feature_names, n_top_words, title):
     "n_samples=%d and n_features=%d..." % (n_samples, n_features)
 )
 t0 = time()
-nmf = NMF(n_components=n_components, random_state=1, alpha=0.1, l1_ratio=0.5).fit(tfidf)
+nmf = NMF(
+    n_components=n_components,
+    random_state=1,
+    init=init,
+    beta_loss="frobenius",
+    alpha_W=0.00005,
+    alpha_H=0.00005,
+    l1_ratio=1,
+).fit(tfidf)
 print("done in %0.3fs." % (time() - t0))

@@ -121,10 +131,12 @@ def plot_top_words(model, feature_names, n_top_words, title):
 nmf = NMF(
     n_components=n_components,
     random_state=1,
+    init=init,
     beta_loss="kullback-leibler",
     solver="mu",
     max_iter=1000,
-    alpha=0.1,
+    alpha_W=0.00005,
+    alpha_H=0.00005,
     l1_ratio=0.5,
 ).fit(tfidf)
 print("done in %0.3fs." % (time() - t0))
@@ -137,6 +149,63 @@ def plot_top_words(model, feature_names, n_top_words, title):
     "Topics in NMF model (generalized Kullback-Leibler divergence)",
 )

+# Fit the MiniBatchNMF model
+print(
+    "\n" * 2,
+    "Fitting the MiniBatchNMF model (Frobenius norm) with tf-idf "
+    "features, n_samples=%d and n_features=%d, batch_size=%d..."
+    % (n_samples, n_features, batch_size),
+)
+t0 = time()
+mbnmf = MiniBatchNMF(
+    n_components=n_components,
+    random_state=1,
+    batch_size=batch_size,
+    init=init,
+    beta_loss="frobenius",
+    alpha_W=0.00005,
+    alpha_H=0.00005,
+    l1_ratio=0.5,
+).fit(tfidf)
+print("done in %0.3fs." % (time() - t0))
+
+
+tfidf_feature_names = tfidf_vectorizer.get_feature_names_out()
+plot_top_words(
+    mbnmf,
+    tfidf_feature_names,
+    n_top_words,
+    "Topics in MiniBatchNMF model (Frobenius norm)",
+)
+
+# Fit the MiniBatchNMF model
+print(
+    "\n" * 2,
+    "Fitting the MiniBatchNMF model (generalized Kullback-Leibler "
+    "divergence) with tf-idf features, n_samples=%d and n_features=%d, "
+    "batch_size=%d..." % (n_samples, n_features, batch_size),
+)
+t0 = time()
+mbnmf = MiniBatchNMF(
+    n_components=n_components,
+    random_state=1,
+    batch_size=batch_size,
+    init=init,
+    beta_loss="kullback-leibler",
+    alpha_W=0.00005,
+    alpha_H=0.00005,
+    l1_ratio=0.5,
+).fit(tfidf)
+print("done in %0.3fs." % (time() - t0))
+
+tfidf_feature_names = tfidf_vectorizer.get_feature_names_out()
+plot_top_words(
+    mbnmf,
+    tfidf_feature_names,
+    n_top_words,
+    "Topics in MiniBatchNMF model (generalized Kullback-Leibler divergence)",
+)
+
 print(
     "\n" * 2,
     "Fitting LDA models with tf features, n_samples=%d and n_features=%d..."
diff --git a/sklearn/decomposition/__init__.py b/sklearn/decomposition/__init__.py
index c280aeb2cf71d..c5f323d3c5d72 100644
--- a/sklearn/decomposition/__init__.py
+++ b/sklearn/decomposition/__init__.py
@@ -5,7 +5,11 @@
 """

-from ._nmf import NMF, non_negative_factorization
+from ._nmf import (
+    NMF,
+    MiniBatchNMF,
+    non_negative_factorization,
+)
 from ._pca import PCA
 from ._incremental_pca import IncrementalPCA
 from ._kernel_pca import KernelPCA
@@ -31,6 +35,7 @@
     "IncrementalPCA",
     "KernelPCA",
     "MiniBatchDictionaryLearning",
+    "MiniBatchNMF",
     "MiniBatchSparsePCA",
     "NMF",
     "PCA",
diff --git a/sklearn/decomposition/_nmf.py b/sklearn/decomposition/_nmf.py
index 443f9c4be1649..7623822ba5912 100644
--- a/sklearn/decomposition/_nmf.py
+++ b/sklearn/decomposition/_nmf.py
@@ -10,14 +10,16 @@
 import numpy as np
 import scipy.sparse as sp
 import time
+import itertools
 import warnings
 from math import sqrt
+from scipy import linalg

 from ._cdnmf_fast import _update_cdnmf_fast
 from .._config import config_context
 from ..base import BaseEstimator, TransformerMixin, _ClassNamePrefixFeaturesOutMixin
 from ..exceptions import ConvergenceWarning
-from ..utils import check_random_state, check_array
+from ..utils import check_random_state, check_array, gen_batches
 from ..utils.extmath import randomized_svd, safe_sparse_dot, squared_norm
 from ..utils.validation import (
     check_is_fitted,
@@ -30,7 +32,7 @@
 def norm(x):
     """Dot product-based Euclidean norm implementation.

-    See: http://fseoane.net/blog/2011/computing-the-vector-norm/
+    See: http://fa.bianp.net/blog/2011/computing-the-vector-norm/

     Parameters
     ----------
@@ -166,6 +168,7 @@ def _beta_divergence(X, W, H, beta, square_root=False):
         res /= beta * (beta - 1)

     if square_root:
+        res = max(res, 0)  # avoid negative number due to rounding errors
         return np.sqrt(2 * res)
     else:
         return res
@@ -493,10 +496,10 @@ def _fit_coordinate_descent(

     References
     ----------
-    Cichocki, Andrzej, and Phan, Anh-Huy. "Fast local algorithms for
-    large scale nonnegative matrix and tensor factorizations."
-    IEICE transactions on fundamentals of electronics, communications and
-    computer sciences 92.3: 708-721, 2009.
+    .. [1] :doi:`"Fast local algorithms for large scale nonnegative matrix and tensor
+       factorizations" <10.1587/transfun.E92.A.708>`
+       Cichocki, Andrzej, and P. H. A. N. Anh-Huy. IEICE transactions on fundamentals
+       of electronics, communications and computer sciences 92.3: 708-721, 2009.
     """
     # so W and Ht are both in C order in memory
     Ht = check_array(H.T, order="C")
@@ -637,11 +640,15 @@ def _multiplicative_update_w(
     if gamma != 1:
         delta_W **= gamma

-    return delta_W, H_sum, HHt, XHt
+    W *= delta_W

+    return W, H_sum, HHt, XHt

-def _multiplicative_update_h(X, W, H, beta_loss, l1_reg_H, l2_reg_H, gamma):
-    """Update H in Multiplicative Update NMF."""
+
+def _multiplicative_update_h(
+    X, W, H, beta_loss, l1_reg_H, l2_reg_H, gamma, A=None, B=None, rho=None
+):
+    """update H in Multiplicative Update NMF."""
     if beta_loss == 2:
         numerator = safe_sparse_dot(W.T, X)
         denominator = np.linalg.multi_dot([W.T, W, H])
@@ -712,14 +719,27 @@ def _multiplicative_update_h(X, W, H, beta_loss, l1_reg_H, l2_reg_H, gamma):
         denominator = denominator + l2_reg_H * H
     denominator[denominator == 0] = EPSILON

-    numerator /= denominator
-    delta_H = numerator
-
-    # gamma is in ]0, 1]
-    if gamma != 1:
-        delta_H **= gamma
+    if A is not None and B is not None:
+        # Updates for the online nmf
+        if gamma != 1:
+            H **= 1 / gamma
+        numerator *= H
+        A *= rho
+        B *= rho
+        A += numerator
+        B += denominator
+        H = A / B
+
+        if gamma != 1:
+            H **= gamma
+    else:
+        delta_H = numerator
+        delta_H /= denominator
+        if gamma != 1:
+            delta_H **= gamma
+        H *= delta_H

-    return delta_H
+    return H

 def _fit_multiplicative_update(
@@ -800,6 +820,8 @@ def _fit_multiplicative_update(

     References
     ----------
+    Lee, D. D., & Seung, H., S. (2001). Algorithms for Non-negative Matrix
+    Factorization. Adv. Neural Inform. Process. Syst.. 13.
     Fevotte, C., & Idier, J. (2011). Algorithms for nonnegative matrix
     factorization with the beta-divergence. Neural Computation, 23(9).
     """
@@ -823,21 +845,35 @@ def _fit_multiplicative_update(
     for n_iter in range(1, max_iter + 1):
         # update W
         # H_sum, HHt and XHt are saved and reused if not update_H
-        delta_W, H_sum, HHt, XHt = _multiplicative_update_w(
-            X, W, H, beta_loss, l1_reg_W, l2_reg_W, gamma, H_sum, HHt, XHt, update_H
+        W, H_sum, HHt, XHt = _multiplicative_update_w(
+            X,
+            W,
+            H,
+            beta_loss=beta_loss,
+            l1_reg_W=l1_reg_W,
+            l2_reg_W=l2_reg_W,
+            gamma=gamma,
+            H_sum=H_sum,
+            HHt=HHt,
+            XHt=XHt,
+            update_H=update_H,
         )
-        W *= delta_W

         # necessary for stability with beta_loss < 1
         if beta_loss < 1:
             W[W < np.finfo(np.float64).eps] = 0.0

-        # update H
+        # update H (only at fit or fit_transform)
         if update_H:
-            delta_H = _multiplicative_update_h(
-                X, W, H, beta_loss, l1_reg_H, l2_reg_H, gamma
+            H = _multiplicative_update_h(
+                X,
+                W,
+                H,
+                beta_loss=beta_loss,
+                l1_reg_H=l1_reg_H,
+                l2_reg_H=l2_reg_H,
+                gamma=gamma,
             )
-            H *= delta_H

             # These values will be recomputed since H changed
             H_sum, HHt, XHt = None, None, None
@@ -902,15 +938,16 @@ def non_negative_factorization(

         .. math::

-            0.5 * ||X - WH||_{loss}^2
+            L(W, H) &= 0.5 * ||X - WH||_{loss}^2
+
+            &+ alpha\\_W * l1\\_ratio * n\\_features * ||vec(W)||_1

-            + alpha\\_W * l1_{ratio} * n\\_features * ||vec(W)||_1
+            &+ alpha\\_H * l1\\_ratio * n\\_samples * ||vec(H)||_1

-            + alpha\\_H * l1_{ratio} * n\\_samples * ||vec(H)||_1
+            &+ 0.5 * alpha\\_W * (1 - l1\\_ratio) * n\\_features * ||W||_{Fro}^2

-            + 0.5 * alpha\\_W * (1 - l1_{ratio}) * n\\_features * ||W||_{Fro}^2
+            &+ 0.5 * alpha\\_H * (1 - l1\\_ratio) * n\\_samples * ||H||_{Fro}^2

-            + 0.5 * alpha\\_H * (1 - l1_{ratio}) * n\\_samples * ||H||_{Fro}^2

     Where:

@@ -1084,13 +1121,14 @@ def non_negative_factorization(

     References
     ----------
-    Cichocki, Andrzej, and P. H. A. N. Anh-Huy. "Fast local algorithms for
-    large scale nonnegative matrix and tensor factorizations."
-    IEICE transactions on fundamentals of electronics, communications and
-    computer sciences 92.3: 708-721, 2009.
-
-    Fevotte, C., & Idier, J. (2011). Algorithms for nonnegative matrix
-    factorization with the beta-divergence. Neural Computation, 23(9).
+    .. [1] :doi:`"Fast local algorithms for large scale nonnegative matrix and tensor
+       factorizations" <10.1587/transfun.E92.A.708>`
+       Cichocki, Andrzej, and P. H. A. N. Anh-Huy. IEICE transactions on fundamentals
+       of electronics, communications and computer sciences 92.3: 708-721, 2009.
+
+    .. [2] :doi:`"Algorithms for nonnegative matrix factorization with the
+       beta-divergence" <10.1162/NECO_a_00168>`
+       Fevotte, C., & Idier, J. (2011). Neural Computation, 23(9).
     """
     X = check_array(X, accept_sparse=("csr", "csc"), dtype=[np.float64, np.float32])

@@ -1120,23 +1158,23 @@ def non_negative_factorization(
 class NMF(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstimator):
     """Non-Negative Matrix Factorization (NMF).

-    Find two non-negative matrices (W, H) whose product approximates the non-
-    negative matrix X. This factorization can be used for example for
-    dimensionality reduction, source separation or topic extraction.
+    Find two non-negative matrices, i.e. matrices with all non-negative elements, (W, H)
+    whose product approximates the non-negative matrix X. This factorization can be used
+    for example for dimensionality reduction, source separation or topic extraction.

     The objective function is:

         .. math::

-            0.5 * ||X - WH||_{loss}^2
+            L(W, H) &= 0.5 * ||X - WH||_{loss}^2

-            + alpha\\_W * l1_{ratio} * n\\_features * ||vec(W)||_1
+            &+ alpha\\_W * l1\\_ratio * n\\_features * ||vec(W)||_1

-            + alpha\\_H * l1_{ratio} * n\\_samples * ||vec(H)||_1
+            &+ alpha\\_H * l1\\_ratio * n\\_samples * ||vec(H)||_1

-            + 0.5 * alpha\\_W * (1 - l1_{ratio}) * n\\_features * ||W||_{Fro}^2
+            &+ 0.5 * alpha\\_W * (1 - l1\\_ratio) * n\\_features * ||W||_{Fro}^2

-            + 0.5 * alpha\\_H * (1 - l1_{ratio}) * n\\_samples * ||H||_{Fro}^2
+            &+ 0.5 * alpha\\_H * (1 - l1\\_ratio) * n\\_samples * ||H||_{Fro}^2

     Where:

@@ -1325,13 +1363,14 @@ class NMF(_ClassNamePrefixFeaturesOutMixin, TransformerMixin, BaseEstimator):

     References
     ----------
-    Cichocki, Andrzej, and P. H. A. N. Anh-Huy. "Fast local algorithms for
-    large scale nonnegative matrix and tensor factorizations."
-    IEICE transactions on fundamentals of electronics, communications and
-    computer sciences 92.3: 708-721, 2009.
+    .. [1] :doi:`"Fast local algorithms for large scale nonnegative matrix and tensor
+       factorizations" <10.1587/transfun.E92.A.708>`
+       Cichocki, Andrzej, and P. H. A. N. Anh-Huy. IEICE transactions on fundamentals
+       of electronics, communications and computer sciences 92.3: 708-721, 2009.

-    Fevotte, C., & Idier, J. (2011). Algorithms for nonnegative matrix
-    factorization with the beta-divergence. Neural Computation, 23(9).
+    .. [2] :doi:`"Algorithms for nonnegative matrix factorization with the
+       beta-divergence" <10.1162/NECO_a_00168>`
+       Fevotte, C., & Idier, J. (2011). Neural Computation, 23(9).

     Examples
     --------
@@ -1632,7 +1671,7 @@ def _fit_transform(self, X, y=None, W=None, H=None, update_H=True):
                 random_state=self.random_state,
             )
         elif self.solver == "mu":
-            W, H, n_iter = _fit_multiplicative_update(
+            W, H, n_iter, *_ = _fit_multiplicative_update(
                 X,
                 W,
                 H,
@@ -1643,8 +1682,8 @@ def _fit_transform(self, X, y=None, W=None, H=None, update_H=True):
                 l1_reg_H,
                 l2_reg_W,
                 l2_reg_H,
-                update_H=update_H,
-                verbose=self.verbose,
+                update_H,
+                self.verbose,
             )
         else:
             raise ValueError("Invalid solver parameter '%s'." % self.solver)
@@ -1729,3 +1768,628 @@ def inverse_transform(self, W):
     def _n_features_out(self):
         """Number of transformed output features."""
         return self.components_.shape[0]
+
+
+class MiniBatchNMF(NMF):
+    """Mini-Batch Non-Negative Matrix Factorization (NMF).
+
+    .. versionadded:: 1.1
+
+    Find two non-negative matrices, i.e. matrices with all non-negative elements,
+    (`W`, `H`) whose product approximates the non-negative matrix `X`. This
+    factorization can be used for example for dimensionality reduction, source
+    separation or topic extraction.
+
+    The objective function is:
+
+        .. math::
+
+            L(W, H) &= 0.5 * ||X - WH||_{loss}^2
+
+            &+ alpha\\_W * l1\\_ratio * n\\_features * ||vec(W)||_1
+
+            &+ alpha\\_H * l1\\_ratio * n\\_samples * ||vec(H)||_1
+
+            &+ 0.5 * alpha\\_W * (1 - l1\\_ratio) * n\\_features * ||W||_{Fro}^2
+
+            &+ 0.5 * alpha\\_H * (1 - l1\\_ratio) * n\\_samples * ||H||_{Fro}^2
+
+    Where:
+
+    :math:`||A||_{Fro}^2 = \\sum_{i,j} A_{ij}^2` (Frobenius norm)
+
+    :math:`||vec(A)||_1 = \\sum_{i,j} abs(A_{ij})` (Elementwise L1 norm)
+
+    The generic norm :math:`||X - WH||_{loss}^2` may represent
+    the Frobenius norm or another supported beta-divergence loss.
+    The choice between options is controlled by the `beta_loss` parameter.
+
+    The objective function is minimized with an alternating minimization of `W`
+    and `H`.
+
+    Note that the transformed data is named `W` and the components matrix is
+    named `H`. In the NMF literature, the naming convention is usually the opposite
+    since the data matrix `X` is transposed.
+
+    Read more in the :ref:`User Guide <MiniBatchNMF>`.
+
+    Parameters
+    ----------
+    n_components : int, default=None
+        Number of components, if `n_components` is not set all features
+        are kept.
+
+    init : {'random', 'nndsvd', 'nndsvda', 'nndsvdar', 'custom'}, default=None
+        Method used to initialize the procedure.
+        Valid options:
+
+        - `None`: 'nndsvda' if `n_components <= min(n_samples, n_features)`,
+          otherwise random.
+
+        - `'random'`: non-negative random matrices, scaled with:
+          `sqrt(X.mean() / n_components)`
+
+        - `'nndsvd'`: Nonnegative Double Singular Value Decomposition (NNDSVD)
+          initialization (better for sparseness).
+
+        - `'nndsvda'`: NNDSVD with zeros filled with the average of X
+          (better when sparsity is not desired).
+
+        - `'nndsvdar'` NNDSVD with zeros filled with small random values
+          (generally faster, less accurate alternative to NNDSVDa
+          for when sparsity is not desired).
+
+        - `'custom'`: use custom matrices `W` and `H`
+
+    batch_size : int, default=1024
+        Number of samples in each mini-batch. Large batch sizes
+        give better long-term convergence at the cost of a slower start.
+
+    beta_loss : float or {'frobenius', 'kullback-leibler', \
+            'itakura-saito'}, default='frobenius'
+        Beta divergence to be minimized, measuring the distance between `X`
+        and the dot product `WH`. Note that values different from 'frobenius'
+        (or 2) and 'kullback-leibler' (or 1) lead to significantly slower
+        fits. Note that for `beta_loss <= 0` (or 'itakura-saito'), the input
+        matrix `X` cannot contain zeros.
+
+    tol : float, default=1e-4
+        Control early stopping based on the norm of the differences in `H`
+        between 2 steps. To disable early stopping based on changes in `H`, set
+        `tol` to 0.0.
+
+    max_no_improvement : int, default=10
+        Control early stopping based on the consecutive number of mini batches
+        that does not yield an improvement on the smoothed cost function.
+        To disable convergence detection based on cost function, set
+        `max_no_improvement` to None.
+
+    max_iter : int, default=200
+        Maximum number of iterations over the complete dataset before
+        timing out.
+
+    alpha_W : float, default=0.0
+        Constant that multiplies the regularization terms of `W`. Set it to zero
+        (default) to have no regularization on `W`.
+
+    alpha_H : float or "same", default="same"
+        Constant that multiplies the regularization terms of `H`. Set it to zero to
+        have no regularization on `H`. If "same" (default), it takes the same value as
+        `alpha_W`.
+
+    l1_ratio : float, default=0.0
+        The regularization mixing parameter, with 0 <= l1_ratio <= 1.
+        For l1_ratio = 0 the penalty is an elementwise L2 penalty
+        (aka Frobenius Norm).
+        For l1_ratio = 1 it is an elementwise L1 penalty.
+        For 0 < l1_ratio < 1, the penalty is a combination of L1 and L2.
+
+    forget_factor : float, default=0.7
+        Amount of rescaling of past information. Its value could be 1 with
+        finite datasets. Choosing values < 1 is recommended with online
+        learning as more recent batches will weight more than past batches.
+
+    fresh_restarts : bool, default=False
+        Whether to completely solve for W at each step. Doing fresh restarts will likely
+        lead to a better solution for a same number of iterations but it is much slower.
+
+    fresh_restarts_max_iter : int, default=30
+        Maximum number of iterations when solving for W at each step. Only used when
+        doing fresh restarts. These iterations may be stopped early based on a small
+        change of W controlled by `tol`.
+
+    transform_max_iter : int, default=None
+        Maximum number of iterations when solving for W at transform time.
+        If None, it defaults to `max_iter`.
+
+    random_state : int, RandomState instance or None, default=None
+        Used for initialisation (when ``init`` == 'nndsvdar' or
+        'random'), and in Coordinate Descent. Pass an int for reproducible
+        results across multiple function calls.
+        See :term:`Glossary <random_state>`.
+
+    verbose : bool, default=False
+        Whether to be verbose.
+
+    Attributes
+    ----------
+    components_ : ndarray of shape (n_components, n_features)
+        Factorization matrix, sometimes called 'dictionary'.
+
+    n_components_ : int
+        The number of components. It is same as the `n_components` parameter
+        if it was given. Otherwise, it will be same as the number of
+        features.
+
+    reconstruction_err_ : float
+        Frobenius norm of the matrix difference, or beta-divergence, between
+        the training data `X` and the reconstructed data `WH` from
+        the fitted model.
+
+    n_iter_ : int
+        Actual number of started iterations over the whole dataset.
+
+    n_steps_ : int
+        Number of mini-batches processed.
+
+    n_features_in_ : int
+        Number of features seen during :term:`fit`.
+
+    feature_names_in_ : ndarray of shape (`n_features_in_`,)
+        Names of features seen during :term:`fit`. Defined only when `X`
+        has feature names that are all strings.
+
+    See Also
+    --------
+    NMF : Non-negative matrix factorization.
+    MiniBatchDictionaryLearning : Finds a dictionary that can best be used to represent
+        data using a sparse code.
+
+    References
+    ----------
+    .. [1] :doi:`"Fast local algorithms for large scale nonnegative matrix and tensor
+       factorizations" <10.1587/transfun.E92.A.708>`
+       Cichocki, Andrzej, and P. H. A. N. Anh-Huy. IEICE transactions on fundamentals
+       of electronics, communications and computer sciences 92.3: 708-721, 2009.
+
+    .. [2] :doi:`"Algorithms for nonnegative matrix factorization with the
+       beta-divergence" <10.1162/NECO_a_00168>`
+       Fevotte, C., & Idier, J. (2011). Neural Computation, 23(9).
+
+    .. [3] :doi:`"Online algorithms for nonnegative matrix factorization with the
+       Itakura-Saito divergence" <10.1109/ASPAA.2011.6082314>`
+       Lefevre, A., Bach, F., Fevotte, C. (2011). WASPA.
+
+    Examples
+    --------
+    >>> import numpy as np
+    >>> X = np.array([[1, 1], [2, 1], [3, 1.2], [4, 1], [5, 0.8], [6, 1]])
+    >>> from sklearn.decomposition import MiniBatchNMF
+    >>> model = MiniBatchNMF(n_components=2, init='random', random_state=0)
+    >>> W = model.fit_transform(X)
+    >>> H = model.components_
+    """
+
+    def __init__(
+        self,
+        n_components=None,
+        *,
+        init=None,
+        batch_size=1024,
+        beta_loss="frobenius",
+        tol=1e-4,
+        max_no_improvement=10,
+        max_iter=200,
+        alpha_W=0.0,
+        alpha_H="same",
+        l1_ratio=0.0,
+        forget_factor=0.7,
+        fresh_restarts=False,
+        fresh_restarts_max_iter=30,
+        transform_max_iter=None,
+        random_state=None,
+        verbose=0,
+    ):
+
+        super().__init__(
+            n_components=n_components,
+            init=init,
+            solver="mu",
+            beta_loss=beta_loss,
+            tol=tol,
+            max_iter=max_iter,
+            random_state=random_state,
+            alpha_W=alpha_W,
+            alpha_H=alpha_H,
+            l1_ratio=l1_ratio,
+            verbose=verbose,
+        )
+
+        self.max_no_improvement = max_no_improvement
+        self.batch_size = batch_size
+        self.forget_factor = forget_factor
+        self.fresh_restarts = fresh_restarts
+        self.fresh_restarts_max_iter = fresh_restarts_max_iter
+        self.transform_max_iter = transform_max_iter
+
+    def _check_params(self, X):
+        super()._check_params(X)
+
+        # batch_size
+        self._batch_size = self.batch_size
+        if not isinstance(self._batch_size, numbers.Integral) or self._batch_size <= 0:
+            raise ValueError(
+                "batch_size must be a positive integer, got "
+                f"{self._batch_size!r} instead."
+            )
+        self._batch_size = min(self._batch_size, X.shape[0])
+
+        # forget_factor
+        self._rho = self.forget_factor ** (self._batch_size / X.shape[0])
+
+        # gamma for Maximization-Minimization (MM) algorithm [Fevotte 2011]
+        if self._beta_loss < 1:
+            self._gamma = 1.0 / (2.0 - self._beta_loss)
+        elif self._beta_loss > 2:
+            self._gamma = 1.0 / (self._beta_loss - 1.0)
+        else:
+            self._gamma = 1.0
+
+        # transform_max_iter
+        self._transform_max_iter = (
+            self.max_iter
+            if self.transform_max_iter is None
+            else self.transform_max_iter
+        )
+
+        return self
+
+    def _solve_W(self, X, H, max_iter):
+        """Minimize the objective function w.r.t W.
+
+        Update W with H being fixed, until convergence. This is the heart
+        of `transform` but it's also used during `fit` when doing fresh restarts.
+        """
+        avg = np.sqrt(X.mean() / self._n_components)
+        W = np.full((X.shape[0], self._n_components), avg, dtype=X.dtype)
+        W_buffer = W.copy()
+
+        # Get scaled regularization terms. Done for each minibatch to take into account
+        # variable sizes of minibatches.
+        l1_reg_W, _, l2_reg_W, _ = self._scale_regularization(X)
+
+        for _ in range(max_iter):
+            W, *_ = _multiplicative_update_w(
+                X, W, H, self._beta_loss, l1_reg_W, l2_reg_W, self._gamma
+            )
+
+            W_diff = linalg.norm(W - W_buffer) / linalg.norm(W)
+            if self.tol > 0 and W_diff <= self.tol:
+                break
+
+            W_buffer[:] = W
+
+        return W
+
+    def _minibatch_step(self, X, W, H, update_H):
+        """Perform the update of W and H for one minibatch."""
+        batch_size = X.shape[0]
+
+        # get scaled regularization terms. Done for each minibatch to take into account
+        # variable sizes of minibatches.
+        l1_reg_W, l1_reg_H, l2_reg_W, l2_reg_H = self._scale_regularization(X)
+
+        # update W
+        if self.fresh_restarts or W is None:
+            W = self._solve_W(X, H, self.fresh_restarts_max_iter)
+        else:
+            W, *_ = _multiplicative_update_w(
+                X, W, H, self._beta_loss, l1_reg_W, l2_reg_W, self._gamma
+            )
+
+        # necessary for stability with beta_loss < 1
+        if self._beta_loss < 1:
+            W[W < np.finfo(np.float64).eps] = 0.0
+
+        batch_cost = (
+            _beta_divergence(X, W, H, self._beta_loss)
+            + l1_reg_W * W.sum()
+            + l1_reg_H * H.sum()
+            + l2_reg_W * (W**2).sum()
+            + l2_reg_H * (H**2).sum()
+        ) / batch_size
+
+        # update H (only at fit or fit_transform)
+        if update_H:
+            H[:] = _multiplicative_update_h(
+                X,
+                W,
+                H,
+                beta_loss=self._beta_loss,
+                l1_reg_H=l1_reg_H,
+                l2_reg_H=l2_reg_H,
+                gamma=self._gamma,
+                A=self._components_numerator,
+                B=self._components_denominator,
+                rho=self._rho,
+            )
+
+            # necessary for stability with beta_loss < 1
+            if self._beta_loss <= 1:
+                H[H < np.finfo(np.float64).eps] = 0.0
+
+        return batch_cost
+
+    def _minibatch_convergence(
+        self, X, batch_cost, H, H_buffer, n_samples, step, n_steps
+    ):
+        """Helper function to encapsulate the early stopping logic"""
+        batch_size = X.shape[0]
+
+        # counts steps starting from 1 for user friendly verbose mode.
+        step = step + 1
+
+        # Ignore first iteration because H is not updated yet.
+        if step == 1:
+            if self.verbose:
+                print(f"Minibatch step {step}/{n_steps}: mean batch cost: {batch_cost}")
+            return False
+
+        # Compute an Exponentially Weighted Average of the cost function to
+        # monitor the convergence while discarding minibatch-local stochastic
+        # variability: https://en.wikipedia.org/wiki/Moving_average
+        if self._ewa_cost is None:
+            self._ewa_cost = batch_cost
+        else:
+            alpha = batch_size / (n_samples + 1)
+            alpha = min(alpha, 1)
+            self._ewa_cost = self._ewa_cost * (1 - alpha) + batch_cost * alpha
+
+        # Log progress to be able to monitor convergence
+        if self.verbose:
+            print(
+                f"Minibatch step {step}/{n_steps}: mean batch cost: "
+                f"{batch_cost}, ewa cost: {self._ewa_cost}"
+            )
+
+        # Early stopping based on change of H
+        H_diff = linalg.norm(H - H_buffer) / linalg.norm(H)
+        if self.tol > 0 and H_diff <= self.tol:
+            if self.verbose:
+                print(f"Converged (small H change) at step {step}/{n_steps}")
+            return True
+
+        # Early stopping heuristic due to lack of improvement on smoothed
+        # cost function
+        if self._ewa_cost_min is None or self._ewa_cost < self._ewa_cost_min:
+            self._no_improvement = 0
+            self._ewa_cost_min = self._ewa_cost
+        else:
+            self._no_improvement += 1
+
+        if (
+            self.max_no_improvement is not None
+            and self._no_improvement >= self.max_no_improvement
+        ):
+            if self.verbose:
+                print(
+                    "Converged (lack of improvement in objective function) "
+                    f"at step {step}/{n_steps}"
+                )
+            return True
+
+        return False
+
+    def fit_transform(self, X, y=None, W=None, H=None):
+        """Learn a NMF model for the data X and returns the transformed data.
+
+        This is more efficient than calling fit followed by transform.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Data matrix to be decomposed.
+
+        y : Ignored
+            Not used, present here for API consistency by convention.
+
+        W : array-like of shape (n_samples, n_components), default=None
+            If `init='custom'`, it is used as initial guess for the solution.
+
+        H : array-like of shape (n_components, n_features), default=None
+            If `init='custom'`, it is used as initial guess for the solution.
+
+        Returns
+        -------
+        W : ndarray of shape (n_samples, n_components)
+            Transformed data.
+        """
+        X = self._validate_data(
+            X, accept_sparse=("csr", "csc"), dtype=[np.float64, np.float32]
+        )
+
+        with config_context(assume_finite=True):
+            W, H, n_iter, n_steps = self._fit_transform(X, W=W, H=H)
+
+        self.reconstruction_err_ = _beta_divergence(
+            X, W, H, self._beta_loss, square_root=True
+        )
+
+        self.n_components_ = H.shape[0]
+        self.components_ = H
+        self.n_iter_ = n_iter
+        self.n_steps_ = n_steps
+
+        return W
+
+    def _fit_transform(self, X, W=None, H=None, update_H=True):
+        """Learn a NMF model for the data X and returns the transformed data.
+
+        Parameters
+        ----------
+        X : {ndarray, sparse matrix} of shape (n_samples, n_features)
+            Data matrix to be decomposed.
+
+        W : array-like of shape (n_samples, n_components), default=None
+            If init='custom', it is used as initial guess for the solution.
+
+        H : array-like of shape (n_components, n_features), default=None
+            If init='custom', it is used as initial guess for the solution.
+            If update_H=False, it is used as a constant, to solve for W only.
+
+        update_H : bool, default=True
+            If True, both W and H will be estimated from initial guesses,
+            this corresponds to a call to the `fit_transform` method.
+            If False, only W will be estimated, this corresponds to a call
+            to the `transform` method.
+
+        Returns
+        -------
+        W : ndarray of shape (n_samples, n_components)
+            Transformed data.
+
+        H : ndarray of shape (n_components, n_features)
+            Factorization matrix, sometimes called 'dictionary'.
+
+        n_iter : int
+            Actual number of started iterations over the whole dataset.
+
+        n_steps : int
+            Number of mini-batches processed.
+        """
+        check_non_negative(X, "NMF (input X)")
+        self._check_params(X)
+
+        if X.min() == 0 and self._beta_loss <= 0:
+            raise ValueError(
+                "When beta_loss <= 0 and X contains zeros, "
+                "the solver may diverge. Please add small values "
+                "to X, or use a positive beta_loss."
+            )
+
+        n_samples = X.shape[0]
+
+        # initialize or check W and H
+        W, H = self._check_w_h(X, W, H, update_H)
+        H_buffer = H.copy()
+
+        # Initialize auxiliary matrices
+        self._components_numerator = H.copy()
+        self._components_denominator = np.ones(H.shape, dtype=H.dtype)
+
+        # Attributes to monitor the convergence
+        self._ewa_cost = None
+        self._ewa_cost_min = None
+        self._no_improvement = 0
+
+        batches = gen_batches(n_samples, self._batch_size)
+        batches = itertools.cycle(batches)
+        n_steps_per_iter = int(np.ceil(n_samples / self._batch_size))
+        n_steps = self.max_iter * n_steps_per_iter
+
+        for i, batch in zip(range(n_steps), batches):
+
+            batch_cost = self._minibatch_step(X[batch], W[batch], H, update_H)
+
+            if update_H and self._minibatch_convergence(
+                X[batch], batch_cost, H, H_buffer, n_samples, i, n_steps
+            ):
+                break
+
+            H_buffer[:] = H
+
+        if self.fresh_restarts:
+            W = self._solve_W(X, H, self._transform_max_iter)
+
+        n_steps = i + 1
+        n_iter = int(np.ceil(n_steps / n_steps_per_iter))
+
+        if n_iter == self.max_iter and self.tol > 0:
+            warnings.warn(
+                f"Maximum number of iterations {self.max_iter} reached. "
+                "Increase it to improve convergence.",
+                ConvergenceWarning,
+            )
+
+        return W, H, n_iter, n_steps
+
+    def transform(self, X):
+        """Transform the data X according to the fitted MiniBatchNMF model.
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Data matrix to be transformed by the model.
+
+        Returns
+        -------
+        W : ndarray of shape (n_samples, n_components)
+            Transformed data.
+        """
+        check_is_fitted(self)
+        X = self._validate_data(
+            X, accept_sparse=("csr", "csc"), dtype=[np.float64, np.float32], reset=False
+        )
+
+        W = self._solve_W(X, self.components_, self._transform_max_iter)
+
+        return W
+
+    def partial_fit(self, X, y=None, W=None, H=None):
+        """Update the model using the data in `X` as a mini-batch.
+
+        This method is expected to be called several times consecutively
+        on different chunks of a dataset so as to implement out-of-core
+        or online learning.
+
+        This is especially useful when the whole dataset is too big to fit in
+        memory at once (see :ref:`scaling_strategies`).
+
+        Parameters
+        ----------
+        X : {array-like, sparse matrix} of shape (n_samples, n_features)
+            Data matrix to be decomposed.
+
+        y : Ignored
+            Not used, present here for API consistency by convention.
+
+        W : array-like of shape (n_samples, n_components), default=None
+            If `init='custom'`, it is used as initial guess for the solution.
+            Only used for the first call to `partial_fit`.
+
+        H : array-like of shape (n_components, n_features), default=None
+            If `init='custom'`, it is used as initial guess for the solution.
+            Only used for the first call to `partial_fit`.
+
+        Returns
+        -------
+        self
+            Returns the instance itself.
+        """
+        has_components = hasattr(self, "components_")
+
+        X = self._validate_data(
+            X,
+            accept_sparse=("csr", "csc"),
+            dtype=[np.float64, np.float32],
+            reset=not has_components,
+        )
+
+        if not has_components:
+            # This instance has not been fitted yet (fit or partial_fit)
+            self._check_params(X)
+            _, H = self._check_w_h(X, W=W, H=H, update_H=True)
+
+            self._components_numerator = H.copy()
+            self._components_denominator = np.ones(H.shape, dtype=H.dtype)
+            self.n_steps_ = 0
+        else:
+            H = self.components_
+
+        self._minibatch_step(X, None, H, update_H=True)
+
+        self.n_components_ = H.shape[0]
+        self.components_ = H
+        self.n_steps_ += 1
+
+        return self
diff --git a/sklearn/utils/estimator_checks.py b/sklearn/utils/estimator_checks.py
index 5e9601ae220a6..cc51507f90d49 100644
--- a/sklearn/utils/estimator_checks.py
+++ b/sklearn/utils/estimator_checks.py
@@ -652,6 +652,9 @@ def _set_checking_parameters(estimator):
         # NMF
         if name == "NMF":
             estimator.set_params(max_iter=500)
+        # MiniBatchNMF
+        if estimator.__class__.__name__ == "MiniBatchNMF":
+            estimator.set_params(max_iter=20, fresh_restarts=True)
         # MLP
         if name in ["MLPClassifier", "MLPRegressor"]:
             estimator.set_params(max_iter=100)

diff --git a/sklearn/decomposition/tests/test_nmf.py b/sklearn/decomposition/tests/test_nmf.py
index cca0fad114ae5..9f3df5b64a803 100644
--- a/sklearn/decomposition/tests/test_nmf.py
+++ b/sklearn/decomposition/tests/test_nmf.py
@@ -1,10 +1,13 @@
 import re
+import sys
+from io import StringIO

 import numpy as np
 import scipy.sparse as sp

 from scipy import linalg
-from sklearn.decomposition import NMF, non_negative_factorization
+from sklearn.decomposition import NMF, MiniBatchNMF
+from sklearn.decomposition import non_negative_factorization
 from sklearn.decomposition import _nmf as nmf  # For testing internals
 from scipy.sparse import csc_matrix

@@ -20,14 +23,17 @@
 from sklearn.exceptions import ConvergenceWarning

-@pytest.mark.parametrize("solver", ["cd", "mu"])
-def test_convergence_warning(solver):
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_convergence_warning(Estimator, solver):
     convergence_warning = (
         "Maximum number of iterations 1 reached. Increase it to improve convergence."
     )
     A = np.ones((2, 2))
     with pytest.warns(ConvergenceWarning, match=convergence_warning):
-        NMF(solver=solver, max_iter=1).fit(A)
+        Estimator(max_iter=1, **solver).fit(A)

 def test_initialize_nn_output():
@@ -39,15 +45,13 @@ def test_initialize_nn_output():
         assert not ((W < 0).any() or (H < 0).any())

+@pytest.mark.filterwarnings(
+    r"ignore:The multiplicative update \('mu'\) solver cannot update zeros present in"
+    r" the initialization"
+)
 def test_parameter_checking():
     A = np.ones((2, 2))
     name = "spam"
-    msg = "Invalid solver parameter: got 'spam' instead of one of"
-    with pytest.raises(ValueError, match=msg):
-        NMF(solver=name).fit(A)
-    msg = "Invalid init parameter: got 'spam' instead of one of"
-    with pytest.raises(ValueError, match=msg):
-        NMF(init=name).fit(A)

     with ignore_warnings(category=FutureWarning):
         # TODO remove in 1.2
@@ -55,21 +59,17 @@ def test_parameter_checking():
         with pytest.raises(ValueError, match=msg):
             NMF(regularization=name).fit(A)

-    msg = "Invalid beta_loss parameter: got 'spam' instead of one"
-    with pytest.raises(ValueError, match=msg):
-        NMF(solver="mu", beta_loss=name).fit(A)
     msg = "Invalid beta_loss parameter: solver 'cd' does not handle beta_loss = 1.0"
     with pytest.raises(ValueError, match=msg):
         NMF(solver="cd", beta_loss=1.0).fit(A)
-
     msg = "Negative values in data passed to"
     with pytest.raises(ValueError, match=msg):
         NMF().fit(-A)
-    with pytest.raises(ValueError, match=msg):
-        nmf._initialize_nmf(-A, 2, "nndsvd")
     clf = NMF(2, tol=0.1).fit(A)
     with pytest.raises(ValueError, match=msg):
         clf.transform(-A)
+    with pytest.raises(ValueError, match=msg):
+        nmf._initialize_nmf(-A, 2, "nndsvd")

     for init in ["nndsvd", "nndsvda", "nndsvdar"]:
         msg = re.escape(
@@ -78,10 +78,59 @@ def test_parameter_checking():
         )
         with pytest.raises(ValueError, match=msg):
             NMF(3, init=init).fit(A)
+        with pytest.raises(ValueError, match=msg):
+            MiniBatchNMF(3, init=init).fit(A)
         with pytest.raises(ValueError, match=msg):
             nmf._initialize_nmf(A, 3, init)

+@pytest.mark.parametrize(
+    "param, match",
+    [
+        ({"n_components": 0}, "Number of components must be a positive integer"),
+        ({"max_iter": -1}, "Maximum number of iterations must be a positive integer"),
+        ({"tol": -1}, "Tolerance for stopping criteria must be positive"),
+        ({"init": "wrong"}, "Invalid init parameter"),
+        ({"beta_loss": "wrong"}, "Invalid beta_loss parameter"),
+    ],
+)
+@pytest.mark.parametrize("Estimator", [NMF, MiniBatchNMF])
+def test_nmf_common_wrong_params(Estimator, param, match):
+    # Check that appropriate errors are raised for invalid values of parameters common
+    # to NMF and MiniBatchNMF.
+    A = np.ones((2, 2))
+    with pytest.raises(ValueError, match=match):
+        Estimator(**param).fit(A)
+
+
+@pytest.mark.parametrize(
+    "param, match",
+    [
+        ({"solver": "wrong"}, "Invalid solver parameter"),
+    ],
+)
+def test_nmf_wrong_params(param, match):
+    # Check that appropriate errors are raised for invalid values specific to NMF
+    # parameters
+    A = np.ones((2, 2))
+    with pytest.raises(ValueError, match=match):
+        NMF(**param).fit(A)
+
+
+@pytest.mark.parametrize(
+    "param, match",
+    [
+        ({"batch_size": 0}, "batch_size must be a positive integer"),
+    ],
+)
+def test_minibatch_nmf_wrong_params(param, match):
+    # Check that appropriate errors are raised for invalid values specific to
+    # MiniBatchNMF parameters
+    A = np.ones((2, 2))
+    with pytest.raises(ValueError, match=match):
+        MiniBatchNMF(**param).fit(A)
+
+
 def test_initialize_close():
     # Test NNDSVD error
     # Test that _initialize_nmf error is less than the standard deviation of
@@ -110,42 +159,97 @@ def test_initialize_variants():

 # ignore UserWarning raised when both solver='mu' and init='nndsvd'
 @ignore_warnings(category=UserWarning)
-@pytest.mark.parametrize("solver", ("cd", "mu"))
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
 @pytest.mark.parametrize("init", (None, "nndsvd", "nndsvda", "nndsvdar", "random"))
 @pytest.mark.parametrize("alpha_W", (0.0, 1.0))
 @pytest.mark.parametrize("alpha_H", (0.0, 1.0, "same"))
-def test_nmf_fit_nn_output(solver, init, alpha_W, alpha_H):
+def test_nmf_fit_nn_output(Estimator, solver, init, alpha_W, alpha_H):
     # Test that the decomposition does not contain negative values
     A = np.c_[5.0 - np.arange(1, 6), 5.0 + np.arange(1, 6)]
-    model = NMF(
+    model = Estimator(
         n_components=2,
-        solver=solver,
         init=init,
         alpha_W=alpha_W,
         alpha_H=alpha_H,
         random_state=0,
+        **solver,
     )
     transf = model.fit_transform(A)
     assert not ((model.components_ < 0).any() or (transf < 0).any())

-@pytest.mark.parametrize("solver", ("cd", "mu"))
-def test_nmf_fit_close(solver):
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_fit_close(Estimator, solver):
     rng = np.random.mtrand.RandomState(42)
     # Test that the fit is not too far away
-    pnmf = NMF(
+    pnmf = Estimator(
         5,
-        solver=solver,
         init="nndsvdar",
         random_state=0,
         max_iter=600,
+        **solver,
     )
     X = np.abs(rng.randn(6, 5))
     assert pnmf.fit(X).reconstruction_err_ < 0.1

-@pytest.mark.parametrize("solver", ("cd", "mu"))
+def test_nmf_true_reconstruction():
+    # Test that the fit is not too far away from an exact solution
+    # (by construction)
+    n_samples = 15
+    n_features = 10
+    n_components = 5
+    beta_loss = 1
+    batch_size = 3
+    max_iter = 1000
+
+    rng = np.random.mtrand.RandomState(42)
+    W_true = np.zeros([n_samples, n_components])
+    W_array = np.abs(rng.randn(n_samples))
+    for j in range(n_components):
+        W_true[j % n_samples, j] = W_array[j % n_samples]
+    H_true = np.zeros([n_components, n_features])
+    H_array = np.abs(rng.randn(n_components))
+    for j in range(n_features):
+        H_true[j % n_components, j] = H_array[j % n_components]
+    X = np.dot(W_true, H_true)
+
+    model = NMF(
+        n_components=n_components,
+        solver="mu",
+        beta_loss=beta_loss,
+        max_iter=max_iter,
+        random_state=0,
+    )
+    transf = model.fit_transform(X)
+    X_calc = np.dot(transf, model.components_)
+
+    assert model.reconstruction_err_ < 0.1
+    assert_allclose(X, X_calc)
+
+    mbmodel = MiniBatchNMF(
+        n_components=n_components,
+        beta_loss=beta_loss,
+        batch_size=batch_size,
+        random_state=0,
+        max_iter=max_iter,
+    )
+    transf = mbmodel.fit_transform(X)
+    X_calc = np.dot(transf, mbmodel.components_)
+
+    assert mbmodel.reconstruction_err_ < 0.1
+    assert_allclose(X, X_calc, atol=1)
+
+
+@pytest.mark.parametrize("solver", ["cd", "mu"])
 def test_nmf_transform(solver):
+    # Test that fit_transform is equivalent to fit.transform for NMF
     # Test that NMF.transform returns close values
     rng = np.random.mtrand.RandomState(42)
     A = np.abs(rng.randn(6, 5))
@@ -154,14 +258,34 @@ def test_nmf_transform(solver):
         n_components=3,
         init="random",
         random_state=0,
-        tol=1e-5,
+        tol=1e-6,
+    )
+    ft = m.fit_transform(A)
+    t = m.transform(A)
+    assert_allclose(ft, t, atol=1e-1)
+
+
+def test_minibatch_nmf_transform():
+    # Test that fit_transform is equivalent to fit.transform for MiniBatchNMF
+    # Only guaranteed with fresh restarts
+    rng = np.random.mtrand.RandomState(42)
+    A = np.abs(rng.randn(6, 5))
+    m = MiniBatchNMF(
+        n_components=3,
+        random_state=0,
+        tol=1e-3,
+        fresh_restarts=True,
     )
     ft = m.fit_transform(A)
     t = m.transform(A)
-    assert_array_almost_equal(ft, t, decimal=2)
+    assert_allclose(ft, t)

-def test_nmf_transform_custom_init():
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_transform_custom_init(Estimator, solver):
     # Smoke test that checks if NMF.transform works with custom initialization
     random_state = np.random.RandomState(0)
     A = np.abs(random_state.randn(6, 5))
@@ -170,7 +294,9 @@ def test_nmf_transform_custom_init():
     H_init = np.abs(avg * random_state.randn(n_components, 5))
     W_init = np.abs(avg * random_state.randn(6, n_components))

-    m = NMF(solver="cd", n_components=n_components, init="custom", random_state=0)
+    m = Estimator(
+        n_components=n_components, init="custom", random_state=0, tol=1e-3, **solver
+    )
     m.fit_transform(A, W=W_init, H=H_init)
     m.transform(A)

@@ -192,17 +318,37 @@ def test_nmf_inverse_transform(solver):
     assert_array_almost_equal(A, A_new, decimal=2)

-def test_n_components_greater_n_features():
+def test_mbnmf_inverse_transform():
+    # Test that MiniBatchNMF.transform followed by MiniBatchNMF.inverse_transform
+    # is close to the identity
+    rng = np.random.RandomState(0)
+    A = np.abs(rng.randn(6, 4))
+    nmf = MiniBatchNMF(
+        random_state=rng,
+        max_iter=500,
+        init="nndsvdar",
+        fresh_restarts=True,
+    )
+    ft = nmf.fit_transform(A)
+    A_new = nmf.inverse_transform(ft)
+    assert_allclose(A, A_new, rtol=1e-3, atol=1e-2)
+
+
+@pytest.mark.parametrize("Estimator", [NMF, MiniBatchNMF])
+def test_n_components_greater_n_features(Estimator):
     # Smoke test for the case of more components than features.
     rng = np.random.mtrand.RandomState(42)
     A = np.abs(rng.randn(30, 10))
-    NMF(n_components=15, random_state=0, tol=1e-2).fit(A)
+    Estimator(n_components=15, random_state=0, tol=1e-2).fit(A)

-@pytest.mark.parametrize("solver", ["cd", "mu"])
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
 @pytest.mark.parametrize("alpha_W", (0.0, 1.0))
 @pytest.mark.parametrize("alpha_H", (0.0, 1.0, "same"))
-def test_nmf_sparse_input(solver, alpha_W, alpha_H):
+def test_nmf_sparse_input(Estimator, solver, alpha_W, alpha_H):
     # Test that sparse matrices are accepted as input
     from scipy.sparse import csc_matrix

@@ -211,14 +357,15 @@ def test_nmf_sparse_input(solver, alpha_W, alpha_H):
     A[:, 2 * np.arange(5)] = 0
     A_sparse = csc_matrix(A)

-    est1 = NMF(
-        solver=solver,
+    est1 = Estimator(
         n_components=5,
         init="random",
         alpha_W=alpha_W,
         alpha_H=alpha_H,
         random_state=0,
-        tol=1e-2,
+        tol=0,
+        max_iter=100,
+        **solver,
     )
     est2 = clone(est1)

@@ -227,24 +374,25 @@ def test_nmf_sparse_input(solver, alpha_W, alpha_H):
     H1 = est1.components_
     H2 = est2.components_

-    assert_array_almost_equal(W1, W2)
-    assert_array_almost_equal(H1, H2)
+    assert_allclose(W1, W2)
+    assert_allclose(H1, H2)

-def test_nmf_sparse_transform():
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_sparse_transform(Estimator, solver):
     # Test that transform works on sparse data.  Issue #2124
     rng = np.random.mtrand.RandomState(42)
     A = np.abs(rng.randn(3, 2))
     A[1, 1] = 0
     A = csc_matrix(A)

-    for solver in ("cd", "mu"):
-        model = NMF(
-            solver=solver, random_state=0, n_components=2, max_iter=400, init="nndsvd"
-        )
-        A_fit_tr = model.fit_transform(A)
-        A_tr = model.transform(A)
-        assert_array_almost_equal(A_fit_tr, A_tr, decimal=1)
+    model = Estimator(random_state=0, n_components=2, max_iter=400, **solver)
+    A_fit_tr = model.fit_transform(A)
+    A_tr = model.transform(A)
+    assert_allclose(A_fit_tr, A_tr, atol=1e-1)

 @pytest.mark.parametrize("init", ["random", "nndsvd"])
@@ -254,6 +402,7 @@ def test_nmf_sparse_transform():
 def test_non_negative_factorization_consistency(init, solver, alpha_W, alpha_H):
     # Test that the function is called in the same way, either directly
     # or through the NMF class
+    max_iter = 500
     rng = np.random.mtrand.RandomState(42)
     A = np.abs(rng.randn(10, 10))
     A[:, 2 * np.arange(5)] = 0
@@ -262,17 +411,19 @@ def test_non_negative_factorization_consistency(init, solver, alpha_W, alpha_H):
         A,
         init=init,
         solver=solver,
+        max_iter=max_iter,
         alpha_W=alpha_W,
         alpha_H=alpha_H,
         random_state=1,
         tol=1e-2,
     )
-    W_nmf_2, _, _ = non_negative_factorization(
+    W_nmf_2, H, _ = non_negative_factorization(
         A,
         H=H,
         update_H=False,
         init=init,
         solver=solver,
+        max_iter=max_iter,
         alpha_W=alpha_W,
         alpha_H=alpha_H,
         random_state=1,
@@ -282,6 +433,7 @@ def test_non_negative_factorization_consistency(init, solver, alpha_W, alpha_H):
     model_class = NMF(
         init=init,
         solver=solver,
+        max_iter=max_iter,
         alpha_W=alpha_W,
         alpha_H=alpha_H,
         random_state=1,
@@ -290,8 +442,8 @@ def test_non_negative_factorization_consistency(init, solver, alpha_W, alpha_H):
     W_cls = model_class.fit_transform(A)
     W_cls_2 = model_class.transform(A)

-    assert_array_almost_equal(W_nmf, W_cls, decimal=10)
-    assert_array_almost_equal(W_nmf_2, W_cls_2, decimal=10)
+    assert_allclose(W_nmf, W_cls)
+    assert_allclose(W_nmf_2, W_cls_2)

 def test_non_negative_factorization_checking():
@@ -360,7 +512,7 @@ def test_beta_divergence():
     n_samples = 20
     n_features = 10
     n_components = 5
-    beta_losses = [0.0, 0.5, 1.0, 1.5, 2.0]
+    beta_losses = [0.0, 0.5, 1.0, 1.5, 2.0, 3.0]

     # initialization
     rng = np.random.mtrand.RandomState(42)
@@ -458,8 +610,8 @@ def test_nmf_multiplicative_update_sparse():
             random_state=42,
         )

-        assert_array_almost_equal(W1, W2, decimal=7)
-        assert_array_almost_equal(H1, H2, decimal=7)
+        assert_allclose(W1, W2, atol=1e-7)
+        assert_allclose(H1, H2, atol=1e-7)

         # Compare with almost same beta_loss, since some values have a specific
         # behavior, but the results should be continuous w.r.t beta_loss
@@ -480,8 +632,8 @@ def test_nmf_multiplicative_update_sparse():
             random_state=42,
         )

-        assert_array_almost_equal(W1, W3, decimal=4)
-        assert_array_almost_equal(H1, H3, decimal=4)
+        assert_allclose(W1, W3, atol=1e-4)
+        assert_allclose(H1, H3, atol=1e-4)

 def test_nmf_negative_beta_loss():
@@ -520,8 +672,25 @@ def _assert_nmf_no_nan(X, beta_loss):
         _assert_nmf_no_nan(X_csr, beta_loss)

-@pytest.mark.parametrize("solver", ("cd", "mu"))
-def test_nmf_regularization(solver):
+@pytest.mark.parametrize("beta_loss", [-0.5, 0.0])
+def test_minibatch_nmf_negative_beta_loss(beta_loss):
+    """Check that an error is raised if beta_loss < 0 and X contains zeros."""
+    rng = np.random.RandomState(0)
+    X = rng.normal(size=(6, 5))
+    X[X < 0] = 0
+
+    nmf = MiniBatchNMF(beta_loss=beta_loss, random_state=0)
+
+    msg = "When beta_loss <= 0 and X contains zeros, the solver may diverge."
+    with pytest.raises(ValueError, match=msg):
+        nmf.fit(X)
+
+
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_regularization(Estimator, solver):
     # Test the effect of L1 and L2 regularizations
     n_samples = 6
     n_features = 5
@@ -531,20 +700,19 @@ def test_nmf_regularization(solver):

     # L1 regularization should increase the number of zeros
     l1_ratio = 1.0
-
-    regul = nmf.NMF(
+    regul = Estimator(
         n_components=n_components,
-        solver=solver,
         alpha_W=0.5,
         l1_ratio=l1_ratio,
         random_state=42,
+        **solver,
     )
-    model = nmf.NMF(
+    model = Estimator(
         n_components=n_components,
-        solver=solver,
         alpha_W=0.0,
         l1_ratio=l1_ratio,
         random_state=42,
+        **solver,
     )

     W_regul = regul.fit_transform(X)
@@ -553,30 +721,31 @@ def test_nmf_regularization(solver):
     H_regul = regul.components_
     H_model = model.components_

-    W_regul_n_zeros = W_regul[W_regul == 0].size
-    W_model_n_zeros = W_model[W_model == 0].size
-    H_regul_n_zeros = H_regul[H_regul == 0].size
-    H_model_n_zeros = H_model[H_model == 0].size
+    eps = np.finfo(np.float64).eps
+    W_regul_n_zeros = W_regul[W_regul <= eps].size
+    W_model_n_zeros = W_model[W_model <= eps].size
+    H_regul_n_zeros = H_regul[H_regul <= eps].size
+    H_model_n_zeros = H_model[H_model <= eps].size

     assert W_regul_n_zeros > W_model_n_zeros
     assert H_regul_n_zeros > H_model_n_zeros

-    # L2 regularization should decrease the mean of the coefficients
+    # L2 regularization should decrease the sum of the squared norm
+    # of the matrices W and H
     l1_ratio = 0.0
-
-    regul = nmf.NMF(
+    regul = Estimator(
         n_components=n_components,
-        solver=solver,
         alpha_W=0.5,
         l1_ratio=l1_ratio,
         random_state=42,
+        **solver,
     )
-    model = nmf.NMF(
+    model = Estimator(
         n_components=n_components,
-        solver=solver,
         alpha_W=0.0,
         l1_ratio=l1_ratio,
         random_state=42,
+        **solver,
     )

     W_regul = regul.fit_transform(X)
@@ -668,35 +837,40 @@ def test_nmf_underflow():
         (np.int64, np.float64),
     ],
 )
-@pytest.mark.parametrize("solver", ["cd", "mu"])
-@pytest.mark.parametrize("alpha_W", (0.0, 1.0))
-@pytest.mark.parametrize("alpha_H", (0.0, 1.0, "same"))
-def test_nmf_dtype_match(dtype_in, dtype_out, solver, alpha_W, alpha_H):
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_dtype_match(Estimator, solver, dtype_in, dtype_out):
     # Check that NMF preserves dtype (float32 and float64)
     X = np.random.RandomState(0).randn(20, 15).astype(dtype_in, copy=False)
     np.abs(X, out=X)
-    nmf = NMF(solver=solver, alpha_W=alpha_W, alpha_H=alpha_H)
+
+    nmf = Estimator(alpha_W=1.0, alpha_H=1.0, tol=1e-2, random_state=0, **solver)

     assert nmf.fit(X).transform(X).dtype == dtype_out
     assert nmf.fit_transform(X).dtype == dtype_out
     assert nmf.components_.dtype == dtype_out

-@pytest.mark.parametrize("solver", ["cd", "mu"])
-def test_nmf_float32_float64_consistency(solver):
+@pytest.mark.parametrize(
+    ["Estimator", "solver"],
+    [[NMF, {"solver": "cd"}], [NMF, {"solver": "mu"}], [MiniBatchNMF, {}]],
+)
+def test_nmf_float32_float64_consistency(Estimator, solver):
     # Check that the result of NMF is the same between float32 and float64
     X = np.random.RandomState(0).randn(50, 7)
     np.abs(X, out=X)
-
-    nmf32 = NMF(solver=solver, random_state=0)
+    nmf32 = Estimator(random_state=0, tol=1e-3, **solver)
     W32 = nmf32.fit_transform(X.astype(np.float32))
-    nmf64 = NMF(solver=solver, random_state=0)
+    nmf64 = Estimator(random_state=0, tol=1e-3, **solver)
     W64 = nmf64.fit_transform(X)

-    assert_allclose(W32, W64, rtol=1e-6, atol=1e-5)
+    assert_allclose(W32, W64, atol=1e-5)

-def test_nmf_custom_init_dtype_error():
+@pytest.mark.parametrize("Estimator", [NMF, MiniBatchNMF])
+def test_nmf_custom_init_dtype_error(Estimator):
     # Check that an error is raise if custom H and/or W don't have the same
     # dtype as X.
     rng = np.random.RandomState(0)
@@ -705,12 +879,75 @@ def test_nmf_custom_init_dtype_error():
     W = rng.random_sample((20, 15))

     with pytest.raises(TypeError, match="should have the same dtype as X"):
-        NMF(init="custom").fit(X, H=H, W=W)
+        Estimator(init="custom").fit(X, H=H, W=W)

     with pytest.raises(TypeError, match="should have the same dtype as X"):
         non_negative_factorization(X, H=H, update_H=False)

+@pytest.mark.parametrize("beta_loss", [-0.5, 0, 0.5, 1, 1.5, 2, 2.5])
+def test_nmf_minibatchnmf_equivalence(beta_loss):
+    # Test that MiniBatchNMF is equivalent to NMF when batch_size = n_samples and
+    # forget_factor 0.0 (stopping criterion put aside)
+    rng = np.random.mtrand.RandomState(42)
+    X = np.abs(rng.randn(48, 5))
+
+    nmf = NMF(
+        n_components=5,
+        beta_loss=beta_loss,
+        solver="mu",
+        random_state=0,
+        tol=0,
+    )
+    mbnmf = MiniBatchNMF(
+        n_components=5,
+        beta_loss=beta_loss,
+        random_state=0,
+        tol=0,
+        max_no_improvement=None,
+        batch_size=X.shape[0],
+        forget_factor=0.0,
+    )
+    W = nmf.fit_transform(X)
+    mbW = mbnmf.fit_transform(X)
+    assert_allclose(W, mbW)
+
+
+def test_minibatch_nmf_partial_fit():
+    # Check fit / partial_fit equivalence. Applicable only with fresh restarts.
+    rng = np.random.mtrand.RandomState(42)
+    X = np.abs(rng.randn(100, 5))
+
+    n_components = 5
+    batch_size = 10
+    max_iter = 2
+
+    mbnmf1 = MiniBatchNMF(
+        n_components=n_components,
+        init="custom",
+        random_state=0,
+        max_iter=max_iter,
+        batch_size=batch_size,
+        tol=0,
+        max_no_improvement=None,
+        fresh_restarts=False,
+    )
+    mbnmf2 = MiniBatchNMF(n_components=n_components, init="custom", random_state=0)
+
+    # Force the same init of H (W is recomputed anyway) to be able to compare results.
+    W, H = nmf._initialize_nmf(
+        X, n_components=n_components, init="random", random_state=0
+    )
+
+    mbnmf1.fit(X, W=W, H=H)
+    for i in range(max_iter):
+        for j in range(batch_size):
+            mbnmf2.partial_fit(X[j : j + batch_size], W=W[:batch_size], H=H)
+
+    assert mbnmf1.n_steps_ == mbnmf2.n_steps_
+    assert_allclose(mbnmf1.components_, mbnmf2.components_)
+
+
 def test_feature_names_out():
     """Check feature names out for NMF."""
     random_state = np.random.RandomState(0)
@@ -719,3 +956,15 @@ def test_feature_names_out():

     names = nmf.get_feature_names_out()
     assert_array_equal([f"nmf{i}" for i in range(3)], names)
+
+
+def test_minibatch_nmf_verbose():
+    # Check verbose mode of MiniBatchNMF for better coverage.
+    A = np.random.RandomState(0).random_sample((100, 10))
+    nmf = MiniBatchNMF(tol=1e-2, random_state=0, verbose=1)
+    old_stdout = sys.stdout
+    sys.stdout = StringIO()
+    try:
+        nmf.fit(A)
+    finally:
+        sys.stdout = old_stdout

Online implementation of Non-negative Matrix Factorizarion (NMF)
<!--
If your issue is a usage question, submit it here instead:
- StackOverflow with the scikit-learn tag: https://stackoverflow.com/questions/tagged/scikit-learn
- Mailing List: https://mail.python.org/mailman/listinfo/scikit-learn
For more information, see User Questions: http://scikit-learn.org/stable/support.html#user-questions
-->

<!-- Instructions For Filing a Bug: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#filing-bugs -->

#### Description
<!-- Example: Joblib Error thrown when calling fit on LatentDirichletAllocation with evaluate_every > 0-->
@GaelVaroquaux Maybe an online version of the NMF could be useful in big data settings.

#### Steps/Code to Reproduce
<!--
Example:
```python
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.decomposition import LatentDirichletAllocation

docs = ["Help I have a bug" for i in range(1000)]

vectorizer = CountVectorizer(input=docs, analyzer='word')
lda_features = vectorizer.fit_transform(docs)

lda_model = LatentDirichletAllocation(
    n_topics=10,
    learning_method='online',
    evaluate_every=10,
    n_jobs=4,
)
model = lda_model.fit(lda_features)
```
If the code is too long, feel free to put it in a public gist and link
it in the issue: https://gist.github.com
-->

#### Expected Results
<!-- Example: No error is thrown. Please paste or describe the expected results.-->

#### Actual Results
<!-- Please paste or specifically describe the actual output or traceback. -->
Here I show the results for an sparse matrix as input of size 1M x 2.8k
The factorization has `n_components = 10` and each point is an entire pass on the data.
The current online implementation only supports the kullback-leibler divergence, but in practice
can be generalized to any beta divergence.

![batch_vs_online_nmf](https://user-images.githubusercontent.com/11759948/53494592-69748b00-3a9e-11e9-8cc8-ce53ce3fe7f2.png)

#### Versions
<!--
Please run the following snippet and paste the output below.
For scikit-learn >= 0.20:
import sklearn; sklearn.show_versions()
For scikit-learn < 0.20:
import platform; print(platform.platform())
import sys; print("Python", sys.version)
import numpy; print("NumPy", numpy.__version__)
import scipy; print("SciPy", scipy.__version__)
import sklearn; print("Scikit-Learn", sklearn.__version__)
-->

<!-- Thanks for contributing! -->

Draft implementation of MiniBatchNMF
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->

#### What does this implement/fix? Explain your changes.

Implementation of a mini-batch NMF with multiplicative updates.
#13308
#### Any other comments?

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->

Nice figure! Let me see with the others if this should go in.
Could you run on a wikipedia text example?
Here is a benchmark on word counts for the first paragraph of 500k wikipedia articles. To speed up calculations, I used a HashingVectorizer with 4096 features. The KL divergence is calculated on 50k test samples.

![online_nmf_benchmark_wiki](https://user-images.githubusercontent.com/11759948/53506787-fc212400-3ab6-11e9-8bf6-6267c9faadce.png)

Can you do a draft PR just adding the code to scikit-learn, so that people can have a quick look at how complicated the code would be.

Don't worry about test, documentation, or full API compatibility for now. Just add the object (which I think should be called MiniBatchNMF).

would be nice to see more full benchmarks but I think in general this would be useful.
> would be nice to see more full benchmarks but I think in general this would be useful.

Alright. Let's work on this!

@amueller, any specific benchmark that you have in mind?

vary n_features and n_samples?
@GaelVaroquaux , @amueller I have synchronized with master the code proposed by @pcerda in #13386. I was unable to access the data set used in the [previous comment](https://github.com/scikit-learn/scikit-learn/issues/13308#issuecomment-467937672), I used the "[Blog Authorship Corpus](http://u.cs.biu.ac.il/~koppel/BlogCorpus.htm)" then.
The attached plot benchmarks n_features, n_samples and n_components with the scikit-learn implemented version of Non-Negative Matrix Factorization (NMF) and the online one proposed by @pcerda.

![bench_topics](https://user-images.githubusercontent.com/1662261/78696445-b95b8d80-78ff-11ea-8953-a8ef66a262ac.png)

The code used to produce the plot is a variation on the [topic extraction example](https://scikit-learn.org/stable/auto_examples/applications/plot_topics_extraction_with_nmf_lda.html#sphx-glr-auto-examples-applications-plot-topics-extraction-with-nmf-lda-py). It is available [here](https://github.com/cmarmo/scikit-learn/blob/modified_nmf_for_minibatch/benchmarks/bench_topics_extraction_with_onlinenmf.py).
As a reminder, the paper to be referred to is [_Online algorithms for nonnegative matrix factorization with the Itakura-Saito divergence_, A Lefevre, F Bach, C Févotte, 2011](https://arxiv.org/pdf/1106.4198v1.pdf).

Comments?
Thanks!
The speed up is impressive. Thank you for highlighting it.
@cmarmo : can you push a PR with the new code. So that we can review and iterate on this.

Thanks!!
> @cmarmo : can you push a PR with the new code. So that we can review and iterate on this.

I'm on it: just trying to get rid of the `NMFOriginal.py` file. Patricio kept it in order to guarantee the comparison with the original implementation, I'm trying to reproduce the original behavior in the same code. I can open a WIP PR if you think it's worth it, but I would rather if you wait before reviewing the code.  Thanks!

> I can open a WIP PR if you think it's worth it,

I think that it's worth it.

> but I would rather if you wait before reviewing the code.

It's likely that I won't have time in the short term. I see this PR more
as a communication tool across the team.

@cmarmo in the benchmarks you made above, you display the convergence time. What is the convergence criterion and do both versions converge to a solution of similar quality ?
@jeremiedbb I'm working to add this information to the plot. Thanks for your comment.
Hi @jeremiedbb and thanks for your patience, my plot behaved fuzzy probably because someone else was heavyly computing on the machine at the same time... waiting for the new version let me just say that, if I understand correctly, the convergence is computed here https://github.com/scikit-learn/scikit-learn/blob/5af23c93e325c35df8ff3f7d3733fe70a9101268/sklearn/decomposition/_nmf.py#L860 using the kullback-leibler divergence as beta_divergence to define the error between the original matrix and its decomposition. My impression is that the minibatch implementation has a lower divergence at the convergence time, but I need to build reliable plots to confirm.
@jeremiedbb this new plot and my previous comment, are they answering your question?
![bench_topics](https://user-images.githubusercontent.com/1662261/80928545-92309880-8da5-11ea-810e-7e3e2cfd9dbe.png)

So in all your benchmarks, online NMF converges faster to a better solution. Pretty convincing !
As I'm starting to play with the source, for the sake of reproducibility [this plot](https://user-images.githubusercontent.com/1662261/78696445-b95b8d80-78ff-11ea-8953-a8ef66a262ac.png) and [this one](https://user-images.githubusercontent.com/1662261/80928545-92309880-8da5-11ea-810e-7e3e2cfd9dbe.png) have been produced with [this version](https://github.com/scikit-learn/scikit-learn/pull/16948/commits/c74e96a401fe78e3640f1082b69dc3afeb9233b5) of the nmf code.
In the paper the mini batch algorithms is actually not that much faster surprisingly. That's kind of odd.
Generally adding a new class for something with 70 citation is not usually something we do.
The paper is the following:

Online algorithms for nonnegative matrix factorization with the Itakura-Saito divergence, A Lefevre, F Bach, C Févotte

Can you also start working on tests.
@pcerda : can you add your benchmarking code. We'll need it to profile / benchmark.
What's the relationship between this PR and #13386?
I think that https://github.com/scikit-learn/scikit-learn/pull/13386 is a second design that we thought was cleaner. @pcerda, can you confirm.

We'll need to finish this, by the way :). Maybe @twsthomas will be able to pitch in.

Waiting for Reviewer
module:decomposition