Answers.Body: <p>The <a href="https://scikit-learn.org/stable/modules/density.html#kernel-density" rel="nofollow noreferrer"><code>KernelDensity</code></a> model learns a probability distribution from the training data. The score reflects how likely it is that any given sample has been drawn from the learned probability distribution.
  The higher this number is, the more the given sample matches the distribution.</p>
  <p>For example: If the sample is <code>x = 0</code> and your model has learned the standard normal distribution <span class="math-container">$\mathcal{N}(0, 1)$</span>, the likelihood would be very high. In contrast, if <code>x = 50</code> the likelihood is low, because the probability to draw 50 from <span class="math-container">$\mathcal{N}(0, 1)$</span> is very low.</p>
  <p>Therefore: The higher the score, the better the model reflects the samples. And in your case, it seems that the model learns a distribution that fits your data best when the bandwidth is <code>0.2585</code>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-10T15:57:09.417Z
Answers.LastActivityDate: 2022-03-10T15:57:09.417Z
Answers.UserDisplayName: Chillston
Body: <p>For <code>sklearn.neighbors.KernelDensity</code>, its <code>score(X)</code> method according to the <a href="https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html#sklearn.neighbors.KernelDensity.score" rel="nofollow noreferrer">sklearn KDE documentation</a> says:</p>
  <blockquote>
  <p>Compute the log-likelihood of each sample under the model</p>
  </blockquote>
  <p>For 'gaussian' kernel, I have implemented hyper-parameter tuning for the 'bandwidth' parameter using Bayesian-Optimization as follows:</p>
  <pre><code># The input data for which 'bandwidth' needs to be tuned-
  data
  # (2880, 64)
  
  def kde_hyperopt_eval(bandwidth):
      params = {}
      params['bandwidth'] = bandwidth
      
      # Initialize a KDE model-
      kde_model = KernelDensity(
          kernel = 'gaussian',
          bandwidth = params['bandwidth']
      )
      
      # Train KDE model on training data-
      kde_model.fit(data)
      
      # Compute the total log-likelihood under the model.
      # Returns the log probability.
      '''
      Total log-likelihood of the data in X. This is normalized to be a
      probability density, so the value will be low for high-dimensional
      data.
      '''
      return kde_model.score(data)
  
  optimizer = BayesianOptimization(
      f = kde_hyperopt_eval,
      pbounds = {
          'bandwidth': (0.01, 10)
          }
  )
  
  optimizer.maximize(n_iter = 40, init_points = 15)
  </code></pre>
  <p>and I'm getting the result as</p>
  <pre><code>|   iter    |  target   | bandwidth |
  -------------------------------------
  |  1        | -5.644e+0 |  8.527    |
  |  2        |  2.142e+0 |  0.3419   |
  |  3        | -1.287e+0 |  0.7963   |
  |  4        | -5.916e+0 |  9.883    |
  |  5        | -3.604e+0 |  2.817    |
  |  6        | -5.71e+05 |  8.835    |
  |  7        | -5.246e+0 |  6.868    |
  |  8        | -4.385e+0 |  4.305    |
  |  9        | -5.546e+0 |  8.082    |
  |  10       | -5.86e+05 |  9.585    |
  |  11       | -5.226e+0 |  6.794    |
  |  12       | -5.196e+0 |  6.685    |
  |  13       | -9.934e+0 |  0.6771   |
  |  14       | -5.766e+0 |  9.111    |
  |  15       |  6.816e+0 |  0.2584   |
  |  16       |  6.565e+0 |  0.01     |
  |  17       |  6.565e+0 |  0.01     |
  |  18       |  6.804e+0 |  0.2585   |
  |  19       |  6.804e+0 |  0.2585   |
  |  20       |  6.804e+0 |  0.2585   |
  |  21       |  6.804e+0 |  0.2585   |
  |  22       |  6.804e+0 |  0.2585   |
  |  23       |  6.804e+0 |  0.2585   |
  |  24       |  6.804e+0 |  0.2585   |
  |  25       |  6.804e+0 |  0.2585   |
  |  26       |  6.804e+0 |  0.2585   |
  |  27       |  6.804e+0 |  0.2585   |
  |  28       |  6.804e+0 |  0.2585   |
  |  29       |  6.804e+0 |  0.2585   |
  |  30       |  6.804e+0 |  0.2585   |
  |  31       |  6.804e+0 |  0.2585   |
  |  32       |  6.804e+0 |  0.2585   |
  |  33       |  6.804e+0 |  0.2585   |
  |  34       |  6.804e+0 |  0.2585   |
  |  35       |  6.804e+0 |  0.2585   |
  |  36       |  6.804e+0 |  0.2585   |
  |  37       |  6.804e+0 |  0.2585   |
  |  38       |  6.804e+0 |  0.2585   |
  |  39       |  6.804e+0 |  0.2585   |
  |  40       |  6.804e+0 |  0.2585   |
  |  41       |  6.804e+0 |  0.2585   |
  |  42       |  6.804e+0 |  0.2585   |
  |  43       |  6.804e+0 |  0.2585   |
  |  44       |  6.804e+0 |  0.2585   |
  |  45       |  6.804e+0 |  0.2585   |
  |  46       |  6.804e+0 |  0.2585   |
  |  47       |  6.804e+0 |  0.2585   |
  |  48       |  6.804e+0 |  0.2585   |
  |  49       |  6.804e+0 |  0.2585   |
  |  50       |  6.804e+0 |  0.2585   |
  |  51       |  6.804e+0 |  0.2585   |
  |  52       |  6.804e+0 |  0.2585   |
  |  53       |  6.804e+0 |  0.2585   |
  |  54       |  6.804e+0 |  0.2585   |
  |  55       |  6.804e+0 |  0.2585   |
  =====================================
  </code></pre>
  <p>How can I interpret this value returned by <code>score(X)</code> method of <code>sklearn.neighbors.KernelDensity</code>?</p>
  <p>For example, in case of Mean Squared Error or Mean Absolute Error, smaller means better, for accuracy, higher percentage value is better.</p>
  
Tags: <hyperparameter-optimization><scikit-learn><bayesian-optimization><density-estimation>
Title: How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?
UserDisplayName: Arun
ThreadText: # How can I interpret the value returned by score(X) method of sklearn.neighbors.KernelDensity?
  
  (Asked by: username_0 on 2022-03-10)
  
  For `sklearn.neighbors.KernelDensity`, its `score(X)` method according to the [sklearn KDE documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KernelDensity.html#sklearn.neighbors.KernelDensity.score) says:
  
  > Compute the log-likelihood of each sample under the model
  
  For 'gaussian' kernel, I have implemented hyper-parameter tuning for the 'bandwidth' parameter using Bayesian-Optimization as follows:
  
  ```
  # The input data for which 'bandwidth' needs to be tuned-
  data
  # (2880, 64)
  
  def kde_hyperopt_eval(bandwidth):
      params = {}
      params['bandwidth'] = bandwidth
      
      # Initialize a KDE model-
      kde_model = KernelDensity(
          kernel = 'gaussian',
          bandwidth = params['bandwidth']
      )
      
      # Train KDE model on training data-
      kde_model.fit(data)
      
      # Compute the total log-likelihood under the model.
      # Returns the log probability.
      '''
      Total log-likelihood of the data in X. This is normalized to be a
      probability density, so the value will be low for high-dimensional
      data.
      '''
      return kde_model.score(data)
  
  optimizer = BayesianOptimization(
      f = kde_hyperopt_eval,
      pbounds = {
          'bandwidth': (0.01, 10)
          }
  )
  
  optimizer.maximize(n_iter = 40, init_points = 15)
  
  ```
  
  and I'm getting the result as
  
  ```
  |   iter    |  target   | bandwidth |
  -------------------------------------
  |  1        | -5.644e+0 |  8.527    |
  |  2        |  2.142e+0 |  0.3419   |
  |  3        | -1.287e+0 |  0.7963   |
  |  4        | -5.916e+0 |  9.883    |
  |  5        | -3.604e+0 |  2.817    |
  |  6        | -5.71e+05 |  8.835    |
  |  7        | -5.246e+0 |  6.868    |
  |  8        | -4.385e+0 |  4.305    |
  |  9        | -5.546e+0 |  8.082    |
  |  10       | -5.86e+05 |  9.585    |
  |  11       | -5.226e+0 |  6.794    |
  |  12       | -5.196e+0 |  6.685    |
  |  13       | -9.934e+0 |  0.6771   |
  |  14       | -5.766e+0 |  9.111    |
  |  15       |  6.816e+0 |  0.2584   |
  |  16       |  6.565e+0 |  0.01     |
  |  17       |  6.565e+0 |  0.01     |
  |  18       |  6.804e+0 |  0.2585   |
  |  19       |  6.804e+0 |  0.2585   |
  |  20       |  6.804e+0 |  0.2585   |
  |  21       |  6.804e+0 |  0.2585   |
  |  22       |  6.804e+0 |  0.2585   |
  |  23       |  6.804e+0 |  0.2585   |
  |  24       |  6.804e+0 |  0.2585   |
  |  25       |  6.804e+0 |  0.2585   |
  |  26       |  6.804e+0 |  0.2585   |
  |  27       |  6.804e+0 |  0.2585   |
  |  28       |  6.804e+0 |  0.2585   |
  |  29       |  6.804e+0 |  0.2585   |
  |  30       |  6.804e+0 |  0.2585   |
  |  31       |  6.804e+0 |  0.2585   |
  |  32       |  6.804e+0 |  0.2585   |
  |  33       |  6.804e+0 |  0.2585   |
  |  34       |  6.804e+0 |  0.2585   |
  |  35       |  6.804e+0 |  0.2585   |
  |  36       |  6.804e+0 |  0.2585   |
  |  37       |  6.804e+0 |  0.2585   |
  |  38       |  6.804e+0 |  0.2585   |
  |  39       |  6.804e+0 |  0.2585   |
  |  40       |  6.804e+0 |  0.2585   |
  |  41       |  6.804e+0 |  0.2585   |
  |  42       |  6.804e+0 |  0.2585   |
  |  43       |  6.804e+0 |  0.2585   |
  |  44       |  6.804e+0 |  0.2585   |
  |  45       |  6.804e+0 |  0.2585   |
  |  46       |  6.804e+0 |  0.2585   |
  |  47       |  6.804e+0 |  0.2585   |
  |  48       |  6.804e+0 |  0.2585   |
  |  49       |  6.804e+0 |  0.2585   |
  |  50       |  6.804e+0 |  0.2585   |
  |  51       |  6.804e+0 |  0.2585   |
  |  52       |  6.804e+0 |  0.2585   |
  |  53       |  6.804e+0 |  0.2585   |
  |  54       |  6.804e+0 |  0.2585   |
  |  55       |  6.804e+0 |  0.2585   |
  =====================================
  
  ```
  
  How can I interpret this value returned by `score(X)` method of `sklearn.neighbors.KernelDensity`?
  
  For example, in case of Mean Squared Error or Mean Absolute Error, smaller means better, for accuracy, higher percentage value is better.
  
  **Tags:** hyperparameter-optimization, scikit-learn, bayesian-optimization, density-estimation
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-03-10. Score: 2 | Accepted answer
  
  The [`KernelDensity`](https://scikit-learn.org/stable/modules/density.html#kernel-density) model learns a probability distribution from the training data. The score reflects how likely it is that any given sample has been drawn from the learned probability distribution.
  The higher this number is, the more the given sample matches the distribution.
  
  For example: If the sample is `x = 0` and your model has learned the standard normal distribution $\mathcal{N}(0, 1)$, the likelihood would be very high. In contrast, if `x = 50` the likelihood is low, because the probability to draw 50 from $\mathcal{N}(0, 1)$ is very low.
  
  Therefore: The higher the score, the better the model reflects the samples. And in your case, it seems that the model learns a distribution that fits your data best when the bandwidth is `0.2585`.
  
  
  ---
  
  