  Identification of clusters of co-expressed genes in transcriptomic data is a
difficult task. Most algorithms used for this purpose can be classified into
two broad categories: distance-based or model-based approaches. Distance-based
approaches typically utilize a distance function between pairs of data objects
and group similar objects together into clusters. Model-based approaches are
based on using the mixture-modeling framework. Compared to distance-based
approaches, model-based approaches offer better interpretability because each
cluster can be explicitly characterized in terms of the proposed model.
However, these models present a particular difficulty in identifying a correct
multivariate distribution that a mixture can be based upon. In this manuscript,
we review some of the approaches used to select a distribution for the needed
mixture model first. Then, we propose avoiding this problem altogether by using
a nonparametric MSL (Maximum Smoothed Likelihood) algorithm. This algorithm was
proposed earlier in statistical literature but has not been, to the best of our
knowledge, applied to transcriptomics data. The salient feature of this
approach is that it avoids explicit specification of distributions of
individual biological samples altogether, thus making the task of a
practitioner easier. When used on a real dataset, the algorithm produces a
large number of biologically meaningful clusters and compares favorably to the
two other mixture-based algorithms commonly used for RNA-seq data clustering.
Our code is publicly available in Github at
https://github.com/Matematikoi/non_parametric_clustering.

Applications