Answers.Body: <p>There are a few possible approaches to deploying a ML model to a microcontroller.</p>
  <p>The main limiting factor to deployment on microcontollers is that ML models are usually a representation of a set of <em>parameters</em> that are intended to be used as input to a <em>prediction algorithm</em> alongside a new datapoint. Most such models assume the presence of an accompanying <em>library</em> that implements the algorithm in question. However, a microcrontoller may use an exotic chip architecture, or have very several or unusual resource constraints that prevent these standard libraries from being deployed easily.</p>
  <p>Presumably you will already have some way to get input into your microcontroller and to program it in order to call some function that you can write. If not, you will need to first figure out how to do that, and the right methods depend on your microcontroller. A common approach is to write assembly code or code in a very limited subset of C or another language. An alternative is to find a distribution of an interpreter for another language (e.g. Java, Python) that has been compiled to work on your chip. Either way, you will need some way to program the chip.</p>
  <p>Presuming you can program the chip, you have two fundamental challenges in deploying the model:</p>
  <ol>
  <li><p>Most models are trained with very wide floating point numbers for their parameters. For example, 128-bit numbers may be used. On a standard computing environment, the CPU or GPU will be equipped to perform operations on wide datatypes efficiently. On a microcontroller, you may be limited to 8-bit or 16-bit integers. To work with your model parameters in an environment like this, you will need to either make the parameters smaller (usually by rounding them to fit in a much smaller numeric format, a process called &quot;quantization&quot;), or by finding or writing software that can simulate the operations you want (probably addition and multiplication) on a large datatype that is represented as a collection of smaller datatypes. The first approach may make the model perform poorly. The second may make model prediction very slow.</p>
  </li>
  <li><p>You need an implementation of the algorithm. Some algorithms like linear regression, linear discriminant analysis, or even decision trees are extremely easy to implement prediction for, and may require only addition, multiplication, and/or comparison. You might be able to write these yourself in a simple subset of C, or even in assembly (for example, prediction with linear regression should be just a simple loop). Other algorithms, like deep neural networks, may contain more complex operations, and may contain many such operations performed in complicated sequences. For these, you generally will need to find an distribution of a library that implements the algorithms, or compile one yourself. Compiling one yourself will require setting up a build toolchain for your specific microcontroller, and can be quite involved.</p>
  </li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T00:30:58.750Z
Answers.LastActivityDate: 2021-01-17T00:30:58.750Z
Answers.UserDisplayName: John Doucette
Answers.Body: <p>I found:</p>
  <ul>
  <li>For scikit-learn like models: <a href="https://github.com/eloquentarduino/micromlgen" rel="nofollow noreferrer">MicroML</a>, <a href="https://github.com/Edge-Learning-Machine/Micro-LM" rel="nofollow noreferrer">Micro-LM</a>, <a href="https://github.com/adarsh1001/micro-learn" rel="nofollow noreferrer">Micro Learn</a>, <a href="https://github.com/nok/sklearn-porter" rel="nofollow noreferrer">sklearn-porter</a>, <a href="https://github.com/emlearn/emlearn" rel="nofollow noreferrer">emlearn</a></li>
  <li>For deep learning models: <a href="https://www.tensorflow.org/lite/microcontrollers" rel="nofollow noreferrer">tensorflow Lite Micro</a>, <a href="https://www.st.com/en/embedded-software/x-cube-ai.html" rel="nofollow noreferrer">X-CUBE-AI</a>, <a href="https://medium.com/pytorch/glow-compiler-optimizes-neural-networks-for-low-power-nxp-mcus-e095abe14942" rel="nofollow noreferrer">Glow</a>, <a href="https://github.com/majianjia/nnom" rel="nofollow noreferrer">NNoM</a></li>
  <li>Both: <a href="https://github.com/Microsoft/EdgeML" rel="nofollow noreferrer">EdgeML</a>, <a href="https://github.com/Microsoft/EdgeML" rel="nofollow noreferrer">ELL</a></li>
  </ul>
  <p>These seems to partly fit my needs. But i am surprised that i cannot find something more general that either convert Python to C or to object file with ML support (to be used in C projects). Indeed, once trained, ML algo are &quot;just&quot; a bunch of additions/multiplications/comparisons.</p>
  <p>It would help to be able to convert scikit-learn pipeline for instance, as ML projects are rarely composed of only a single ML model taking raw data. But it seems that EdgeAI/TinyML is mainly focused on compiling deep learning models when it comes to deploy ML models on &quot;bare-metal&quot; microcontrollers.</p>
  <p>Thanks for your answers btw, it helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-12-30T23:13:33.193Z
Answers.Text: Is there any particular preprocessing / feature extraction things you have in mind, apart from the model itself?
Answers.UserDisplayName: Jon Nordby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T12:45:34.050Z
Answers.Text: say the signal comes online. The pipeline could be: the signal is acqquired for n samples (windowed) and some features are extracted from it (mean, var, and other dedicated ones). An ACP can then be used to reduce the dimentionality, and then the reduced vector is fed to a classifier. This is a classic approach actually, but i did not find a simple way to covert the whole pipeline (at the time. But i did not search recently). But i will need to go back to it in the coming weeks.
Answers.UserDisplayName: Hattori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T14:54:52.797Z
Answers.Text: Yes the overlapped windows and feature calculation is something one almost always have to do with time-series. However this is not entirely standardized even in Python, tends to be custom per project. But I am interested in implementing this in emlearn, with C implementations tailored for microcontroller use (streaming, low-memory, no dynamic allocations).
Answers.UserDisplayName: Jon Nordby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-04T14:56:26.917Z
Answers.Text: What is ACP? Did you mean PCA dimensionality reduction?
Answers.UserDisplayName: Jon Nordby
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-01-05T19:58:15.257Z
Answers.Text: oops indeed, PCA ! ACP is the french name. I will further explore what is possible to do in the coming weeks, and come back here if anything relevant to add.
Answers.UserDisplayName: Hattori
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-18T12:40:28.337Z
Answers.LastActivityDate: 2021-01-27T20:39:00.100Z
Answers.LastEditDate: 2021-01-27T20:39:00.100Z
Answers.UserDisplayName: Hattori
Answers.Body: <p>If the library running the model can be compiled for your microcontroller,
  then you can run your model on that microcontroller.</p>
  <p>If you train using one library and deploy using another library, you possible can convert your model to that library: <a href="https://en.wikipedia.org/wiki/Open_Neural_Network_Exchange" rel="nofollow noreferrer">ONNX</a>.</p>
  <p>Some library links on <a href="https://en.wikipedia.org/wiki/Edge_computing" rel="nofollow noreferrer">Edge Computing</a> in <a href="https://en.wikipedia.org/wiki/Machine_learning" rel="nofollow noreferrer">ML</a>:</p>
  <ul>
  <li><a href="https://www.tensorflow.org/lite/microcontrollers?hl=en" rel="nofollow noreferrer">Microcontroller support for Tensorflow Lite</a></li>
  <li><a href="https://github.com/Microsoft/EdgeML" rel="nofollow noreferrer">Edge ML</a></li>
  <li><a href="https://github.com/pytorch/pytorch" rel="nofollow noreferrer">PyTorch</a> supports android</li>
  <li><a href="https://github.com/scikit-learn/scikit-learn" rel="nofollow noreferrer">scikit-learn</a> uses only python libraries. <a href="https://github.com/v923z/micropython-ulab" rel="nofollow noreferrer">ulab</a> partly replaces numpy for  <a href="https://github.com/micropython/micropython" rel="nofollow noreferrer">micropython</a>.</li>
  </ul>
  <p>To speed things up specific hardware is used: GPU's of course, but also <a href="https://en.wikipedia.org/wiki/Tensor_Processing_Unit" rel="nofollow noreferrer">Tensor Processing Units</a>.</p>
  <p>One can manually transfer an NN trained model to C code
  (<a href="https://arxiv.org/pdf/2007.01348.pdf" rel="nofollow noreferrer">paper</a>),
  but there are also compilers:</p>
  <ul>
  <li><a href="https://www.tensorflow.org/xla?hl=en" rel="nofollow noreferrer">XLA</a> of tensorflow</li>
  <li><a href="https://github.com/pytorch/glow" rel="nofollow noreferrer">Glow</a> of pytorch</li>
  <li><a href="https://github.com/iml130/nncg" rel="nofollow noreferrer">NNCG</a></li>
  </ul>
  <p>XLA and Glow support x86-64 and ARM64.
  The NNCG approach generates C code and thus should be more readily portable to general MCU's (<a href="https://arxiv.org/pdf/2001.05572.pdf" rel="nofollow noreferrer">paper</a>).</p>
  <p>Further links: <a href="https://github.com/crespum/edge-ai" rel="nofollow noreferrer">adge-ai</a>,
  <a href="https://stackoverflow.com/questions/36720498/convert-keras-model-to-c">stackoverlow question</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T14:22:52.360Z
Answers.Text: Isn't ONNX only for neural networks? The OP is interested in deploying any ML model (so not just NNs). By the way, welcome to AI SE :)
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T21:57:23.730Z
Answers.Text: ONNX is for NN exchange. ML can be seen as a very broad field. Any C code can do ML. As such a one for all exchange/deploy format would mean quite a bit of coordination and standardization. As the previous answer said, ML models normally are split into two parts: data relative to a library. Every model has its own library. One needs to have that availability on the target platform.
Answers.UserDisplayName: Roland Puntaier
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-17T12:53:23.943Z
Answers.LastActivityDate: 2021-01-18T16:28:29.570Z
Answers.LastEditDate: 2021-01-18T16:28:29.570Z
Answers.UserDisplayName: Roland Puntaier
Answers.Body: <p>One can use simpler approach with <a href="https://github.com/ai-techsystems/deepC/" rel="nofollow noreferrer">deepC compiler</a> and convert exported onnx model to c++.</p>
  <p>Check out simple example at <a href="https://github.com/ai-techsystems/deepC/tree/master/test/compiler/mnist" rel="nofollow noreferrer">deepC compiler sample test</a>
  Compile onnx model for your target machine</p>
  <p>Checkout <a href="https://github.com/ai-techsystems/deepC/blob/master/test/compiler/mnist/mnist.ir" rel="nofollow noreferrer">mnist.ir</a></p>
  <p><strong>Step 1:</strong></p>
  <p>Generate intermediate code</p>
  <pre><code>% onnx2cpp mnist.onnx
  </code></pre>
  <p><strong>Step 2:</strong></p>
  <p>Optimize and compile</p>
  <pre><code>% g++ -O3 mnist.cpp -I ../../../include/ -isystem ../../../packages/eigen-eigen-323c052e1731/ -o mnist.exe
  </code></pre>
  <p><strong>Step 3:</strong></p>
  <p>Test run</p>
  <p>% ./mnist.exe</p>
  <p>Here is a <a href="https://www.youtube.com/watch?v=tzYMq3PJwdw" rel="nofollow noreferrer">link to YouTube Video</a> for elaborate instructions.</p>
  <p>Fur details, read more at  <a href="https://link.medium.com/IhbcJzi4jgb" rel="nofollow noreferrer">https://link.medium.com/IhbcJzi4jgb</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-21T03:02:35.427Z
Answers.LastActivityDate: 2021-05-23T13:47:33.030Z
Answers.LastEditDate: 2021-05-23T13:47:33.030Z
Answers.UserDisplayName: cppRohit
Body: <p>Say I have a machine learning model trained on a laptop and I then want to embed/deploy the model on a microcontroller. How can I do this?</p>
  <p>I know that TensorflowLite Micro generates a C header to be added in the project and then be embedded, but every example I read shows how it is done with neural networks, which seems legit as TensorFlow is primarily used for deep learning.</p>
  <p>But how can I do the same with any type of model, like the ones there is in scikit-learn? So, I'm not interested in necessarily doing this with TensorflowLite Micro, but I'm interested in the general approach to solve this problem.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-18T11:43:13.583Z
Comments.Text: Please, rather than editing your post to add an answer to your own question, add a formal answer below and provide the relevant details from that blog post in it.
Comments.UserDisplayName: nbro
Tags: <machine-learning><tensorflow><models>
Title: How to embed/deploy an arbitrary machine learning model on microcontrollers?
UserDisplayName: Hattori
ThreadText: # How to embed/deploy an arbitrary machine learning model on microcontrollers?
  
  (Asked by: username_0 on 2021-01-14)
  
  Say I have a machine learning model trained on a laptop and I then want to embed/deploy the model on a microcontroller. How can I do this?
  
  I know that TensorflowLite Micro generates a C header to be added in the project and then be embedded, but every example I read shows how it is done with neural networks, which seems legit as TensorFlow is primarily used for deep learning.
  
  But how can I do the same with any type of model, like the ones there is in scikit-learn? So, I'm not interested in necessarily doing this with TensorflowLite Micro, but I'm interested in the general approach to solve this problem.
  
  **Tags:** machine-learning, tensorflow, models
  
  > **Comments:**
  > **username_1**: Please, rather than editing your post to add an answer to your own question, add a formal answer below and provide the relevant details from that blog post in it.  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2021-01-17. Score: 6
  
  There are a few possible approaches to deploying a ML model to a microcontroller.
  
  The main limiting factor to deployment on microcontollers is that ML models are usually a representation of a set of *parameters* that are intended to be used as input to a *prediction algorithm* alongside a new datapoint. Most such models assume the presence of an accompanying *library* that implements the algorithm in question. However, a microcrontoller may use an exotic chip architecture, or have very several or unusual resource constraints that prevent these standard libraries from being deployed easily.
  
  Presumably you will already have some way to get input into your microcontroller and to program it in order to call some function that you can write. If not, you will need to first figure out how to do that, and the right methods depend on your microcontroller. A common approach is to write assembly code or code in a very limited subset of C or another language. An alternative is to find a distribution of an interpreter for another language (e.g. Java, Python) that has been compiled to work on your chip. Either way, you will need some way to program the chip.
  
  Presuming you can program the chip, you have two fundamental challenges in deploying the model:
  
  1. Most models are trained with very wide floating point numbers for their parameters. For example, 128-bit numbers may be used. On a standard computing environment, the CPU or GPU will be equipped to perform operations on wide datatypes efficiently. On a microcontroller, you may be limited to 8-bit or 16-bit integers. To work with your model parameters in an environment like this, you will need to either make the parameters smaller (usually by rounding them to fit in a much smaller numeric format, a process called "quantization"), or by finding or writing software that can simulate the operations you want (probably addition and multiplication) on a large datatype that is represented as a collection of smaller datatypes. The first approach may make the model perform poorly. The second may make model prediction very slow.
  2. You need an implementation of the algorithm. Some algorithms like linear regression, linear discriminant analysis, or even decision trees are extremely easy to implement prediction for, and may require only addition, multiplication, and/or comparison. You might be able to write these yourself in a simple subset of C, or even in assembly (for example, prediction with linear regression should be just a simple loop). Other algorithms, like deep neural networks, may contain more complex operations, and may contain many such operations performed in complicated sequences. For these, you generally will need to find an distribution of a library that implements the algorithms, or compile one yourself. Compiling one yourself will require setting up a build toolchain for your specific microcontroller, and can be quite involved.
  
  
  ---
  
  ### Answer by username_0 on 2021-01-18. Score: 4
  
  I found:
  
  * For scikit-learn like models: [MicroML](https://github.com/eloquentarduino/micromlgen), [Micro-LM](https://github.com/Edge-Learning-Machine/Micro-LM), [Micro Learn](https://github.com/adarsh1001/micro-learn), [sklearn-porter](https://github.com/nok/sklearn-porter), [emlearn](https://github.com/emlearn/emlearn)
  * For deep learning models: [tensorflow Lite Micro](https://www.tensorflow.org/lite/microcontrollers), [X-CUBE-AI](https://www.st.com/en/embedded-software/x-cube-ai.html), [Glow](https://medium.com/pytorch/glow-compiler-optimizes-neural-networks-for-low-power-nxp-mcus-e095abe14942), [NNoM](https://github.com/majianjia/nnom)
  * Both: [EdgeML](https://github.com/Microsoft/EdgeML), [ELL](https://github.com/Microsoft/EdgeML)
  
  These seems to partly fit my needs. But i am surprised that i cannot find something more general that either convert Python to C or to object file with ML support (to be used in C projects). Indeed, once trained, ML algo are "just" a bunch of additions/multiplications/comparisons.
  
  It would help to be able to convert scikit-learn pipeline for instance, as ML projects are rarely composed of only a single ML model taking raw data. But it seems that EdgeAI/TinyML is mainly focused on compiling deep learning models when it comes to deploy ML models on "bare-metal" microcontrollers.
  
  Thanks for your answers btw, it helps.
  
  > **Comments:**
  > **username_3**: Is there any particular preprocessing / feature extraction things you have in mind, apart from the model itself?  
  > **username_0**: say the signal comes online. The pipeline could be: the signal is acqquired for n samples (windowed) and some features are extracted from it (mean, var, and other dedicated ones). An ACP can then be used to reduce the dimentionality, and then the reduced vector is fed to a classifier. This is a classic approach actually, but i did not find a simple way to covert the whole pipeline (at the time. But i did not search recently). But i will need to go back to it in the coming weeks.  
  > **username_3**: Yes the overlapped windows and feature calculation is something one almost always have to do with time-series. However this is not entirely standardized even in Python, tends to be custom per project. But I am interested in implementing this in emlearn, with C implementations tailored for microcontroller use (streaming, low-memory, no dynamic allocations).  
  > **username_3**: What is ACP? Did you mean PCA dimensionality reduction?  
  > **username_0**: oops indeed, PCA ! ACP is the french name. I will further explore what is possible to do in the coming weeks, and come back here if anything relevant to add.  
  
  ---
  
  ### Answer by username_4 on 2021-01-17. Score: 2
  
  If the library running the model can be compiled for your microcontroller,
  then you can run your model on that microcontroller.
  
  If you train using one library and deploy using another library, you possible can convert your model to that library: [ONNX](https://en.wikipedia.org/wiki/Open_Neural_Network_Exchange).
  
  Some library links on [Edge Computing](https://en.wikipedia.org/wiki/Edge_computing) in [ML](https://en.wikipedia.org/wiki/Machine_learning):
  
  * [Microcontroller support for Tensorflow Lite](https://www.tensorflow.org/lite/microcontrollers?hl=en)
  * [Edge ML](https://github.com/Microsoft/EdgeML)
  * [PyTorch](https://github.com/pytorch/pytorch) supports android
  * [scikit-learn](https://github.com/scikit-learn/scikit-learn) uses only python libraries. [ulab](https://github.com/v923z/micropython-ulab) partly replaces numpy for [micropython](https://github.com/micropython/micropython).
  
  To speed things up specific hardware is used: GPU's of course, but also [Tensor Processing Units](https://en.wikipedia.org/wiki/Tensor_Processing_Unit).
  
  One can manually transfer an NN trained model to C code
  ([paper](https://arxiv.org/pdf/2007.01348.pdf)),
  but there are also compilers:
  
  * [XLA](https://www.tensorflow.org/xla?hl=en) of tensorflow
  * [Glow](https://github.com/pytorch/glow) of pytorch
  * [NNCG](https://github.com/iml130/nncg)
  
  XLA and Glow support x86-64 and ARM64.
  The NNCG approach generates C code and thus should be more readily portable to general MCU's ([paper](https://arxiv.org/pdf/2001.05572.pdf)).
  
  Further links: [adge-ai](https://github.com/crespum/edge-ai),
  [stackoverlow question](https://stackoverflow.com/questions/36720498/convert-keras-model-to-c)
  
  > **Comments:**
  > **username_1**: Isn't ONNX only for neural networks? The OP is interested in deploying any ML model (so not just NNs). By the way, welcome to AI SE :)  
  > **username_4**: ONNX is for NN exchange. ML can be seen as a very broad field. Any C code can do ML. As such a one for all exchange/deploy format would mean quite a bit of coordination and standardization. As the previous answer said, ML models normally are split into two parts: data relative to a library. Every model has its own library. One needs to have that availability on the target platform.  
  
  ---
  
  ### Answer by username_5 on 2021-04-21. Score: 1
  
  One can use simpler approach with [deepC compiler](https://github.com/ai-techsystems/deepC/) and convert exported onnx model to c++.
  
  Check out simple example at [deepC compiler sample test](https://github.com/ai-techsystems/deepC/tree/master/test/compiler/mnist)
  Compile onnx model for your target machine
  
  Checkout [mnist.ir](https://github.com/ai-techsystems/deepC/blob/master/test/compiler/mnist/mnist.ir)
  
  **Step 1:**
  
  Generate intermediate code
  
  ```
  % onnx2cpp mnist.onnx
  
  ```
  
  **Step 2:**
  
  Optimize and compile
  
  ```
  % g++ -O3 mnist.cpp -I ../../../include/ -isystem ../../../packages/eigen-eigen-323c052e1731/ -o mnist.exe
  
  ```
  
  **Step 3:**
  
  Test run
  
  % ./mnist.exe
  
  Here is a [link to YouTube Video](https://www.youtube.com/watch?v=tzYMq3PJwdw) for elaborate instructions.
  
  Fur details, read more at <https://link.medium.com/IhbcJzi4jgb>
  
  
  ---
  
  