Answers.Body: <p>8-bit microcontrollers (such as the AVR on the nano) are unlikely to do much in the way of inference.  One could in theory enlist/write a matrix library that worked with 8-bit floats but it's unlikely to be particularly efficient.</p>
  <p>Modern ARM microcontrollers (such as those on the Nano 32/33) with 32-bit cores and with a floating point unit are much more likely candidates.  ARM very helpfully provides the CMSIS NN library to run pre-trained models.  Size of model will be limited compared to desktop/server deployed models but some stuff can be successfully run.  There are bleeding edge microcontrollers with neural processing units (NPUs) at the time of this writing.</p>
  <p>Another option is an FPGA.  That is a whole other skill set and the learning curve is steep but they can be much higher performance than a microcontroller.</p>
  <p>On the largest side of embedded, application cores with Mali GPUs/etc can do a bit more work.  There are USB powered NPUs for processors without a GPU (or powerful enough GPU).  We're even beginning to see application cores with built in NPUs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-28T16:33:20.550Z
Answers.LastActivityDate: 2023-08-28T16:33:20.550Z
Answers.UserDisplayName: foreverska
Body: <p>I want to learn more about EdgeAI or EmbeddedML, whatever term you like best.</p>
  <p>I played with an Arduino Nano, running some of the tutorial projects (motion detection), but I don't think it is up to much.</p>
  <p>If anyone here works in this area, what hardware do you use?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T11:32:45.480Z
Comments.Text: This question is too broad, opinion-based and off-topic. So, it clearly should be closed. Please, see our on-topic page for more info. We focus on the theoretical aspects of AI. This is related to specific hardware that people are using. So, off-topic.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T14:18:53.583Z
Comments.Text: @nbro Apologies, can you point me towards the on-topic page? I cannot seem to locate it. Also, hardware should not be off-topic, as models have to run on something. Just a suggestion: I will aim to keep my questions more to the theoretical aspects going forward.
Comments.UserDisplayName: ofithch79
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-31T15:06:43.917Z
Comments.Text: See https://ai.stackexchange.com/help/on-topic. There's also Data Science SE, where this question may be more suitable or even on-topic.
Comments.UserDisplayName: nbro
Tags: <ai-design><embedded-design>
Title: Hardware for EdgeML/EmbeddedML
UserDisplayName: ofithch79
ThreadText: # Hardware for EdgeML/EmbeddedML
  
  (Asked by: username_0 on 2023-08-28)
  
  I want to learn more about EdgeAI or EmbeddedML, whatever term you like best.
  
  I played with an Arduino Nano, running some of the tutorial projects (motion detection), but I don't think it is up to much.
  
  If anyone here works in this area, what hardware do you use?
  
  **Tags:** ai-design, embedded-design
  
  > **Comments:**
  > **username_1**: This question is too broad, opinion-based and off-topic. So, it clearly should be closed. Please, see our on-topic page for more info. We focus on the theoretical aspects of AI. This is related to specific hardware that people are using. So, off-topic.  
  > **username_0**: @username_1 Apologies, can you point me towards the on-topic page? I cannot seem to locate it. Also, hardware should not be off-topic, as models have to run on something. Just a suggestion: I will aim to keep my questions more to the theoretical aspects going forward.  
  > **username_1**: See https://ai.stackexchange.com/help/on-topic. There's also Data Science SE, where this question may be more suitable or even on-topic.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-08-28. Score: 3 | Accepted answer
  
  8-bit microcontrollers (such as the AVR on the nano) are unlikely to do much in the way of inference. One could in theory enlist/write a matrix library that worked with 8-bit floats but it's unlikely to be particularly efficient.
  
  Modern ARM microcontrollers (such as those on the Nano 32/33) with 32-bit cores and with a floating point unit are much more likely candidates. ARM very helpfully provides the CMSIS NN library to run pre-trained models. Size of model will be limited compared to desktop/server deployed models but some stuff can be successfully run. There are bleeding edge microcontrollers with neural processing units (NPUs) at the time of this writing.
  
  Another option is an FPGA. That is a whole other skill set and the learning curve is steep but they can be much higher performance than a microcontroller.
  
  On the largest side of embedded, application cores with Mali GPUs/etc can do a bit more work. There are USB powered NPUs for processors without a GPU (or powerful enough GPU). We're even beginning to see application cores with built in NPUs.
  
  
  ---
  
  