Answers.Body: <p>Face filters works by first detecting and localizing the face, then predicting the so called <em>facial landmarks</em> (a set of points that depict the geometry of the face, like its contour, shape of eyes, nose, mouth, ecc), and lastly applying the filter potentially yield by some generative model.</p>
  <p>These are all heavy work. For example, you can have a pipeline that first detects a face, if no faces you stop the processing. Otherwise you can have a multi-task model that yields the face bounding-box and landmarks directly, so improving the runtime because you reuse most of the model for both predictions. However, the part related for the filters is probably always separated, moreover you have to deal with computer graphics to render the filter and also apply some nice FX.</p>
  <p>What I want to say is that both solutions are possible, and even a combination of them.</p>
  <ul>
  <li>Many camera software (especially on mobile) have a very fast implementation of face detection (probably an improvement of <a href="https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework" rel="nofollow noreferrer">Viola-Jones</a> method). So once you detect a face, you can send that frame on the cloud for fast processing. Once you have the landmarks and, I guess, also the correct geometry of the filter, to enable real-time effects usually <a href="https://pyimagesearch.com/2018/07/30/opencv-object-tracking/" rel="nofollow noreferrer">tracking</a> is used. There exist very efficient tracking algorithms that rely un classical computer vision, that can either track objects and even landmarks. Basically you run the expensive part (or cloud processing) one frame every <span class="math-container">$N$</span>, and for the other you use only tracking.</li>
  </ul>
  <ul>
  <li>Scaled-down CNN models or GAN, are also possible. There exist optimization frameworks like <a href="https://www.tensorflow.org/lite?hl=en" rel="nofollow noreferrer">tf-lite</a> that allow you to compress (by quantization and pruning), optimize, and even deploy a DL model. Indeed, you need to care about specific optimizations for the CPU and GPU.</li>
  </ul>
  <p>If you plan to implement something similar on your on PC, you can use <a href="https://opencv.org/" rel="nofollow noreferrer">OpenCV</a> (either with Python or C++) for, e.g., tracking, then compress a DL model with tf-lite in case you use tensorflow and/or Keras: you can take a pre-trained model, and compress with such tool. In that way, you can target optimized CPU and GPU models. For the graphics part you should find some optimized library for GPU, or write the code yourself like in OpenGL or similar. You can also have a look at <a href="http://dlib.net/" rel="nofollow noreferrer">Dlib</a>: it's a nice lib, that has many image processing, computer vision, ML implementations, and even pre-trained models.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T14:47:55.250Z
Answers.Text: Thank you Luca. By "one frame every N", I assume you mean "one frame every N number of seconds"? So the tracking algorithms run on the phone to track facial features. Any such algorithm names you remember?
Answers.UserDisplayName: Julian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T17:03:37.277Z
Answers.Text: To achieve real-time processing you should process at least 24 image frames in a second (say up to 60 for best smoothness). It's not necessary to either gather more of them, so you should capture camera images on a fixed interval basis. Assuming that, you do all the heavy work at frame 1, then for the next $N$ frames (e.g. 10) you do tracking ($N$ should be not too large otherwise you loose accuracy). At frame $N+1$ you do the heavy work again, and so on. For tracking you can start from the [Lucas-Kanade method](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html).
Answers.UserDisplayName: Luca Anzalone
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-31T07:52:30.373Z
Answers.LastActivityDate: 2023-05-31T07:52:30.373Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I see many realtime face swap filters and appearance enhancement filters on smartphone apps. Even apps that can make you look like a granny or show you having a frown, no matter what your actual facial expression is.<br />
  On searching for open source algorithms/code to apply such effects to my desktop PC's webcam, all I find are hugely resource-heavy programs that require a dedicated GPU or TPU.<br />
  May I know how smartphone apps process face filters? If they send the data to a server for processing, how does it work in realtime? Or do they use scaled-down CNN models or GAN that can run on the phone's processor? If so, are there lightweight models or algorithms that can run on the desktop PC's CPU or integrated GPU (APU)?</p>
  
Tags: <convolutional-neural-networks><generative-adversarial-networks><filters><face-detection><deep-face>
Title: Which face filter algorithms can work on CPU or integrated GPU?
UserDisplayName: Julian
ThreadText: # Which face filter algorithms can work on CPU or integrated GPU?
  
  (Asked by: username_0 on 2023-05-28)
  
  I see many realtime face swap filters and appearance enhancement filters on smartphone apps. Even apps that can make you look like a granny or show you having a frown, no matter what your actual facial expression is.  
  On searching for open source algorithms/code to apply such effects to my desktop PC's webcam, all I find are hugely resource-heavy programs that require a dedicated GPU or TPU.  
  May I know how smartphone apps process face filters? If they send the data to a server for processing, how does it work in realtime? Or do they use scaled-down CNN models or GAN that can run on the phone's processor? If so, are there lightweight models or algorithms that can run on the desktop PC's CPU or integrated GPU (APU)?
  
  **Tags:** convolutional-neural-networks, generative-adversarial-networks, filters, face-detection, deep-face
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-05-31. Score: 2 | Accepted answer
  
  Face filters works by first detecting and localizing the face, then predicting the so called *facial landmarks* (a set of points that depict the geometry of the face, like its contour, shape of eyes, nose, mouth, ecc), and lastly applying the filter potentially yield by some generative model.
  
  These are all heavy work. For example, you can have a pipeline that first detects a face, if no faces you stop the processing. Otherwise you can have a multi-task model that yields the face bounding-box and landmarks directly, so improving the runtime because you reuse most of the model for both predictions. However, the part related for the filters is probably always separated, moreover you have to deal with computer graphics to render the filter and also apply some nice FX.
  
  What I want to say is that both solutions are possible, and even a combination of them.
  
  * Many camera software (especially on mobile) have a very fast implementation of face detection (probably an improvement of [Viola-Jones](https://en.wikipedia.org/wiki/Viola%E2%80%93Jones_object_detection_framework) method). So once you detect a face, you can send that frame on the cloud for fast processing. Once you have the landmarks and, I guess, also the correct geometry of the filter, to enable real-time effects usually [tracking](https://pyimagesearch.com/2018/07/30/opencv-object-tracking/) is used. There exist very efficient tracking algorithms that rely un classical computer vision, that can either track objects and even landmarks. Basically you run the expensive part (or cloud processing) one frame every $N$, and for the other you use only tracking.
  
  * Scaled-down CNN models or GAN, are also possible. There exist optimization frameworks like [tf-lite](https://www.tensorflow.org/lite?hl=en) that allow you to compress (by quantization and pruning), optimize, and even deploy a DL model. Indeed, you need to care about specific optimizations for the CPU and GPU.
  
  If you plan to implement something similar on your on PC, you can use [OpenCV](https://opencv.org/) (either with Python or C++) for, e.g., tracking, then compress a DL model with tf-lite in case you use tensorflow and/or Keras: you can take a pre-trained model, and compress with such tool. In that way, you can target optimized CPU and GPU models. For the graphics part you should find some optimized library for GPU, or write the code yourself like in OpenGL or similar. You can also have a look at [Dlib](http://dlib.net/): it's a nice lib, that has many image processing, computer vision, ML implementations, and even pre-trained models.
  
  > **Comments:**
  > **username_0**: Thank you username_1. By "one frame every N", I assume you mean "one frame every N number of seconds"? So the tracking algorithms run on the phone to track facial features. Any such algorithm names you remember?  
  > **username_1**: To achieve real-time processing you should process at least 24 image frames in a second (say up to 60 for best smoothness). It's not necessary to either gather more of them, so you should capture camera images on a fixed interval basis. Assuming that, you do all the heavy work at frame 1, then for the next $N$ frames (e.g. 10) you do tracking ($N$ should be not too large otherwise you loose accuracy). At frame $N+1$ you do the heavy work again, and so on. For tracking you can start from the [Lucas-Kanade method](https://docs.opencv.org/3.4/d4/dee/tutorial_optical_flow.html).  
  
  ---
  
  