Answers.Body: <p>I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.</p>
  
  <p>Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line". </p>
  
  <p>One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index <code>idx</code> and the one right after, at index <code>idx+1</code> (captured at instants <span class="math-container">$t$</span> and <span class="math-container">$t + \Delta t$</span>, where <span class="math-container">$\Delta t$</span> is the sampling interval).</p>
  
  <p>At coordinates <span class="math-container">$(i,j)$</span>, this derivative could look something like: <code>timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT</code>. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.</p>
  
  <p>Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.</p>
  
  <p>When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.</p>
  
  <p>This is the main idea, which can be improved with:</p>
  
  <ul>
  <li>Defining the area of the blackboard, either by hand or automatically</li>
  <li>Thresholding the images to better isolate the chalk trace from the blackboard</li>
  <li>Using a tracker such as a Kalman filter to know where to look next for the chalk</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-15T19:52:55.777Z
Answers.Text: Thanks for your answer. I think movement of the teacher in front of the chalkboard will cause hard interferences to your proposal of algorithm.
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-15T21:37:16.997Z
Answers.Text: Indeed. To cut out the teacher, you can threshold then apply a blob extraction algorithm, such as the two pass algorithm. This can identify the teacher if done correctly. You can then cut the teacher out of the image and color his pixels like the blackboard. This comes of course before the start of what I wrote above.
Answers.UserDisplayName: David Cian
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-28T01:26:51.030Z
Answers.Text: Or maybe just divide the shalkboard in regions and update them only if the teacher isn't present. Just tracking how much the green part is changed in a region with a threshold could be a simple and effective way to identify the presence of a big object and reject the update. A heavily pixelated version of the video can work too as "region" selector
Answers.UserDisplayName: Ernesto Iglesias
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-15T17:17:05.293Z
Answers.LastActivityDate: 2019-12-15T17:17:05.293Z
Answers.UserDisplayName: David Cian
Answers.Body: <p>Maybe Just use simple Convnets (Pre-trained perhaps) and train it on the images of the teacher on the blackboard. You <em>could</em> use a GAN to remove the teacher and complete the rest of the image (<a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwj9lKKSh43uAhWbfH0KHeaDC1UQFjAAegQIAxAC&amp;url=http%3A%2F%2Fstanford.edu%2Fclass%2Fee367%2FWinter2018%2Ffu_guan_yang_ee367_win18_report.pdf&amp;usg=AOvVaw2tG3bStRlys_NPLX9-XPep" rel="nofollow noreferrer">https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwj9lKKSh43uAhWbfH0KHeaDC1UQFjAAegQIAxAC&amp;url=http%3A%2F%2Fstanford.edu%2Fclass%2Fee367%2FWinter2018%2Ffu_guan_yang_ee367_win18_report.pdf&amp;usg=AOvVaw2tG3bStRlys_NPLX9-XPep</a>) But that would be too troublesome.</p>
  <p>The best way would be to take real-time video chunks, use a Convolutional Network to detect the shapes you want, and return bounding boxes for the appropriate shapes (for the location purposes) and their 'classification' - whether a straight line, curved line, or some other user defined shape. You could also choose to use some other YOLO (You Only Look Once) technique. You can check out with: <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAJegQIARAC&amp;url=https%3A%2F%2Ftowardsdatascience.com%2Fobject-detection-using-deep-learning-approaches-an-end-to-end-theoretical-perspective-4ca27eee8a9a&amp;usg=AOvVaw2TkW094_O5Q7-mcVMJ5SEN" rel="nofollow noreferrer">https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAJegQIARAC&amp;url=https%3A%2F%2Ftowardsdatascience.com%2Fobject-detection-using-deep-learning-approaches-an-end-to-end-theoretical-perspective-4ca27eee8a9a&amp;usg=AOvVaw2TkW094_O5Q7-mcVMJ5SEN</a> ;</p>
  <p>You also won't have to deal with the pesky teacher with the above method (assuming he doesn't stand in one place and constantly block a part of the BB). These methods are near SOTA and would be <strong>extremely</strong> effective than conventional algos. Not to mention that using <code>Keras</code> is a piece of cake with a giant community and endless resources to help you in case you get stuck on some problem. Since it is easy to use, you can setup a prototype in almost no time.</p>
  <p>Beginner's guide and Introduction-&gt; <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjALegQIGhAC&amp;url=https%3A%2F%2Fmachinelearningmastery.com%2Fobject-recognition-with-deep-learning%2F&amp;usg=AOvVaw3M-b1gYnbTdzzwPahTTWWR" rel="nofollow noreferrer">https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjALegQIGhAC&amp;url=https%3A%2F%2Fmachinelearningmastery.com%2Fobject-recognition-with-deep-learning%2F&amp;usg=AOvVaw3M-b1gYnbTdzzwPahTTWWR</a> ;</p>
  <p>A research paper from ArXiv--&gt; <a href="https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAPegQIHhAC&amp;url=https%3A%2F%2Farxiv.org%2Fpdf%2F1807.05511&amp;usg=AOvVaw33CWoOX4LlrA3T_f75zVZu" rel="nofollow noreferrer">https://www.google.com/url?sa=t&amp;rct=j&amp;q=&amp;esrc=s&amp;source=web&amp;cd=&amp;cad=rja&amp;uact=8&amp;ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAPegQIHhAC&amp;url=https%3A%2F%2Farxiv.org%2Fpdf%2F1807.05511&amp;usg=AOvVaw33CWoOX4LlrA3T_f75zVZu</a></p>
  <p>Training with YOLO: <a href="https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/" rel="nofollow noreferrer">https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-09T15:54:19.737Z
Answers.Text: please, recall note at the end of the question
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-09T17:11:07.197Z
Answers.Text: @pasabaporaqui You want someone to implement the code for you? Apart from that, the only details I can add in my answer is an explanation on CNN's, if you want.
Answers.UserDisplayName: neel g
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-09T20:04:32.407Z
Answers.Text: What I do not want, as the note says, is a empty "use a GN" or "use a CNN" followed by a useless list of 3 links explaining the generics of Deep Learning or object detection. Instead, it is expected a reference to a paper, algorithm or method, something as "in xxx work xxx shows how he/she has combined the Hough transform with a CNN to obtain the xxx result in a scenario similar to the one of the question".
Answers.UserDisplayName: pasaba por aqui
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-01-08T19:27:23.157Z
Answers.LastActivityDate: 2021-01-08T19:38:37.693Z
Answers.LastEditDate: 2021-01-08T19:38:37.693Z
Answers.UserDisplayName: neel g
Body: <p>I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it. </p>
  
  <p>It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...</p>
  
  <p>After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.</p>
  
  <p>Any hint ?</p>
  
  <p>Input will be a sequence as <a href="https://www.shutterstock.com/video/clip-9146711-teacher-writing-on-blackboard-class-ultra-hd" rel="nofollow noreferrer">this one</a>,  <a href="https://videohive.net/item/young-male-teacher-or-student-holding-chalk-writing-on-chalkboard-in-classroom/17391669" rel="nofollow noreferrer">this one</a> or <a href="https://www.youtube.com/watch?v=gBpqT3IEIQU" rel="nofollow noreferrer">this one</a> (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.</p>
  
  <p>Thanks.</p>
  
  <p>Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references. </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-14T18:53:38.390Z
Comments.Text: I would look into Hough transform https://en.wikipedia.org/wiki/Hough_transform
Comments.UserDisplayName: George White
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-14T19:31:36.593Z
Comments.Text: @GeorgeWhite: thanks for your comment. As you know, Hough transform is mainly to detect straight lines. It can be generalized to some other parametric curves, but I do not see promising use it in the case of free forms.
Comments.UserDisplayName: pasaba por aqui
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-12-14T19:33:40.613Z
Comments.Text: From ""start of line at xxx,xxx", "continue line at xxx,xxx"" I thought you were assuming straight lines.
Comments.UserDisplayName: George White
Tags: <object-detection><image-processing>
Title: Recognition of lines in a chalkboard
UserDisplayName: pasaba por aqui
ThreadText: # Recognition of lines in a chalkboard
  
  (Asked by: username_0 on 2019-12-14)
  
  I'm trying to develop a real-time application that, from the sequence of chalkboard images captured by a webcam, recognizes the lines being draw on it.
  
  It must be able of recognize the lines from the chalkboard background, filter the presence in the image of the teacher, and translate these lines to some representation, something as a list of basic events like "start of line at xxx,xxx", "continue line at xxx,xxx", ...
  
  After several days looking for references and bibliography, none is found. The most similar are the character recognition applications, in particular when they have a stroke recognition stage.
  
  Any hint ?
  
  Input will be a sequence as [this one](https://www.shutterstock.com/video/clip-9146711-teacher-writing-on-blackboard-class-ultra-hd), [this one](https://videohive.net/item/young-male-teacher-or-student-holding-chalk-writing-on-chalkboard-in-classroom/17391669) or [this one](https://www.youtube.com/watch?v=gBpqT3IEIQU) (just without the presence of the students). I've expect the teacher not hidding his hand. We could imagine a start with an empty chalkboard.
  
  Thanks.
  
  Note: I am looking for more than an answer which says only something similar to "you can use a deep learning training it with two classes", without details or references.
  
  **Tags:** object-detection, image-processing
  > **Comments:**
  > **username_1**: I would look into Hough transform https://en.wikipedia.org/wiki/Hough_transform  
  > **username_0**: @GeorgeWhite: thanks for your comment. As you know, Hough transform is mainly to detect straight lines. It can be generalized to some other parametric curves, but I do not see promising use it in the case of free forms.  
  > **username_1**: From ""start of line at xxx,xxx", "continue line at xxx,xxx"" I thought you were assuming straight lines.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-12-15. Score: 0
  
  I will assume that the camera is stable (no change in position, zoom or other settings during the video recording), otherwise the task becomes markedly more complicated.
  
  Let's say that your dataset is an array of rasters (images in array format). You mention that you want to detect events "start line" and "end line".
  
  One way of doing this would be to compute an approximate time derivative of your image series. For instance, take the image raster at index `idx` and the one right after, at index `idx+1` (captured at instants $t$ and $t + \Delta t$, where $\Delta t$ is the sampling interval).
  
  At coordinates $(i,j)$, this derivative could look something like: `timeDerivative = (images[idx][j][i] - images[idx+1][j][i])/DeltaT`. This is a crude estimate, and there are better ways of computing an approximate discrete derivative, but you get the idea.
  
  Following this, you could declare a state of the recording: drawing line or not drawing line. The states, we assume are always alternating, as a teacher has to take his hand off the blackboard to draw a new line.
  
  When a derivative with large values is detected (a region in the image goes from being black to white suddenly) and the state is "not drawing", the event "start line" is recorded and the state switches to "drawing". While a derivative with large values continues to be detected as time goes on in the vicinity of the previous spot with a large derivative, nothing changes. Once this is no longer true, the state changes to "not drawing" and the event "stop line" is recorded at the last location with a large derivative.
  
  This is the main idea, which can be improved with:
  
  * Defining the area of the blackboard, either by hand or automatically
  * Thresholding the images to better isolate the chalk trace from the blackboard
  * Using a tracker such as a Kalman filter to know where to look next for the chalk
  
  > **Comments:**
  > **username_0**: Thanks for your answer. I think movement of the teacher in front of the chalkboard will cause hard interferences to your proposal of algorithm.  
  > **username_2**: Indeed. To cut out the teacher, you can threshold then apply a blob extraction algorithm, such as the two pass algorithm. This can identify the teacher if done correctly. You can then cut the teacher out of the image and color his pixels like the blackboard. This comes of course before the start of what I wrote above.  
  > **username_3**: Or maybe just divide the shalkboard in regions and update them only if the teacher isn't present. Just tracking how much the green part is changed in a region with a threshold could be a simple and effective way to identify the presence of a big object and reject the update. A heavily pixelated version of the video can work too as "region" selector  
  
  ---
  
  ### Answer by username_4 on 2021-01-08. Score: -1
  
  Maybe Just use simple Convnets (Pre-trained perhaps) and train it on the images of the teacher on the blackboard. You *could* use a GAN to remove the teacher and complete the rest of the image (<https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwj9lKKSh43uAhWbfH0KHeaDC1UQFjAAegQIAxAC&url=http%3A%2F%2Fstanford.edu%2Fclass%2Fee367%2FWinter2018%2Ffu_guan_yang_ee367_win18_report.pdf&usg=AOvVaw2tG3bStRlys_NPLX9-XPep>) But that would be too troublesome.
  
  The best way would be to take real-time video chunks, use a Convolutional Network to detect the shapes you want, and return bounding boxes for the appropriate shapes (for the location purposes) and their 'classification' - whether a straight line, curved line, or some other user defined shape. You could also choose to use some other YOLO (You Only Look Once) technique. You can check out with: <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAJegQIARAC&url=https%3A%2F%2Ftowardsdatascience.com%2Fobject-detection-using-deep-learning-approaches-an-end-to-end-theoretical-perspective-4ca27eee8a9a&usg=AOvVaw2TkW094_O5Q7-mcVMJ5SEN> ;
  
  You also won't have to deal with the pesky teacher with the above method (assuming he doesn't stand in one place and constantly block a part of the BB). These methods are near SOTA and would be **extremely** effective than conventional algos. Not to mention that using `Keras` is a piece of cake with a giant community and endless resources to help you in case you get stuck on some problem. Since it is easy to use, you can setup a prototype in almost no time.
  
  Beginner's guide and Introduction-> <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjALegQIGhAC&url=https%3A%2F%2Fmachinelearningmastery.com%2Fobject-recognition-with-deep-learning%2F&usg=AOvVaw3M-b1gYnbTdzzwPahTTWWR> ;
  
  A research paper from ArXiv--> <https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwiei4C3iI3uAhXXfX0KHQXxAVAQFjAPegQIHhAC&url=https%3A%2F%2Farxiv.org%2Fpdf%2F1807.05511&usg=AOvVaw33CWoOX4LlrA3T_f75zVZu>
  
  Training with YOLO: <https://machinelearningmastery.com/how-to-perform-object-detection-with-yolov3-in-keras/>
  
  > **Comments:**
  > **username_0**: please, recall note at the end of the question  
  > **username_4**: @pasabaporaqui You want someone to implement the code for you? Apart from that, the only details I can add in my answer is an explanation on CNN's, if you want.  
  > **username_0**: What I do not want, as the note says, is a empty "use a GN" or "use a CNN" followed by a useless list of 3 links explaining the generics of Deep Learning or object detection. Instead, it is expected a reference to a paper, algorithm or method, something as "in xxx work xxx shows how he/she has combined the Hough transform with a CNN to obtain the xxx result in a scenario similar to the one of the question".  
  
  ---
  
  