Answers.Body: <p>Any ML model's evaluation has to come from an independent test set which is supposed to be simple random samples from the target population, therefore only one instance is not evidence of huge variation in competency across different models, only an experienced fluke. To gather evidence, you need to prepare much more test samples perhaps beyond 10,000 or much more for a face recognition model. And in your case the result difference may not be due to the underlying ML model, but some reference database containing people's information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-17T03:53:44.963Z
Answers.Text: perhaps beyond 10,000 test samples? I was wondering whether to (1) clip closeups of the other two actresses from the same movie, or (2) find a couple more "based on real events" movies and do similar close-up clip searches with them. Are you saying it would still not be significant if Yandex correctly identified the relevant real-world events from two similar clips created by either of those two methods?
Answers.UserDisplayName: FumbleFingers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-17T05:34:18.360Z
Answers.Text: Face recognition models generally use standardized benchmark dataset like LFW (Labeled Faces in the Wild) or MegaFace. While testing additional clips can be insightful, two or three clips (or even a dozen) are still too few to draw meaningful conclusions. The results from such a small-scale test would not statistically represent the model's or service's general performance which is not specifically for real-world events portrayed in movies. The threshold of 10,000 samples is a general guideline for large-scale ML systems, however, for a targeted use case a smaller dataset might suffice.
Answers.UserDisplayName: cinch
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-17T11:33:53.757Z
Answers.Text: I'm very grateful you took the trouble to respond. It took me quite a while to construct this question, after which I was keen to get back to the movie! But I *did* check a clip of Julia Roberts in Erin Brockovich (2000) afterwards. Bing and lensio positively identified the actress, but only yandex clearly homed in on the exact frame within the movie (again, the clip was less than 5% of the total frame from which it was taken). Some sites seem to work better than others with that kind of image.
Answers.UserDisplayName: FumbleFingers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-17T11:39:40.800Z
Answers.Text: ...but I shall now look into what api functionality yandex offers. I don't know if I could create it myself, but I could certainly use a VLC plugin to let me pause playback and quickly draw a box around a "person of interest" in a movie. With a quick response identifying the actor and/or the character they're playing in that movie!
Answers.UserDisplayName: FumbleFingers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-17T03:35:21.363Z
Answers.LastActivityDate: 2024-11-17T03:35:21.363Z
Answers.UserDisplayName: cinch
Body: <p>I'm not very good at recognizing faces, so I knew it would be taxing to watch <a href="https://www.imdb.com/title/tt6394270/" rel="nofollow noreferrer">Bombshell (2019)</a>, since all three lead actresses (Charlize Theron, Margot Robbie, and Nicole Kidman) look very similar to me. But I'm such a fan of Margot Robbie I made the effort.</p>
  <p>Cutting to the chase, there was a point (46m) where I didn't know who I was looking at, so I paused the video, clipped this from a screenshot...</p>
  <p><a href="https://i.sstatic.net/8FGrWwTK.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/8FGrWwTK.png" alt="enter image description here" /></a></p>
  <p>...and uploaded it to <em>Reverse Image Search</em> on <a href="https://www.duplichecker.com/reverse-image-search.php" rel="nofollow noreferrer">duplichecker.com</a>. Which unsurprisingly didn't find the &quot;original&quot;. But it offered &quot;Find similar images in different search engines&quot;.</p>
  <p>The first few different search engines all gave worthless results ranging from <a href="https://www.bing.com/images/search?view=detailv2&amp;iss=sbi&amp;form=SBIVSP&amp;sbisrc=UrlPaste&amp;q=imgurl:https%3A%2F%2Ftemp-ca.s3.amazonaws.com%2Fris299343413sst1731800415.png%3FX-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIA2YDWKLAVXKBZDYMV%252F20241116%252Fus-east-1%252Fs3%252Faws4_request%26X-Amz-Date%3D20241116T234015Z%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Expires%3D86400%26X-Amz-Signature%3D88a8e316bf46d26d75454a92be153b594b40a0dfa15c3c2f9667710ed3271e13" rel="nofollow noreferrer">Bing's</a> <em>We couldn't find any results</em> to <a href="https://lenso.ai/en/results/07e80b1128rsdiigzl6bpcj7mkhl4u8ickwkchyhhwxtnffoi7ttd5zcz3dc4o3v?w=1&amp;h=1&amp;z=1&amp;cW=1&amp;cH=1" rel="nofollow noreferrer">Lensio</a> showing me 4 different blond[e?] actresses (none were in <em>Bombshell</em>). But <a href="https://yandex.com/images/search?cbir_id=4027797%2F8ndLntkgJvZ3FkrDlMR3pQ486&amp;rpt=imageview&amp;source=collections&amp;url=https%3A%2F%2Favatars.mds.yandex.net%2Fget-images-cbir%2F4027797%2F8ndLntkgJvZ3FkrDlMR3pQ486%2Forig" rel="nofollow noreferrer">yandex.com</a> returned this...</p>
  <p><a href="https://i.sstatic.net/FyNTz5LV.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/FyNTz5LV.png" alt="enter image description here" /></a></p>
  <p>If you don't know, the picture is actress Charlize Theron playing media personality Megyn Kelly - who at the exact point where I paused the movie is agonizing between staying at Fox or moving to NBC. Yandex has apparently seamlessly moved from less than 3% of a single video frame in a movie to the real-life &quot;celebrity news&quot; context being depicted.</p>
  <p>If this <em>isn't</em> a wild fluke, is it evidence of huge variation in competency across different free AI-powered image recognition services?</p>
  
Tags: <image-recognition><contextual-bandits>
Title: Is this just a wild fluke?
UserDisplayName: FumbleFingers
ThreadText: # Is this just a wild fluke?
  
  (Asked by: username_0 on 2024-11-17)
  
  I'm not very good at recognizing faces, so I knew it would be taxing to watch [Bombshell (2019)](https://www.imdb.com/title/tt6394270/), since all three lead actresses (Charlize Theron, Margot Robbie, and Nicole Kidman) look very similar to me. But I'm such a fan of Margot Robbie I made the effort.
  
  Cutting to the chase, there was a point (46m) where I didn't know who I was looking at, so I paused the video, clipped this from a screenshot...
  
  [![enter image description here](https://i.sstatic.net/8FGrWwTK.png)](https://i.sstatic.net/8FGrWwTK.png)
  
  ...and uploaded it to *Reverse Image Search* on [duplichecker.com](https://www.duplichecker.com/reverse-image-search.php). Which unsurprisingly didn't find the "original". But it offered "Find similar images in different search engines".
  
  The first few different search engines all gave worthless results ranging from [Bing's](https://www.bing.com/images/search?view=detailv2&iss=sbi&form=SBIVSP&sbisrc=UrlPaste&q=imgurl:https%3A%2F%2Ftemp-ca.s3.amazonaws.com%2Fris299343413sst1731800415.png%3FX-Amz-Content-Sha256%3DUNSIGNED-PAYLOAD%26X-Amz-Algorithm%3DAWS4-HMAC-SHA256%26X-Amz-Credential%3DAKIA2YDWKLAVXKBZDYMV%252F20241116%252Fus-east-1%252Fs3%252Faws4_request%26X-Amz-Date%3D20241116T234015Z%26X-Amz-SignedHeaders%3Dhost%26X-Amz-Expires%3D86400%26X-Amz-Signature%3D88a8e316bf46d26d75454a92be153b594b40a0dfa15c3c2f9667710ed3271e13) *We couldn't find any results* to [Lensio](https://lenso.ai/en/results/07e80b1128rsdiigzl6bpcj7mkhl4u8ickwkchyhhwxtnffoi7ttd5zcz3dc4o3v?w=1&h=1&z=1&cW=1&cH=1) showing me 4 different blond[e?] actresses (none were in *Bombshell*). But [yandex.com](https://yandex.com/images/search?cbir_id=4027797%2F8ndLntkgJvZ3FkrDlMR3pQ486&rpt=imageview&source=collections&url=https%3A%2F%2Favatars.mds.yandex.net%2Fget-images-cbir%2F4027797%2F8ndLntkgJvZ3FkrDlMR3pQ486%2Forig) returned this...
  
  [![enter image description here](https://i.sstatic.net/FyNTz5LV.png)](https://i.sstatic.net/FyNTz5LV.png)
  
  If you don't know, the picture is actress Charlize Theron playing media personality Megyn Kelly - who at the exact point where I paused the movie is agonizing between staying at Fox or moving to NBC. Yandex has apparently seamlessly moved from less than 3% of a single video frame in a movie to the real-life "celebrity news" context being depicted.
  
  If this *isn't* a wild fluke, is it evidence of huge variation in competency across different free AI-powered image recognition services?
  
  **Tags:** image-recognition, contextual-bandits
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-11-17. Score: 2
  
  Any ML model's evaluation has to come from an independent test set which is supposed to be simple random samples from the target population, therefore only one instance is not evidence of huge variation in competency across different models, only an experienced fluke. To gather evidence, you need to prepare much more test samples perhaps beyond 10,000 or much more for a face recognition model. And in your case the result difference may not be due to the underlying ML model, but some reference database containing people's information.
  
  > **Comments:**
  > **username_0**: perhaps beyond 10,000 test samples? I was wondering whether to (1) clip closeups of the other two actresses from the same movie, or (2) find a couple more "based on real events" movies and do similar close-up clip searches with them. Are you saying it would still not be significant if Yandex correctly identified the relevant real-world events from two similar clips created by either of those two methods?  
  > **username_1**: Face recognition models generally use standardized benchmark dataset like LFW (Labeled Faces in the Wild) or MegaFace. While testing additional clips can be insightful, two or three clips (or even a dozen) are still too few to draw meaningful conclusions. The results from such a small-scale test would not statistically represent the model's or service's general performance which is not specifically for real-world events portrayed in movies. The threshold of 10,000 samples is a general guideline for large-scale ML systems, however, for a targeted use case a smaller dataset might suffice.  
  > **username_0**: I'm very grateful you took the trouble to respond. It took me quite a while to construct this question, after which I was keen to get back to the movie! But I *did* check a clip of Julia Roberts in Erin Brockovich (2000) afterwards. Bing and lensio positively identified the actress, but only yandex clearly homed in on the exact frame within the movie (again, the clip was less than 5% of the total frame from which it was taken). Some sites seem to work better than others with that kind of image.  
  > **username_0**: ...but I shall now look into what api functionality yandex offers. I don't know if I could create it myself, but I could certainly use a VLC plugin to let me pause playback and quickly draw a box around a "person of interest" in a movie. With a quick response identifying the actor and/or the character they're playing in that movie!  
  
  ---
  
  