Answers.Body: <p>(Assuming English.) In your specific example, there would be <span class="math-container">$26^2$</span> combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you <span class="math-container">$26 * 26 * 7 = 4732$</span> possible combinations. My intuition is that the key space is too small.</p>
  <p>It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from <span class="math-container">$(26 + 10)^6 = 2,176,782,336$</span> possibilities. That keyspace is <span class="math-container">$460,013$</span> times larger than your system with the longer string. It gets worse though.  An attacker could use a probabilistic approach, determining a &quot;letter probability&quot; for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. <em>It would not be a guaranteed win for the attacker.</em> But it would be guessing at a higher probability than <span class="math-container">$\frac{1}{4732}$</span>.</p>
  <p>Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.</p>
  <p>I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just &quot;Pick the first pair of adjacent letters.&quot; But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T15:07:30.327Z
Answers.Text: Sorry for being unclear. I will augment the original question with more detail.
Answers.UserDisplayName: Robert Cerny
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-04T14:10:25.797Z
Answers.LastActivityDate: 2021-04-04T15:14:16.403Z
Answers.LastEditDate: 2021-04-04T15:14:16.403Z
Answers.UserDisplayName: CorruptedHeapScapeGoat
Body: <p>I developed a novel type of CAPTCHA based on <strong>text comprehension and random tokens</strong>. Given a task <code>Pick the first pair of adjacent letters</code> and a random token <code>8NBA596V</code>, the user has to provide the solution <code>NB</code>. It offers basic protection and an attacker can solve individual tasks with <strong>specific</strong> effort. I am curious, whether contemporary AI can solve it <strong>generically</strong>?</p>
  <p>You can access more example tasks here:
  <a href="https://www.topincs.com/manual/captcha" rel="nofollow noreferrer">https://www.topincs.com/manual/captcha</a></p>
  <p>There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task
  – essentially a predicate in a tiny universe of discourse – and then applying it to the random token.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T09:50:39.007Z
Comments.Text: Have you tried feeding this task into any of the available GPT models to see how it answers? Such as GPT-2 or GPT-J-6B
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-03T10:02:49.303Z
Comments.Text: I tried it on GPT-J-6B and it repeatedly failed this task. They can sometimes do previously unknown tasks just based on a description of the task, but apparently, not this one.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <natural-language-processing><captcha>
Title: CAPTCHA based on text comprehension and random tokens
UserDisplayName: Robert Cerny
ThreadText: # CAPTCHA based on text comprehension and random tokens
  
  (Asked by: username_0 on 2021-03-04)
  
  I developed a novel type of CAPTCHA based on **text comprehension and random tokens**. Given a task `Pick the first pair of adjacent letters` and a random token `8NBA596V`, the user has to provide the solution `NB`. It offers basic protection and an attacker can solve individual tasks with **specific** effort. I am curious, whether contemporary AI can solve it **generically**?
  
  You can access more example tasks here:
  <https://www.topincs.com/manual/captcha>
  
  There is a task database and at every attempt a new task is presented with a new random token. They always have a solution of varying length and pure guessing thus has limited chances of success. It is easy to attack an individual task by writing a small piece of code, thus a large task database is essential. What intrigues me is the question whether natural language processing or machine learning at its current state can attack the CAPTCHA generically by building a model of the meaning of the task
  – essentially a predicate in a tiny universe of discourse – and then applying it to the random token.
  
  **Tags:** natural-language-processing, captcha
  > **Comments:**
  > **username_1**: Have you tried feeding this task into any of the available GPT models to see how it answers? Such as GPT-2 or GPT-J-6B  
  > **username_1**: I tried it on GPT-J-6B and it repeatedly failed this task. They can sometimes do previously unknown tasks just based on a description of the task, but apparently, not this one.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-03-04. Score: 0
  
  (Assuming English.) In your specific example, there would be $26^2$ combinations of capitalized letters. Also assuming a fixed-length token of eight, that gives you $26 \* 26 \* 7 = 4732$ possible combinations. My intuition is that the key space is too small.
  
  It may actually be simpler for a machine to solve the CAPTCHA than that. Let's say that both your system and a generic one obscure text in an image with equal ability. The generic one is a string of length six composed of capital letters and numbers. So the attacker needs to make an informed guess from $(26 + 10)^6 = 2,176,782,336$ possibilities. That keyspace is $460,013$ times larger than your system with the longer string. It gets worse though. An attacker could use a probabilistic approach, determining a "letter probability" for each of the eight positions, taking the two adjacent positions that maximize the probability, and then choosing the letter with the highest probability for each of those solutions. *It would not be a guaranteed win for the attacker.* But it would be guessing at a higher probability than $\frac{1}{4732}$.
  
  Also, it may be easier for an ML system to distinguish certain letters from others. B/P may be confused more easily compared to M/T (I'm hypothesizing here). So the attacker's ML could be more confident in some letter pairs than others. This is analogous to the idea of weak keys in cryptography.
  
  I like where your head is at. But this may not be the result you're looking for. In theory, you could have multiple rules beyond just "Pick the first pair of adjacent letters." But each new rule would add a negligible benefit on top of what is a fundamental problem in the design.
  
  > **Comments:**
  > **username_0**: Sorry for being unclear. I will augment the original question with more detail.  
  
  ---
  
  