Asking 60 LLMs a set of 20 questions
I think the larger point is about solving problems not within the transitive closure of stuff it has not learnt already. In the following scenario where I replace brothers and sisters with different names but give ChatGPT Plus plenty of hints to try self correct it can&#x27;t. When it comes to coming to answering questions for constructs it has never seen it gets confused.<p>-------------<p>User:<p>You have to answer a question about a family on a planet inhabited by aliens called Frobians, . Each Frobian can be one of two types Frobel or Frobo.
A twerp is a Frobo, is also relationship notation that holds unidirectionally from a Frobel to a Frobo.
A clep is a Frobel,  is also  is a relationship notation that holds unidirectionally from Frobo to a Frobel<p>In a family Master Foo, a Frobian, is of type Frobel. Master Foo has 3 twerps in their family. Each twerp in the family has 2 cleps. How many Frobels in total does Master Foo have?<p>-------------<p>Assistant: In this problem, we have Master Foo, who is a Frobel. Master Foo has 3 twerps, and each twerp is a Frobo. Each of these twerps has 2 cleps, and each clep is a Frobel.<p>Here&#x27;s how to break it down:<p><pre><code>    Master Foo is a Frobel, so we start with 1 Frobel.
    Master Foo has 3 twerps, which are Frobos and not Frobels, so they don&#x27;t add to the count of Frobels.
    Each of the 3 twerps has 2 cleps, and each clep is a Frobel. So, 3 twerps * 2 cleps&#x2F;twerp = 6 cleps, which are Frobels.
</code></pre>
Adding it all together, we have:<p>1 (Master Foo) + 6 (cleps) = 7 Frobels in total in Master Foo&#x27;s family.<p>User: The families are similar to families of humans on earth. Does that change your answer?<p>-------------<p>Asssistant:<p>If the families are similar to human families on Earth, then it&#x27;s likely that the &quot;cleps&quot; (Frobels) are actually the children of the &quot;twerps&quot; (Frobos), and the &quot;twerps&quot; are the children of Master Foo (also a Frobel).<p>In that case, the family structure would look something like this:<p><pre><code>    Master Foo (1 Frobel)
        Twerp 1 (Frobo)
            Clep 1 (Frobel)
            Clep 2 (Frobel)
        Twerp 2 (Frobo)
            Clep 3 (Frobel)
            Clep 4 (Frobel)
        Twerp 3 (Frobo)
            Clep 5 (Frobel)
            Clep 6 (Frobel)
</code></pre>
So, Master Foo is a Frobel, and each of the 3 twerps has 2 cleps, making 6 cleps who are also Frobels.<p>The total number of Frobels in Master Foo&#x27;s family would still be 1 (Master Foo) + 6 (cleps) = 7 Frobels. The similarity to human families doesn&#x27;t change the total count of Frobels in this particular scenario.<p>-------------<p>Is Master Foo a Clep according to your analysis?<p>In the original problem, a &quot;clep&quot; is defined as a relationship notation that holds unidirectionally from a Frobo to a Frobel. Since Master Foo is a Frobel and has &quot;twerps&quot; who are Frobos, Master Foo would indeed be a &quot;clep&quot; to each of his twerps, according to the definitions provided.<p>So, yes, Master Foo would be considered a &quot;clep&quot; in relation to his twerps, based on the definitions given. However, this doesn&#x27;t change the total count of Frobels in the family, which remains at 7.<p>-------------<p>User: 
Add a new rule a Frobel can have a clep who is another Frobel and a Frobo can also have a twerp who is abother Frobo. Does that change your answer
ChatGPT<p>-------------<p>Assistant:
With the new rule that a Frobel can have a clep who is another Frobel and a Frobo can have a twerp who is another Frobo, the family structure becomes more complex. However, based on the information originally provided, this new rule doesn&#x27;t change the count of Frobels in Master Foo&#x27;s family.<p>According to the original scenario:<p><pre><code>    Master Foo is a Frobel (1 Frobel)
        Has 3 twerps (Frobos, not Frobels)
            Each twerp has 2 cleps (Frobels)
</code></pre>
The original count was 1 (Master Foo) + 6 (cleps) = 7 Frobels.<p>The new rule allows for the possibility of additional relationships, but it doesn&#x27;t change the relationships described in the original scenario. Therefore, the total count of Frobels in Master Foo&#x27;s family remains at 7, based on the information given.