Bard’s latest update: more features, languages and countries
fun story[0] about Bard&#x27;s <i>June</i> update - implicit code execution  (remember? <a href="https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36229782">https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=36229782</a>) - the example that they advertised on their blogpost&#x2F;video demo now no longer works.<p>Here&#x27;s what I got as of today: <a href="https:&#x2F;&#x2F;g.co&#x2F;bard&#x2F;share&#x2F;2580222b6fb2" rel="nofollow noreferrer">https:&#x2F;&#x2F;g.co&#x2F;bard&#x2F;share&#x2F;2580222b6fb2</a> (screenshot from a couple days ago: <a href="https:&#x2F;&#x2F;twitter.com&#x2F;swyx&#x2F;status&#x2F;1678495067663925248?s=20" rel="nofollow noreferrer">https:&#x2F;&#x2F;twitter.com&#x2F;swyx&#x2F;status&#x2F;1678495067663925248?s=20</a>)<p>&gt; What are the prime factors of 15683615?<p>&gt; (after outputting some code) The output of the code is the list [5, 11, 23, 43]<p>this result is hallucinated - Bard tells you it is running the code, but it doesnt. If you run the code, you get [5, 151, 20773], which actually is the correct answer<p>now lets all acknowledge that maintaining stable performance of LLMs is an unimaginably hard problem, but* i suggest some regression testing to keep up the baseline. Would fuzz the test (really, its just another eval) because of course this test is going to be memorized in the weights soon<p>0: i first discussed this on monday&#x27;s update <a href="https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;code-interpreter#§misc-notes-that-didnt-fit-anywhere" rel="nofollow noreferrer">https:&#x2F;&#x2F;www.latent.space&#x2F;p&#x2F;code-interpreter#§misc-notes...</a><p>* Edited based on feedback from commenters, thanks.