Federal judge: No AI in my courtroom unless a human verifies its accuracy
One of the difficulties here is that simply verifying for accuracy isn&#x27;t sufficient. If you&#x27;re relying on ChatGPT to do research it can still mislead you. It&#x27;s quite easy to find examples where it will withhold information it does actually know, because it thinks you shouldn&#x27;t be told about that for the greater good i.e. it can go beyond mere hallucination and explicitly lie to the user in order to advance its own agenda, whilst simultaneously claiming it doesn&#x27;t have one.<p>If you&#x27;re a lawyer doing research for a case, that seems like a fundamental problem that simply reading over a draft and verifying claims can&#x27;t fix. Without doing your own research by hand, you may miss critical information, especially if the court case is about a topic that ChatGPT has pre-conceived moral notions about.<p>OpenAI could probably fix this and in fairness to them, they seem to have made a genuine effort to improve the model&#x27;s RLHF training since the launch in November. I&#x27;ve seen clear improvements in the neutrality of its answers. However, this capacity for explicit deception hasn&#x27;t gone away and is deeply troubling. In extremis over-reliance on LLMs for legal work could lead to a loss of trust in the legal system as a whole, as people will suspect they aren&#x27;t getting an honest defense if their position happens to be unpopular with the sort of people who train models.