A new study released on Wednesday claims that artificial intelligence can now outperform most law school graduates on the bar exam, the two-day test aspiring attorneys must pass to practice law in the United States. GPT-4, the upgraded AI model released this week by Microsoft-backed OpenAI, scored 297 on the bar exam in an experiment conducted by two law professors and two employees of legal technology company Casetext. That places GPT-4 in the 90th percentile of actual test takers and is enough to be admitted to practice law in most states, the researchers found.
The bar exam assesses knowledge and reasoning and includes essays and performance tests meant to simulate legal work, as well as multiple-choice questions. “Large language models can meet the standard applied to human lawyers in nearly all jurisdictions in the United States by tackling complex tasks requiring deep legal knowledge, reading comprehension, and writing ability,” the authors wrote. The newer GPT-4 got nearly 76% of the bar exam’s multiple-choice questions right, outperforming the average human test-taker by more than 7%.
The National Conference of Bar Examiners, which designs the multiple-choice section, said that attorneys have unique skills gained through education and experience that “AI cannot currently match.” Study co-author Daniel Martin Katz, a professor at Chicago-Kent College of Law, said that he was most surprised by GPT-4’s ability to produce largely relevant and coherent essay and performance test answers.
“AI has come a long way in recent years and can now perform well on other standardized tests, including the SAT and the GRE, but the bar exam has garnered more attention,” said bar exam tutor Sean Silverman. This year’s first-time pass rate on the attorney licensing exam was 78% among test takers who spent three years in law school.
The researchers noted that AI can help provide legal services to those who cannot afford traditional legal representation. The study also noted that AI can help lawyers process large amounts of data, enabling them to better understand legal documents and improve their legal practice. However, it is unclear whether AI will replace human lawyers altogether.
The focus on the bar exam is due to its widely recognized difficulty, according to Silverman. He said people may be less impressed to learn that AI can pass a test designed for high schoolers, like the SAT, “rather than the test to become a lawyer.” It should be noted that less than four months ago, two of the same researchers concluded that OpenAI’s earlier large language model, ChatGPT, fell short of a passing score on the bar exam, highlighting how rapidly the technology is improving.
AI has already begun to transform the legal industry, from drafting legal documents to predicting court decisions. Legal technology companies have developed AI-powered software that can sift through thousands of pages of legal documents and highlight the relevant information. AI has also been used to predict the outcome of legal cases based on past judicial decisions.
While AI may not replace human lawyers entirely, it is clear that it will play an increasingly important role in the legal industry in the coming years. The results of this study demonstrate the potential for AI to improve legal services and access to justice. As AI technology continues to improve, it will be interesting to see how it transforms the legal industry and how lawyers adapt to these changes.
