Icon (Close Menu)


AI’s Legal Magic: Despite Pitfalls, Attorneys Test Machine Learning

6 min read

Despite some data “hallucinations” and a detour into the magical world of Harry Potter, artificial intelligence makes a strong case for itself in the legal profession.

Big law firms like McGuireWoods of Chicago and Thompson Hine of Washington are training hundreds of professionals in AI use, including lessons in crafting prompts to get useful machine-learning responses.

But Arkansas firms are wading carefully into the AI world, with human lawyers double- and triple-checking AI’s work in e-discovery, brief-writing and research.

The American Bar Association has urged courts and lawyers to confront ethical and legal issues surrounding AI use. It also asked the government not to use pretrial risk assessment tools unless the data supporting them is transparent, publicly disclosed and validated against bias.

Chris Cameron, director of information technology at the Center for Arkansas Legal Services, thinks AI offers benefits for lawyers, including efficiency in research and discovery. But he cited some glaring AI pitfalls.

“There was a judge in a case out of New York, reacting to responses made by ChatGPT, and the judge called them ‘utterly and unusually unpersuasive,’” Cameron said. “And then there was a famous case, Mata v. Avianca Inc., where the AI actually cited fake cases.” Judge P. Kevin Castel of the Southern District of New York sanctioned two attorneys who relied on AI in that case.

Cameron also recalled a lawyer who used AI to search for cases favorable to his client. “When they looked into it, they couldn’t find those cases,” he said. “In the early days of AI, it wasn’t surprising to find these kinds of ‘hallucinations.’” One case that the AI program cited was eventually found in Harry Potter fan fiction. “It’s hilarious, but AI is only as good as its input, which is human.”

Undeniable Potential

Despite the downside of using unchecked AI work, the likely benefits are undeniable, Cameron and several Arkansas lawyers said. But they don’t see AI having a great effect on business any time soon.

“It can help with summarizing 100 pages of documentation quickly,” Cameron said. “It can help with research, and point to other relevant cases very quickly and accurately. It can help you outline responses and even give analysis of your arguments and your opponents’ arguments. All of this is saving time. It’s almost like having a genius law student clerking for you. They’re not the lawyer, of course, and the lawyers are still responsible for that work.”

Westlaw Edge is a tool that uses AI to automate legal searches. Other tools, like BriefCatch, employ machine learning to help write documents.

Taylor King

The Center for Arkansas Legal Services isn’t loading any privileged information into any AI programs for now, and is intentionally “going slowly,” Cameron said. His legal aid group is limiting the use of personally identifiable information and states “very explicitly that AI tools are meant to augment our legal expertise and not replace it.”

Taylor King, whose Arkadelphia-based personal injury and disability practice includes 22 lawyers in seven offices around the state, cautioned against relying too heavily on AI until the technology matures.

“We’ve been cautious after seeing various conferences and reading about it,” King said. “You have to be really careful and never just assume that the results are accurate. As I understand it, it’s calling on information that’s on the web, and that might not be accurate. I don’t know that it’s making stuff up, but I would urge lawyers to use it kind of like a draft and double-check the work.”

King said AI, as it stands, won’t likely affect his business or staffing. “I guess there are areas where it might replace some positions eventually, but I don’t know that it’s really going to affect my business now. AI is not ready to replace my people. I think it’s just a tool that we can use to create more efficiency.”

Testing the Tools

Rickie J. Smith, director of public relations at Wright Lindsey Jennings LLP of Little Rock, said her firm is testing Westlaw’s AI legal research tool. She noted it relies on Westlaw’s database rather than the broader internet. She said the firm also sees potential for artificial intelligence in electronic discovery, boiling down hundreds or thousands of e-documents that would otherwise require hours of human concentration.

Tiara King, WLJ’s director of e-discovery and project management, leads the firm’s AI evaluation effort in conjunction with attorneys Gary Marts, Quinten Whiteside and Jaimie Moss, Smith said.

“We’re cautiously optimistic that AI is a way we can augment processes and workflows to be more efficient, and ultimately pass those efficiencies and cost savings to the client,” Smith said. “We’re researching and demoing products and tools just like everyone else.”

WLJ doesn’t plan for machine learning to take anyone’s job. “Ultimately, you have to guard against some response that’s totally out there,” Smith said. “The attorney still is going to be ethically obligated to make sure all the information is factual. We’re looking to improve processes and workflows, save a little time and turn that into savings for clients.”

Big Firms Dive In

In a recent ABA web discussion, lawyers Peter A. Geovanes and William T. Garcia discussed how national firms are deploying AI.

Geovanes, chief innovation and AI officer for McGuireWoods LLP of Chicago, said his firm began testing generative AI tools like Casetext last year in a phased rollout. It is training attorneys to create effective prompts for the algorithms, and discussing ethical issues and technological downsides in training sessions.

Thompson Hine has used AI for predictive coding, case analytics and contract review for several years, said Garcia, the firm’s chief practice innovation officer. The firms call on AI tools to generate deposition questions, flag inconsistencies in witness statements and search through case law for clients’ advantage.

Neither Geovanes nor Garcia expects AI to take jobs, though they say the tools could change employees’ duties and skill sets. Proper training will be essential, they said.

At McGuireWoods, an initial test phase obtained 100 software use licenses from vendors to distribute AI across the firm, and attorneys signed a statement promising to upload no client data.

Garcia noted that AI tools for lawyers are still constantly evolving.

“We don’t believe in buying new shiny objects just to keep up with public statements of some of our competitors,” Garcia said. “We want to understand where and how these tools might be used in the firm.”

A LexisNexis survey of more than 1,000 lawyers last year found that 80% had not yet used generative AI at work, and that 87% had ethical doubts about the technology.

But algorithms have proved themselves for years in e-discovery, according to Bourgon Reynolds, managing member, northwest Arkansas, for the Rose Law Firm.

“It’s a big part of electronic discovery now,” Reynolds said. “Every email and every document you put on your computer, that can get swept up into [a client’s] discovery record.

“A huge amount of information has to be sorted through, searched and then eventually produced. And as you might imagine, a lot of that stuff is just irrelevant.”

AI tools called TAR, for technology-assisted review, quickly scan documents for relevant and responsive documents. The chaff, like Williams Sonoma promotion emails, to cite one of Reynolds’ examples, are sent to the bottom of the list.

Document review programs like Relativity or CS Disco can easily outperform lawyers, Reynolds said.

“Humans get tired,” she said. “It’s intensive, and you can have variations from person to person.”

The Rose Law Firm has been using TAR programs since as early as 2017, Reynolds said.

“Once trained by attorneys, the TAR tool can help reviewers quickly identify the documents that are relevant,” Reynolds said.

“And after proper testing of the tool’s performance, the balance of the documents not identified by the tool, sometimes numbering in the hundreds of thousands or even millions, need never be reviewed by a human.

“This equates to a huge time and cost savings for our clients, and you tend to get a better-quality document review, too, because you take out that human factor.”

Send this to a friend