Will IBM’s Watson transform contract review and law practice?

Watson is almost certainly the most significant technology ever to come to law, and it will give lawyers permission to think innovatively and open up the conversation about what is possible in a field that has been somewhat “stuck.” “10 predictions about how IBM’s Watson will impact the legal profession“, Paul Lippe and Daniel Martin Katz, ABA Journal

IBM’s Watson AI has received a lot of attention for how it might change law practice. Should it? Or should commentators expecting “Watson” to change the world instead refocus their attention on “artificial intelligence” or “machine learning”?

Recently, legal market observer Ron Friedmann wrote, in a post on potential business models for Watson in the legal space: ‘if we aim to improve the efficiency of the legal market, there is no lack of technology to choose from. Whether Watson is the best place to bet remains an open question‘.

It is one thing to say that machine learning and AI will deeply impact legal practice. It is another to say that Watson will have a deep impact, or a more significant impact than other technologies. Watson is partially a machine learning offering, but there are many other machine learning offerings. Critically, there is no one machine learning model to rule them all. Rather, there are a lot of different flavors of machine learning that are optimal for different machine learning tasks, and the view of which is best for a given task is changing all the time.

To understand how much variation there is in the understanding of what is state of the art, consider sentiment analysis, a niche area of AI. For the past few years, recursive neural tensor networks (a type of neural network) have been viewed as the leading way to do sentiment analysis. Recently, provocative new research suggesting a better approach has emerged and is being debated. Illustrating how much more complex this is, recursive tensor networks require heavily annotated training data, which means linear classifiers (a different technique) often end up being the best solution for this work. One niche, three different potentially best ways to solve problems within it. By the way, none of the three best-practice options are Watson-connected.

 To defensibly say that Watson—and not machine learning or AI software in general, or some other proprietary machine learning or AI system—will change legal practice more than other approaches, we should consider three questions: 

1. What is Watson currently good it, and is it even the best avenue for automating tasks where it is strongest?
2. Is Watson strong in the most promising areas for legal automation?
3. Will Watson grow to become leading machine learning or AI technology across the board, or will it remain high quality only for question answering?

Watson, Currently

From Wikipedia: “Watson is an artificially intelligent computer system capable of answering questions posed in natural language”. Today, it appears to be a leading machine learning offering for question answering of a very specific sort (as we will cover below). IBM looks to be attempting to build Watson out into leading general purpose artificial intelligence software, but there is no consensus that it is at or better than the state of the art in areas beyond question answering. Indeed, early reviews on released Watson APIs have been underwhelming. As Ron Friedmann points out, it is not even clear whether Watson is a better technological approach for legal question answering tasks than, say, Neota Logic.

Apparently, one vendor is currently using Watson to extract data from contracts. I have yet to see any data suggesting that identifying contract provisions is in Watson’s sweet spot (by data, I mean information such as, say, published provision extraction accuracy numbers for a system built using Watson; the vendor claims their “goal today is to deliver a 20% cost reduction for a law firm in a typical diligence exercise”, which would not stand out relative to claims from other contract review software vendors (e.g., our clients tell us they find from 20–45% time savings on page-by-page review using our contract review software, and 60–90% time savings when they rely on it more heavily)).

Is Current Watson Right for Legal Problems?

Watson is currently a leading technology for question answering tasks. Are most legal tasks that could be impacted by software question answering tasks?

“Question answering” could be very broad, and most or all legal tasks could be interpreted as giving answers to questions. However, today Watson only stands out for performance on a narrow definition of question answering.

Each of the following questions illustrates a type of legal problem technology could help solve:

1. Is it illegal for individuals to have ferrets in the state of California? (Watson-type question answering)
2. Which of these 1 million documents are relevant to determining if anti-competitive behavior occured in this specific case? (eDiscovery technology-assisted review)
3. How long does it tend to take for cases to get to trial in front of Judge Vernon Broderick? (Lex Machina)
4. Who will the Supreme Court decide for in King v. Burwell? (Katz/Bommarito/Blackman algorithm)
5. Can you draft a brief for us to submit to the court for this case? (NarrativeScience, Automated Insights (neither appear to be currently targeting legal))
6. Do any expense items on this legal bill seem inappropriate? (SimpleLegal)
7. Which of these contracts have change of control or exclusivity clauses? (us (Kira) and others)

Despite how all of these legal automation areas were phrased as questions, current Watson seems to only have documented high quality performance on tasks similar to the first, legal researchey issue. Is there any data to support the idea that Watson could best other currently-existing technology solutions on the other questions?

Alternatively, is being great at legal research question answering sufficient to make Watson the leading legal technology? Are all the other areas insignificant compared with an ability to know the law? As an ex-corporate lawyer, not to me. I know there are large numbers of lawyers that hardly ever do legal research.*

Will Future Watson Be Better Than Alternatives?

A Watson proponent might say maybe Watson is only truly great at question answering right now, but it will grow into the accross the board leading AI technology. After all, Watson won Jeopardy!, and IBM is pouring tons of resources into bettering it. As Friedmann states, after listing off a number of companies—including us—who build contract review software:

But as Paul points out, Watson’s R&D investment is probably 100x all these companies combined, and so has the potential to ride a much steeper performance curve.

Since machine learning does not yet have one approach that is better than others across the board, it is hard to say how much value Watson’s extensive R&D investment matters in the contract review software space (or, for that matter, in most other areas of technology, legal or otherwise). The argument that Watson will dominate outside its core because of overall R&D investment is akin to arguing that Lance Armstrong, straight off winning seven consecutive Tour de Frances, would win the 2006 New York Marathon. After all, he had incredible aerobic capacity, slow twitch muscle training, toughness, and more spent on his training than others in the field. Plus, running is basically just putting one foot in front of the other. For what it’s worth, Armstrong finished in 856th place his first try, and 232nd the next year.

Winning Jeopardy! is great, but there have been many other very impressive machine learning feats, including self driving cars; translation, including of live speech; and writing decent enough quality news articles that human reviewers could not necessarily tell the difference. Even on question answering, was Ken Jennings easy competition at Jeopardy! relative to a different current AI system like Google DeepMind?

Moving past Jeopardy!, IBM may be putting significant resources into Watson, but other companies are doing the same. Some equally large companies to IBM, including Google, HP, Facebook, and Baidu, are also putting a lot of resources into machine learning. Why will IBM beat them? Why will IBM even beat out newer AI focused startups such as DeepMind (bought by Google in 2014 for $650 Million), MetaMind, or many others? IBM itself appears to recoginize that others are building valuable machine learning technology, acquiring deep learning focused AlchemyAPI in May. 

Lots of companies are building AI technology for specific verticals (like us with contract review). Current machine learning is quite problem-specific, and these companies are getting experience honing their technology for their particular use cases. Will Watson’s technology really be better for specific verticals than companies focused on those specific verticals? Would you use Watson for eDiscovery ahead of offerings from companies who have been focused on that challenge for years? Will Watson do machine learning fraud detection better than well-funded Sift Science? Or movie recommendation better than Netflix? Will Watson even be better on legal research than something Thomson Reuters builds? IBM may be a lot bigger than TR, but TR is not small and has to nail this, whereas IBM does not need to get legal research right.

There are a lot of different legal tasks ripe for automation. There are also a lot of different technological approaches to solving AI problems. I suspect we’re a long way out from saying any one vendor’s technology is going to transform law practice on the whole.

* There is one other way Watson could transform law practice that I did not discuss here. Perhaps question answering lies behind as-yet-undiscovered-but-transformative legal applications. No doubt, there is a lot of opportunity to improve law practice through technology.

Author

Noah Waisberg, is co-founder and CEO of Kira Systems. Prior to founding Kira Systems, Noah practiced at Weil, Gotshal & Manges in New York, where he focused on private equity, M&A, and securities. Noah is an expert on contract analysis, legal technology, and artificial intelligence. He holds a J.D. from the NYU School of Law, an A.M. from Brown University, and a B.A. with honours from McGill University.

Photo credit: U.S. Centers for Disease Control via Wikimedia Commons