1.1 Introduction

AI is entering the in-house legal domain quickly, but it's still unclear how well it performs on the work lawyers actually do. Most resources available to in-house lawyers today on AI performance in the legal domain are polished vendor demos, academic papers, or ideal-condition benchmarks, which don’t reflect everyday workflows. 

And they rarely answer the question:

Can the tool do the job, and how much human input does it need?

This report is a step in that direction of answering these questions. 

We focused on putting AI to the test on  Information Extraction Tasks first because it underpins much of legal work from contract review to issue spotting, and is often a starting point for legal AI adoption.

We designed the evaluation around 2 principles:

  • Real-world inputs. Tasks were submitted by in-house counsel, using documents with redactions and formatting issues.

  • Practical usefulness. We assessed not just accuracy, but whether the output was usable—clear, scoped appropriately, and supported by features like citations or multi-document processing. [2]

The key findings are as follows:

%
Copilot scored the lowest in overall accuracy (38.9%), underscoring the need for caution when relying on AI assistants, even for seemingly simple information extraction tasks.
%
Legal AI tools did not outperform general-purpose LLM chatbots in accuracy. ChatGPT and DeepSeek answered 12 out of 18 tasks (66.7%) accurately, matching or exceeding the overall accuracy of legal AI tools.
Legal AI tools ranked highest on qualitative measures of usefulness. Across 3 dimensions-helpfulness, adequate length, and feature support- legal AI tools outperformed all general-purpose AI tools, making them more suited to the needs of in-house lawyers handling daily tasks.

[2] As much as we considered mirroring classic NLP-style Q&A tasks, that approach didn't reflect the realities of in-house legal work. A question like "Is there a limitation of liability clause in this agreement?" might warrant a yes or no, but in practice, lawyers need more: what the clause says and where to find it etc.