3.3 AI Assistant Strengths

Across tasks, all AI Assistants showed 2 clear strengths:

• Enhanced Search Capabilities: Acting like a supercharged Ctrl+F, AI Assistants are able to quickly find answers when questions are clear and narrow and answers are in one document, and even understand semantically similar phrases, allowing them to pull answers that a simple keyword search might miss; and
• Consistent Formatting: AI Assistants consistently produced structured first drafts that help lawyers skip the drafting step and jump straight to review.

For example:

• In Task 1, which asked each AI Assistant to extract the contract term clause in an MSA, all 6 AI Assistants correctly identified the “Term and Termination” clause.
• If asked to provide clause citations or references, they would include clause numbers or headings from the document.
• In Task 16, when asked to extract the governing law for multiple contracts in a table format, the AI Assistants that successfully processed all 11 documents were able to present the output in structured tables and accurately identify applicable law, even when explicit “governing law” language was absent. This interpretive strength is valuable but can become a risk when it overreaches and infers answers not grounded in the text (see discussion in the next section).

Through conversations with legal teams, we identified the following practical use cases where AI assistants are already being successfully deployed for information extraction tasks: