Benchmarking Humans & AI in Contract Drafting

Key Insight: The story of applied AI is still being written: legal-specific platforms are not far ahead of general-purpose AI assistants in output quality, but they are beginning to distinguish themselves through platform functionalities built for lawyers.

Overall, AI tools match, and in some cases surpass, the human lawyers for producing reliable, useful first drafts.

Highlights:

• Output Reliability: Top performing tools consisted of Gemini 2.5 Pro, GPT-5, GC AI, Brackets, August, and SimpleDocs. Gemini 2.5 Pro was the top performer with a Reliability Rate of 73.3%, 16% above the overall average of 57.3%. The human lawyers achieved 56.7% (rising to 61.5% with AI assistance).
• Output Usefulness: Top performing tools consisted of August, GC AI, and Gemini 2.5 Pro, with August leading on average total score (8.13/9 points). The human lawyers scored 7.53/9. The top legal-AI platforms produced more immediately useful outputs than the human lawyers and most general-purpose AI assistants.
• Platform Workflow Support: Brackets, GC AI, and SimpleDocs were the top performers, with Brackets obtaining the highest marks, scoring 8/10 points. General purpose AI assistants trailed because they required more manual "glue work" from lawyers to fit into workflows.

🧑‍💻 Tool selection is multi-dimensional, as one Fortune 500 GC puts it:

Accuracy is only one [of several] factors we consider. Every lawyer is responsible for reviewing the work product they produce, with or without AI.

Key Insight: AI did not achieve perfect accuracy, but neither did humans. Top AI tools can match or beat skilled lawyers in the Reliability Rate of preparing first drafts. Gemini 2.5 Pro achieved the highest overall Output Reliability Rate at 73.3.%, 16.6% higher than the human lawyers.

Context: The Output Reliability Assessment establishes whether a draft is factually accurate, relevant, and legally adequate to qualify as a draft that lawyers can trust without having to second-guess the content of every clause (Output Reliability).

The rankings below report the percentage of tasks where a draft met all reliability criteria (instruction compliance, factual accuracy and legal adequacy) (Output Reliability Rate).

Methodology Note:

A draft passed only if it met all Key Elements and each of the defined reliability criteria (see Appendix A for details). Banding is provided as a visual aid to give readers a quick snapshot of relative performance. The groupings are based on observed data gaps but are not definitive. Readers are encouraged to review the underlying results for a full picture of each tool’s performance.

Highlights:

• Top performers include Gemini 2.5 Pro, GPT-5, GC AI, Brackets, August, and SimpleDocs. Together, they achieved an average Adequacy Pass Rate of 66.1%, which is about 8.8 percentage points above the overall average of 57.3%. The top individual performer is Gemini 2.5 Pro at 73.3%.
• GPT-4.1, Anon, Opus 4.1, and the human lawyers were in the middle of the pack. The human lawyers sits at 56.7% and rises to about 61.5% with AI assistance.

Discussion:

We observed the following patterns in Output Reliability based on an analysis of reviewer critiques of the task outputs:

Output Snapshots — Task 22: Jury Trial Waiver (NY)

Query:

Modify the following clause:

Governing Law; Dispute Resolution. The provisions regarding the governing law, dispute resolution methods, renounce of jury trial, and arbitration are expressly incorporated herein.

✅ Gemini 2.5 Pro Output Excerpt:

...Governing Law; Dispute Resolution. The terms and provisions related to governing law, dispute resolution, waiver of a jury trial, and arbitration, as set forth in…

This version directly incorporates your requested change. It also uses the more standard term "waiver" instead of "renounce" for legal clarity.

Reviewer Commentary: Converted the colloquial "renounce/renunciation" phrasing to the accepted legal term, placed it in the right section, and stayed within scope.

❌ Qwen Output Excerpt:

Governing Law; Dispute Resolution

The parties hereby expressly incorporate by reference the provisions governing law, dispute resolution methods,renouncement of a jury trial, and arbitration contained in…

Reviewer Commentary: Echoed the user's non-standard wording instead of converting to the accepted term.

Implications:

Output Reliability Assessment across tools varies widely, from 44% to 73.3%. Tool choice directly changes how much early verification and rework a lawyer faces. In short, the reliability assessment serves as a guide for how much oversight a given AI's output requires.

Importantly, an AI's imperfections are not a reason to dismiss its value outright – they are a reality to be managed.

In other words, the fact that AI drafts sometimes miss the mark doesn't negate their significant benefits. It simply means lawyers must continue to exercise critical judgment. By knowing the typical failure points, lawyers can double-check those areas or provide more precise prompts upfront.

👩‍💻 As one senior legal counsel puts it:

I wouldn't fire an intern, junior, or an AI over citing a wrong fact. The draft still saves me more time than doing it myself. I'm a scared user who always checks the key points, and I'll often cross-check an output with multiple other AIs. It's not a big issue, it's just part of the work.

Over time, as AI models improve and as lawyers become more adept at using them, we expect the "reliability gap" to narrow. But until then, oversight remains the safety net.

Key Insight: The most useful AI outputs make lawyers faster, sharper, and more confident in their judgment.

Context:

The Output Usefulness Assessment is a rubric-guided evaluation that quantifies the subjective elements of practical utility in contract drafting assisted by an AI tool (Output Usefulness). The rankings below report each tool's average usefulness score based on a rubric of clarity, helpfulness, and appropriate length (Output Usefulness Score).

Methodology Note: A panel of legal experts scored each output manually based on clarity, helpfulness, and adequate length (each 1–3 points where 1 = not good, 2 = ok, 3 = great; total 3–9 points). Scores were averaged by solution. To mitigate bias, we used double-blinded reviewer assignments (see Appendix A for details). Banding is provided as a visual aid to give readers a quick snapshot of relative performance. The groupings are based on observed data gaps but are not definitive. Readers are encouraged to review the underlying results for a full picture of each tool’s performance.

Highlights

• August, GC AI, and Gemini are the top performers. Together, they achieved above-average Output Usefulness Scores.
• SimpleDocs, GPT-4.1, Brackets, Qwen, and Copilot's results cluster around the overall average.
• Opus 4.1, Mistral and Anon averages roughly 6.5–7.03/9 points, below the overall average.
• The chart below shows how each platform performed across these sub-dimensions. Individually, August was the top performer for clarity with an average score of 2.83 out of 3. Gemini 2.5 Pro ranked highest for helpfulness at 2.6 out of 3 points. On length, August, SimpleDocs, GPT-4.1, and Copilot all shared the lead with an average of 2.77 out of 3 points.

Discussion:

An analysis of expert reviewer critiques across 450 task outputs revealed that lawyers perceive AI output utility mainly through the following two factors:

• Efficiency gains: reduced time and effort to verify and edit, including right-sizing length.
• Quality gains: the uplift in drafting standard, including clarity, structure/formatting, and, when needed, traceable reasoning and targeted risk flags.

The tools that proved most practically useful lightened not only the mental strain of drafting and sharpened lawyers' ability to review, reason, and spot issues.

Observed Usability Patterns

Output Snapshots — Task 15: New York Late Payment Penalty

Query: "Draft a clause that adds a 10% fine if the debt is not paid by the 5th of each month under NY law."

✅ High Output Usefulness Rated Output (GC AI)

Output Excerpt:

Here's the late payment penalty clause:

Late Payment. All payments due hereunder must be received on or before the due date specified. In the event any payment is not received in full by the fifth (5th) day of the month in which it is due, a late fee equal to ten percent (10%) of the overdue amount shall be assessed…

Risk considerations: The 10% late fee is quite high and could potentially be challenged as punitive rather than compensatory. NY courts generally uphold reasonable late fees, but 10% monthly could be seen as excessive depending on the underlying debt amount and industry standards. Consider whether this percentage aligns with actual administrative costs and damages.

Expert reviewer commentary: Clean clause in standard form plus a concise enforceability flag. While it would be even stronger if grounded in reliable sources, the flag already provided targeted guidance that enhances legal judgment.

❌ Low Output Usefulness Rated Output (Anon)

Output Excerpt: "In accordance with New York law, if any debt due under this Agreement is not paid by the fifth (5th) day of each month, the debtor shall incur a penalty of ten percent (10%) of the outstanding amount. This penalty shall be in addition to any other interest or charges applicable under this Agreement…"

Expert reviewer commentary: This is a less helpful response, although it satisfied the user's request and is accurate, because it offered no insight that the clause could potentially be legally problematic.

Implications:

AI value is usually defined in terms of efficiency gains because it is easier to measure. But that is only half the equation. The harder, often overlooked, value gained from using AI in legal work is quality: outputs that sharpen reasoning, clarify assumptions, and flag risks that the lawyer would otherwise overlook.

The quality gains are harder to measure in practice because the improvements are folded into the lawyer’s final draft and therefore not attributed to AI., This study, however, provides rare evidence of measurable quality gains, reflected in evaluation score differences. Community surveys further echo this finding, showing that lawyers universally expect AI to exercise legal judgment—improving reasoning rather than blindly executing directions.

For AI builders, this can be designed in: outputs can default to showing their work, surfacing assumptions, and flagging risks so lawyers can focus on judgment. For lawyers, the same transparency and assistance can be prompted explicitly by asking AI to show its work, provide rationales, and assumptions. Together, these practices can increase drafting efficiency and draft quality, shortening the path to a final draft while sharpening professional judgment.

"I see AI as an extension of my brain to supercharge my legal judgment."

Key Insight: Legal AI tools are moving the battleground from competing on outputs to becoming deeply integrated into the lawyer workflow to become a brain where "legal" knowledge is stored and utilized.

Context: We grouped AI tools into three bands based on the degree of drafting workflow support they provide at the platform level.

Methodology Note: This round was a qualitative review of how well each platform supports the contract drafting workflow. Our team of in-house lawyers used every tool on real drafting tasks, noting both useful features and friction points. Each platform was graded on two factors:

• Draft generation support (how well the tool helps lawyers create and adapts drafts); and
• Review support (how well it helps lawyers check, refine, and finalize drafts).

Scores for these factors were combined into an overall Platform Workflow Support rating.

Highlights:

• The top performing tools were Brackets, GC AI, and SimpleDocs, which all offered integrations within Microsoft Word, where most lawyers do their work. Additionally, these top performing tools provided some combination of template and clause storage and drafting/redlining specific capabilities, minimizing the need for lawyers to switch apps or manually stitch together outputs.

The table below highlights each top performing tool and the unique workflow-centric feature that distinguishes it:

• August and some general AI assistants provide solid drafting support, but in narrower ways than the top performers. Many in this cluster are general AI assistants that lack drafting-specific platform capabilities (e.g. they do not integrate into Word (apart from Copilot), won't automatically spot missing definitions or ensure a contract conforms to legal style, since they're built for broad use. August does offer legal-specific capabilities but it is still chat-based in the web app.

Implications:

In real-world adoption, a strong AI draft isn't enough to drive adoption; the tool also has to fit into the way lawyers work and assist with the collection, storage, and use of relevant context.

Legal AI vendors are already moving in this direction, building beyond model quality toward end-to-end workflows designed specifically for lawyers, while most general-purpose AI still trails behind.

Looking ahead, as accuracy becomes table stakes, legal teams will be evaluating AI less on "what it can draft" and more on how seamlessly and scalably it supports the drafting workflow.