DESIGN defensible AI review workflow
Step 1: Define Review Objectives
Before anything is reviewed, we define:
- What is responsive?
- What is privileged?
- What issues are being coded?
- What date ranges apply?
- What custodians matter?
Step 2: Train the AI Process
Consider a database containing 5 million documents.
Instead of engaging 100 reviewers, the workflow might look like:
- Senior attorney codes sample sets.
- AI analyzes decisions.
- AI identifies likely responsive documents.
- AI categorizes issues.
- Humans validate the results.
This is very similar to Continuous Active Learning (CAL/TAR 2.0), except generative AI also summarizes, explains relevance, identifies people, and spots themes.
Step 3: Validate Everything
This is where firms need experienced attorneys.
A defensible workflow never says:
"The AI said it wasn't responsive, so we produced nothing."
Instead it says:
"AI identified documents, attorneys conducted QC sampling, measured recall and precision, reviewed outliers, and validated the results."
That validation process is what creates defensibility.
The difference is that you're QC'ing the AI rather than only QC'ing reviewers.
Step 4: Handle Privilege Carefully
This is the area where firms remain most nervous.
AI can:
- Flag privilege
- Draft privilege log descriptions
- Identify attorney communications
But most firms still require human review before final decisions are made.
A workflow consultant would help define things like:
- What confidence score triggers review?
- What documents must always receive human review?
- How are privilege decisions escalated
- What sampling thresholds apply?
Step 5: Document the Process
The firm should be able to answer questions such as:
- What AI tool was used?
- Who supervised it?
- What prompts were used?
- What validation testing occurred?
- How many documents were sampled?
- What error rate was identified?
- What corrective action was taken?
Many firms currently have no written policies for this.
Someone has to build them.