Legal AI competency
Project Dreyfus
Developing legal AI competence through evidence, practice and judgment.
Research framework, assessment method and product in development
- Initial scope
- Text-based generative AI in US and Canadian legal contexts
- Proposed approach
- Observed work, source verification, targeted practice and reassessment
- Connected outputs
- Research, a reusable development program and an assessment product
The inquiry
What demonstrates competent legal work with AI?
A strong result matters. So do the decisions, verification and understanding behind it.
Project Dreyfus asks how legal professionals can develop and demonstrate the ability to select, use, verify, explain and oversee AI-supported work within a defined role and scope.
It connects three proposed outputs: a research framework, a reusable learning program, and an assessment and upskilling product. Legal knowledge and AI proficiency will be considered separately, alongside the quality of the work produced.
A common competency core will sit above modules for particular roles, practice areas, jurisdictions and tools. US and Canadian modules will need their own sources, professional requirements and expert review.
A conceptual starting point
From explicit rules to contextual judgment.
The project takes its name from Stuart and Hubert Dreyfus’s model of skill acquisition.
Stuart Dreyfus’s 2004 account describes five stages, progressing from rule-guided action toward experience-based recognition of what matters in a situation. The stages offer a lens for thinking about learning; their application to legal AI remains a research proposal.
| Stage | Core idea |
|---|---|
| Novice | Follows explicit rules with limited situational experience. |
| Advanced beginner | Begins to recognize meaningful features of real situations. |
| Competent | Chooses priorities and plans through deliberate decisions. |
| Proficient | Recognizes what matters in the situation while still deliberating about action. |
| Expert | Draws on extensive experience for refined, intuitive responses. |
Source: Stuart E. Dreyfus, The Five-Stage Model of Adult Skill Acquisition (2004).
The provisional framework
Eleven dimensions of legal AI competency.
Each dimension needs observable, task-specific criteria. The framework begins with practical legal research and source verification, alongside contract and document analysis.
Public or synthetic materials will support initial development.
- AI foundations: understand capabilities, uncertainty and limits.
- Task selection and framing: define the objective, stakes and appropriate role for AI.
- Instructions and context: supply relevant facts, permitted sources and clear constraints.
- Legal research and verification: check authority, proposition support, quotations, jurisdiction and currency.
- Legal reasoning and work product: apply law to facts, consider alternatives and preserve material terms.
- Output evaluation: examine accuracy, completeness, relevance, bias and consequences.
- Information handling: make defensible confidentiality, privilege, privacy and security decisions.
- Workflow and supervision: assign responsibility, review, handoffs and escalation.
- Metacognition and accountability: calibrate confidence, explain decisions and correct errors.
- Role-appropriate technical understanding: understand retrieval, permissions, retention and tool changes.
- Experimentation, teaching and research: evaluate systems, document failures and produce inspectable evidence.
Provisional proficiency descriptions
- Guided
- Identifies issues with scaffolding and needs review to complete material checks.
- Reliable within scope
- Completes familiar tasks and documents appropriate checks within a defined role and oversight arrangement.
- Adaptive
- Handles unfamiliar variants, detects hidden assumptions and justifies workflow changes or escalation.
- Expert contributor
- Demonstrates performance across tasks and tools over time; supports evaluation, supervision, teaching and reproducible work.
These descriptions are provisional, with no validated cutoffs. “Not observed” records missing evidence. One strong task does not establish a level across all dimensions.
Assessment method in development
Observe the work. Test the understanding.
The proposed method separates assisted output quality from what a person can explain, verify and carry into unfamiliar work.
Work samples and review
Candidate tasks include matched legal work with and without AI, deliberately flawed outputs, source checks and a written or oral defence of the result. A delayed task would examine transfer to a new problem.
AI-free work would ordinarily retain specified legal sources and conventional research aids. Alternate tasks and recorded tool conditions would help distinguish learning from familiarity with a particular exercise.
A profile of demonstrated ability
Feedback would report work quality, material errors, verification, confidence calibration, time including checking and revision, and transfer. Evidence coverage and scoring disagreements would remain visible.
Initial assessment would be formative and criterion-referenced, using defined task standards. Human legal reviewers would control consequential scoring; critical errors would prompt task-specific review and remediation.
A reusable development program
Practice that responds to evidence.
A proposed 12-week architecture moves from baseline and foundations, through targeted verification and reasoning, into supervised workflows, a defended portfolio and delayed reassessment.
The program would adapt to demonstrated needs in legal knowledge, tool use, verification or calibration. Progress would depend on observed work rather than time spent alone.
The learning cycle: attempt → predict confidence → verify → receive feedback → explain the correction → revisit a different task after a delay.
An assessment product in development
A reviewed diagnostic and a practical next step.
The first product concept is a manually delivered service: intake, reviewed simulation, a domain report, focused practice and reassessment.
Potential users include legal professionals and law-firm learning, knowledge, innovation and risk teams. Buyer demand, pricing and delivery economics remain hypotheses to test.
Research, the development program and the product would share task definitions and evidence while keeping practice exercises separate from secure assessment and unfamiliar transfer tasks.
Current status
A blueprint ready for refinement.
Project Dreyfus is a provisional research and design blueprint. The framework, assessment method, study plans, development program and product are in development.
It is not yet a validated assessment or certification. Population benchmarks, predictive claims and professional-readiness decisions would require evidence for their particular users and intended uses.
The next phase
Refine the evidence map, develop observable scoring criteria and sample tasks, obtain legal and assessment review, and prepare a small feasibility pilot. Broader claims will need independent participants, assessors and appropriate research oversight.
Discuss the framework or a pilotAn enquiry starts a conversation. Pilot arrangements and participation materials are still to be developed.
Assessment design reference: AERA, APA & NCME, Standards for Educational and Psychological Testing (2014). Validation must support the proposed interpretation and use of scores.