, Nils-Erik Jansson
Pilot criteria for legal tech: what to agree before signing
Write the pilot's criteria before it starts, make them measurable and let the vendor know them. Without criteria there is nothing to measure against, and the decision is made on feeling or on who spoke last. Here is how I set them, how long the pilot should be, who should take part, and how the decision is made.
Updated by Nils-Erik Jansson
Write the criteria first
Before the pilot starts there is a two-page document that every participant and the vendor have read: what is being tested, by whom, for how long, which criteria apply, which levels mean yes, and when the decision is made. The document does not change during the pilot. The LegalBenchmarks evaluation framework describes a staged evaluation [1]. This is my protocol for the pilot stage. I recommend a pilot of four to eight weeks, with the end date and the decision meeting booked before it starts.
Threshold requirements
These are requirements that must be met however good the tool is otherwise. They are tested before the pilot starts. Data protection: the five questions in the data protection checklist have answers, and the pilot runs under an agreement that covers the data used, or with anonymised data. Security: sign-in through the company's identity provider, permissions that match yours, certifications with reports. Exit: export of what was created during the pilot in an open format, and deletion afterwards. A tool that does not meet the threshold requirements does not go to pilot. Points do not come into it.
Four measures
| Measure | How it is measured | Level for yes |
|---|---|---|
| Benefit | Time or quality in the two or three workflows the pilot covers, against a baseline taken beforehand. For contract review: time from receiving a contract to an annotated draft, and the number of deviations from the playbook compared with a manual review of the same contract | Set in advance, for example a quarter less time with no loss of accuracy |
| Quality | The share of answers the assessors approve without rewriting, and the number of substantive errors, in a sample of twenty tasks during the pilot | No substantive errors that would have gone through without review, and an approval rate set after the demo |
| Use | The share of participants who used the tool at least three times a week in the last two weeks, and which features they used | A clear majority |
| Operation | Number of disruptions, response time from the vendor's support, and how many questions needed the vendor's help | The team can run the tool itself after implementation |
Low use with high benefit means something in the workflow is wrong, and that is a result.
Participants
Four to eight people: at least two lawyers, at least two from the business that requests legal work, and the person who will own the tool afterwards. When I built a contract management platform, legal was often the function that preferred the tool, but rarely the one holding the budget. Do not pick only the enthusiasts. Include a sceptic, so that not all the data comes from people who have already made up their minds. Everyone gets the same one-hour introduction and the same access.
Logging
Fifteen minutes a week per participant: what they used the tool for, what worked, what did not, and one example of each. Collected by the pilot owner, not by the vendor. The vendor gets a summary every week and a chance to fix things, and what you then see is how the vendor works, which is also a result.
The decision
One meeting, booked before the pilot started, with the criteria on the table. Three outcomes: yes, with a contract in line with the requirements list; no, with one sentence for each criterion not met; or an extension of one week for a reason that is written down. An extension without a reason is a no that nobody dares to say.
The decision also covers what happens to the data and to what was created: export or deletion, confirmed by the vendor.
How HOVR Legal works with this
The pilot is one of the steps in legal technology selection, after the requirements list and the demo on your own contracts. HOVR Legal writes the criteria with you, runs the logging and the compilation, and delivers the basis for the decision. How the legal tech fact sheets are made is set out on the page How the fact sheets are made.
Frequently asked questions
- How long should a pilot be?
- I recommend four to eight weeks. A shorter pilot does not get past the novelty and the first technical problems. A longer one turns into a creeping roll-out without a decision, and then the vendor has won without the criteria being tested. Set the end date and the decision meeting before the pilot starts.
- Should we pay for the pilot?
- Preferably a low fixed price, so that both sides take it seriously and the vendor does not make it conditional on a purchase. Check that a free pilot is not limited in features or time, so that you test what you are going to buy. Agree that the pilot does not commit you to anything, that data is deleted afterwards if you do not go ahead, and that the purchase price is fixed during the pilot.
- What if the tool meets the criteria but nobody wants to use it?
- Then use was not a criterion, and it should have been. The share of participants who used the tool at least three times a week in the last two weeks is one of the four measures in my protocol. A tool that delivers the benefit but not the use has an interface or workflow problem, and that will not resolve itself after the purchase.
Sources
Insights on the topic
Back to Technology selection
More under Technology selection
- Technical review of legal tech vendors
An independent review of what a legal tech or AI tool does behind the demo. You get a report on technology, data and vendor, with a recommendation.
- Legal technology selection: from need to decision
HOVR Legal leads the selection of legal tech, CLM or AI tools: requirements, long and short list, pre-demo questions, demo on your contracts, pilot, decision.
- Legal tech implementation: from selection to daily use
HOVR Legal brings the tool you chose into your organisation: owners, templates and rules, migration, data protection, a pilot group, training and handover.
- CLM requirements for a Swedish legal team: before the demo
CLM requirements for a legal team in Sweden, before the demo: workflow, templates, signing, archive, AI, integrations, data protection, exit, price.
- Demo on your own contracts: a protocol for legal tech
How a legal team runs a demo of legal tech and AI tools on its own contracts in place of the vendor's examples.
- Legal AI in Sweden 2026: tools and framework deals
An overview for lawyers working with Sweden: the AI tools sold to Swedish law firms and legal teams, and what vendors say about sources and data.
- E-signatures under eIDAS, with Sweden as an example
The three eIDAS signature levels and their legal effect across the EU, what evidence rests on, and Sweden as an example: BankID and form requirements.