Pilot criteria for legal tech: what to agree before signing

Write the pilot's criteria before it starts, make them measurable and let the vendor know them. Without criteria there is nothing to measure against, and the decision is made on feeling or on who spoke last. Here is how I set them, how long the pilot should be, who should take part, and how the decision is made.

Updated by Nils-Erik Jansson

Write the criteria first

Before the pilot starts there is a two-page document that every participant and the vendor have read: what is being tested, by whom, for how long, which criteria apply, which levels mean yes, and when the decision is made. The document does not change during the pilot. The LegalBenchmarks evaluation framework describes a staged evaluation [1]. This is my protocol for the pilot stage. I recommend a pilot of four to eight weeks, with the end date and the decision meeting booked before it starts.

Threshold requirements

These are requirements that must be met however good the tool is otherwise. They are tested before the pilot starts. Data protection: the five questions in the data protection checklist have answers, and the pilot runs under an agreement that covers the data used, or with anonymised data. Security: sign-in through the company's identity provider, permissions that match yours, certifications with reports. Exit: export of what was created during the pilot in an open format, and deletion afterwards. A tool that does not meet the threshold requirements does not go to pilot. Points do not come into it.

Four measures

Measure How it is measured Level for yes
Benefit Time or quality in the two or three workflows the pilot covers, against a baseline taken beforehand. For contract review: time from receiving a contract to an annotated draft, and the number of deviations from the playbook compared with a manual review of the same contract Set in advance, for example a quarter less time with no loss of accuracy
Quality The share of answers the assessors approve without rewriting, and the number of substantive errors, in a sample of twenty tasks during the pilot No substantive errors that would have gone through without review, and an approval rate set after the demo
Use The share of participants who used the tool at least three times a week in the last two weeks, and which features they used A clear majority
Operation Number of disruptions, response time from the vendor's support, and how many questions needed the vendor's help The team can run the tool itself after implementation

Low use with high benefit means something in the workflow is wrong, and that is a result.

Participants

Four to eight people: at least two lawyers, at least two from the business that requests legal work, and the person who will own the tool afterwards. When I built a contract management platform, legal was often the function that preferred the tool, but rarely the one holding the budget. Do not pick only the enthusiasts. Include a sceptic, so that not all the data comes from people who have already made up their minds. Everyone gets the same one-hour introduction and the same access.

Logging

Fifteen minutes a week per participant: what they used the tool for, what worked, what did not, and one example of each. Collected by the pilot owner, not by the vendor. The vendor gets a summary every week and a chance to fix things, and what you then see is how the vendor works, which is also a result.

The decision

One meeting, booked before the pilot started, with the criteria on the table. Three outcomes: yes, with a contract in line with the requirements list; no, with one sentence for each criterion not met; or an extension of one week for a reason that is written down. An extension without a reason is a no that nobody dares to say.

The decision also covers what happens to the data and to what was created: export or deletion, confirmed by the vendor.

The pilot is one of the steps in legal technology selection, after the requirements list and the demo on your own contracts. HOVR Legal writes the criteria with you, runs the logging and the compilation, and delivers the basis for the decision. How the legal tech fact sheets are made is set out on the page How the fact sheets are made.

Download

Free. No sign-up and no email address.

Frequently asked questions

How long should a pilot be?
I recommend four to eight weeks. A shorter pilot does not get past the novelty and the first technical problems. A longer one turns into a creeping roll-out without a decision, and then the vendor has won without the criteria being tested. Set the end date and the decision meeting before the pilot starts.
Should we pay for the pilot?
Preferably a low fixed price, so that both sides take it seriously and the vendor does not make it conditional on a purchase. Check that a free pilot is not limited in features or time, so that you test what you are going to buy. Agree that the pilot does not commit you to anything, that data is deleted afterwards if you do not go ahead, and that the purchase price is fixed during the pilot.
What if the tool meets the criteria but nobody wants to use it?
Then use was not a criterion, and it should have been. The share of participants who used the tool at least three times a week in the last two weeks is one of the four measures in my protocol. A tool that delivers the benefit but not the use has an interface or workflow problem, and that will not resolve itself after the purchase.

Sources

  1. [1]LegalBenchmarks.ai, The Evaluation Framework

Insights on the topic

Back to Technology selection

More under Technology selection