Skip to main content

Buyer workspace · this week · ranked by community votes

Sign in
List your SaaS

Running a Team Pilot Before You Buy

Evaluating · 11 min read ·

A pilot tests a tool with real people on real work before you commit. How to scope, run and judge a four-week software pilot and decide on evidence.

Illustration: An indigo timeline of a four-week pilot with labelled stages Set up, Use, Review, Decide and a compact grid of pilot users and tasks beneath

If a demo is a performance, a pilot is a rehearsal with the real cast. It is the moment when a tool leaves the vendor's stage and meets your people, your data and your Tuesday afternoons. For a team about to commit to software they will rely on for years, it is the single most valuable step in the buying process.

It is also easy to waste. A pilot without a plan becomes an aimless trial that fades when people get busy. This guide shows how to scope, run and judge a pilot so that you finish with evidence, not impressions.

What a pilot is, and is not

A pilot is a small-scale, time-limited use of a tool in real conditions, to find out whether it will work before committing fully. It is related to a proof of concept, which demonstrates that an idea is feasible, but a pilot goes further: it tests the tool in real work with real users.

A pilot is not:

  • A free-for-all trial where everyone logs in once.
  • A demo, which is controlled by the vendor.
  • A formality before a decision you have already made.
  • A full rollout with a few people.

It is a structured experiment with questions to answer.

Decide the questions first

Before you start, write the questions the pilot must answer. Draw them from your requirements.

Examples:

  • Can the team complete the core workflow without help?
  • Does it handle our real volume and data?
  • Does it connect to our existing systems as needed?
  • Can administrators manage it with reasonable effort?
  • Do people prefer it to the current way of working?
  • Does it meet our security and data requirements in practice?
  • Is the vendor's support good when we need it?

Rank them. The pilot's job is to resolve the biggest uncertainties, the places where you are least sure.

Set success criteria in advance

Decide what a good result looks like before you see the results. Otherwise it is too easy to explain whatever happens.

Make criteria specific and measurable.

  • "At least eight of ten pilot users complete the core task unaided by the end of week two."
  • "The monthly summary can be produced in under ten minutes, compared with the current half-day."
  • "No more than two critical issues remain unresolved at the end."
  • "Average satisfaction rating of the pilot group is at least four out of five."

Choose a few. Record them. These become your decision rule.

Choose the pilot group

The right group makes the difference.

Small. Five to ten people is plenty for most tools. More adds confusion.

Representative. Include different roles, levels of experience and working styles.

Willing, and sceptical. Include an enthusiast who will try everything, and a sceptic who will say what is wrong. Avoid a group made only of fans.

Doing real work. The pilot should replace a real task, not run alongside as an extra.

With a lead. One person owns the pilot, keeps the schedule and collects feedback.

Tell the group plainly what the pilot is, how long it lasts, what you expect of them and how their feedback will be used. Change management, the discipline of helping people through a change, begins here: people who understand why are more likely to engage.

Agree terms with the vendor

A pilot should have simple terms.

  • A fixed period, with a start and end date.
  • Access for the whole pilot group with the right features, matching what you would buy.
  • A named contact at the vendor for questions.
  • Support at the level you would expect as a customer.
  • Data handling. What data you will use, where it lives and what happens at the end. Use anonymised or low-risk data unless you have assessed the risk.
  • No obligation to buy, and no automatic conversion to a paid plan. Check the sign-up terms carefully so a pilot does not roll into a contract.
  • Pricing preview. The quote you would receive if you proceed, in writing.
  • Exit. Export of pilot data on request.

Ask for what you need. Reputable vendors expect it.

Prepare the ground

A week of preparation pays for itself.

  • Set up the tool according to your requirements: fields, roles, permissions.
  • Create realistic data. Import a small, anonymised sample of real work.
  • Configure sign-in the way you would in production, including single sign-on if it matters.
  • Write short instructions for the pilot group: how to log in, what to do first, where to ask.
  • Hold a short kick-off, thirty minutes, to explain goals and answer questions.
  • Plan the schedule: check-ins, a mid-point review and an end review.
  • Set up a feedback channel: a shared note, a form or a chat channel.
  • Record a baseline: how long key tasks take now, how many errors occur, how people feel about the current process.

The baseline is crucial. Without it, you cannot tell whether the tool improved anything.

Run it

Week by week

Week one: learn. People find their feet. Expect questions and some frustration. Collect first impressions and fix set-up issues quickly.

Week two: work. Use the tool for real tasks. Capture time and effort. Note workarounds.

Week three: stretch. Test edge cases, integrations and admin tasks. Try something that goes wrong.

Week four: judge. Gather final feedback, measure against the criteria and prepare the decision.

For a longer cycle of work, extend the pilot to cover at least one full cycle, such as a month-end.

Keep the rhythm

  • Short check-ins, twice a week at first, then weekly. Fifteen minutes.
  • A visible log of issues, with status.
  • Quick answers. If people wait days for help, the pilot becomes a test of your patience.
  • Gentle reminders to use the tool for real work.

Test the awkward parts

Plan scenarios that stress the tool.

  • Add and remove a user.
  • Change a permission.
  • Import data with errors.
  • Export everything.
  • Use it on a phone.
  • Use it with only a keyboard, or with assistive technology if anyone on the team relies on it. Accessibility problems found in a pilot are far cheaper than those found after rollout.
  • Contact support with a real question, at a busy time.
  • Simulate someone leaving the team.
  • Check how it behaves when the connection is slow.

These are the situations a demo avoids and a year of use will find.

Collect evidence of several kinds

Numbers

  • Tasks completed.
  • Time per task, compared with baseline.
  • Errors or rework.
  • Number of support requests.
  • Login frequency, and what people did.

Words

  • Short weekly notes from each pilot user: what worked, what annoyed.
  • Quotes that capture feelings.
  • Observations from watching someone use it.

Ratings

  • A simple score at the end: how likely would you be to want to keep using this? A one-to-five scale is enough.
  • A scale for ease of use.

Vendor behaviour

  • Speed and quality of support.
  • Honesty about problems.
  • Delivery of promised fixes.

Numbers say how much, words say why, and vendor behaviour says what life as a customer will be like.

Beware of pilot traps

The honeymoon. Early enthusiasm fades. Keep the pilot long enough to pass it.

The expert effect. A vendor's helper who does the hard bits for you makes the tool look easier than it is. Ask what you would have to do alone.

The special account. Features not in the plan you would buy. Check what is included.

The tiny sample. If two people do all the work, you have learned about two people.

The ignored baseline. Without before and after, "feels faster" is a guess.

The sunk-cost nudge. After weeks of effort, there is pressure to say yes. Stick to your criteria.

The unmeasured cost. Set-up and training time counts. Record it.

The decision meeting

At the end, hold a short meeting with the pilot lead, representatives from the group and the decision-maker.

  1. Review the criteria. Which were met, partly met, not met?
  2. Review the numbers against baseline.
  3. Read the words. The main themes, positive and negative.
  4. Review vendor performance.
  5. List the deal-breakers and whether any occurred.
  6. Estimate the real cost, including set-up, training and administration.
  7. Decide: proceed, proceed with conditions, extend or stop.
  8. Record the reasons.

If you proceed, negotiate with the knowledge you now have, and carry over lessons into the rollout plan. If you stop, tell the vendor why, and thank them. If you extend, define precisely what needs to be resolved.

If a free trial is all you get

Some vendors offer only a short self-serve trial. Make the most of it.

  • Write the script before you start.
  • Use the trial for a defined, high-stakes scenario.
  • Invite three or four colleagues to test different roles.
  • Test export and sign-in first.
  • Ask for an extension if you need more time.
  • Check the trial's terms for automatic conversion.

After the pilot

Whatever the decision, keep the records. The pilot has produced a lot of knowledge about your own process. Even if you reject the tool, the baseline and the requirements are valuable for the next candidate.

If you proceed, thank the pilot group. They will be your champions in the rollout, and the next article in this series covers making a tool stick.

On this site

The categories and search pages help you find tools to pilot, the launches page shows new entrants and the submit page is where vendors list a product. The blog has more guides for buyers and teams.

The short version

A pilot is a small, time-limited, structured trial with real users on real work. Write the questions and success criteria first, choose a small representative group including sceptics, agree simple terms with the vendor, record a baseline, run it over a full cycle with regular check-ins, test the awkward parts and gather numbers, words, ratings and evidence of vendor behaviour. Decide with your criteria, not your mood.

Questions and answers

What is a software pilot?
A time-limited trial with a small group using a tool for real work, to test whether it meets your needs before a full commitment.
How long should a pilot last?
Long enough to cover a full cycle of the work, often three to six weeks, with a fixed end date.
How many people should take part?
A small, representative group, perhaps five to ten, including people sceptical of the change.
What should a pilot measure?
Whether the must-have requirements are met, how long tasks take, how people feel about using it and what problems arise.
What if the vendor offers only a short free trial?
Ask for an extension or a paid pilot with clear terms, or run the trial against a prepared script to make the most of the time.

Sources

Related guides

Need help?