How to Evaluate an External CTO Before You Sign the Contract

What does not work

The default evaluation for an external CTO is a 60-minute call. The founder asks about their experience, they tell impressive stories, the founder feels good about the conversation, and a contract is signed. Six months later it is not working and nobody is sure why.

The reason: a 60-minute conversation tests for charisma and storytelling, not for the skills the engagement actually requires. Here is what tests for the real thing.

Step 1: A paid trial scope

Before signing a multi-month contract, hire them for a small piece of bounded work. Two to three weeks, fixed scope, fixed fee. Typical trials:

Review the architecture and produce a 5-page assessment.
Run the audit phase of the 90-day playbook.
Draft a hiring rubric and screen the first 10 candidates for a role you are filling.

The cost of a trial like this is $3k–$8k. The signal you get is enormous. You see how they think, how they write, how they handle questions they do not have the answer to, and how they hand off work to you.

Step 2: Two reference calls — the right ones

Founders ask for references and get sent two happy clients. This is useful but limited. Ask instead for:

A former client where the engagement ended. Not because something went wrong, just an end. Ask them: "What did the CTO do that made the engagement valuable? What did they not do that you wish they had?"
An engineer who reported to the CTO at some point in their career. Senior or junior. Ask: "What was the CTO like to work with technically? Did they make decisions clearly? Did they push back when they should?"

The first reveals what the CTO actually delivers. The second reveals how they behave on the inside, not in sales mode.

Step 3: A real code review session

If you have a codebase, walk them through it on a screen-share. Ask them to identify three things they would change in the first month. Watch:

Do they ask before they answer? Good seniors ask questions that reveal context before pronouncing judgment.
Do they prioritize? Most codebases have ten things wrong; a good CTO can name the two that actually matter and explain why the others are deferred.
Do they explain themselves at the right level? You are not technical enough to follow a deep dive, and a good CTO calibrates to that without dumbing it down.

A CTO who walks through this and produces actionable, founder-friendly takeaways is showing you exactly what the next 12 months will be like.

Step 4: The "what would you not take on" question

In the final conversation, ask: "Of the things you have done in your past CTO roles, what are the things you would not take on at our stage and why?"

The honest CTO will name 2 or 3 things they have learned to decline — usually because they take too much senior time relative to value, or because the founder cannot use the output. The CTO who says "I would take on anything" is selling to you, not advising you.

Step 5: Contract structure

Once you have signed, the contract should specify:

Time commitment per week and what happens if you need more.
Decision rights (what they can do without your sign-off).
A defined first-90-day output set (see the 90-day playbook post).
A 30-day off-ramp on either side.
Confidentiality and an IP assignment that names your company.

A CTO who pushes back on writing any of these specifics is signaling they want flexibility you do not. The CTO worth hiring will help you tighten the contract because clear expectations are what makes the engagement work.

The pattern

Evaluating an external CTO is not about finding the smartest person. It is about finding the senior most likely to translate their experience into your specific situation. The five steps above test for translation, not for résumé.