You can tell whether a candidate can actually use AI by watching them do four things on a real problem: scope the task before they ask the model anything, check what comes back instead of pasting it, iterate when the first answer is wrong, and explain the result in their own words. Vocabulary proves nothing. Those four behaviours are hard to fake in twenty minutes, and they are what the job needs.
Why the usual signals stopped working
Two years ago "uses AI tools" on a CV meant something, because few people did. Now almost everyone does. The Stack Overflow 2025 Developer Survey found 84% of developers using or planning to use AI tools in their work. The same survey found 46% of them distrust the accuracy of what those tools produce, and 66% say their biggest frustration is output that is "almost right, but not quite".
Read those three numbers together and the hiring problem becomes clear. Nearly everyone uses the tools. Nearly half do not trust them. And the failure mode is not obvious nonsense but plausible, nearly correct work. The skill an employer is paying for is no longer "can operate the tool". It is "can tell when the tool is wrong, and can fix it". A portfolio of finished projects does not show that skill, because a finished project shows the outcome and hides the process. An interview question about prompt engineering does not show it either, because the answer can be memorised.
You have to watch the process.
The six tells
These are the behaviours that separate someone who can use AI from someone who can talk about it. Each is observable in a short live exercise, and each is something a candidate cannot prepare a script for.
1. They scope before they prompt. A capable person reads the problem, works out what "done" looks like, and decides which parts the model should touch. A weak one pastes the whole task into a chat window and hopes. Watch the first two minutes. If the candidate starts by restating the problem, naming the constraints, or asking you a clarifying question, that is the strongest early signal you will get.
2. Their prompts carry context, a goal and constraints. When Square 1 AI grades a prompt, it scores five dimensions: context, role and goal, constraints, specificity, and whether the output would actually work. A prompt that says "write a launch email for our product" scores near zero on all five. One that names the audience, the length, the tone, the one call to action and the things to leave out scores near full marks. You do not need our rubric to see the difference. You need to look at what the candidate typed.
3. They read the output before they use it. This is the tell that matters most and the one most candidates fail. Give them a task where the model will produce something plausible and wrong, which is easy to arrange, because it is what models do. Does the candidate run it, test it, check a number against the source, or read the code line by line? Or do they paste it and move on? Someone who verifies has the habit the job needs. Someone who does not will ship the "almost right" answer to your customers.
4. They iterate instead of restarting or surrendering. When the first answer is wrong, a capable person narrows the prompt, supplies the missing fact, or asks the model to explain its reasoning. A weak one either accepts the wrong answer, retypes the same prompt, or gives up and does it by hand. Iteration is where most of the real productivity gain lives, and it is a learned skill.
5. They know when not to use it. Ask them, mid-exercise, whether the model should be doing this part at all. A strong candidate will have an opinion: this is a lookup the model will hallucinate, this involves data we cannot send to a third party, this is faster to write by hand than to explain. A weak candidate has never thought about it.
6. They can explain the result without the tool. At the end, close the laptop and ask them to walk you through what they built and why it works. If the explanation is fluent, they understood it. If it collapses, the model did the work and they were the courier. This single question is the cheapest integrity check you have.
A twenty-minute exercise that surfaces all six
Pick a small, real task from your own work. Not a puzzle, not a whiteboard algorithm. Something like: "Here is a CSV export from our support tool. Tell me which three issue categories grew fastest last quarter, and draft the two-paragraph summary I would send to the product team."
Tell the candidate they may use whatever AI tools they normally use. Then watch, and keep notes against the six tells rather than against the answer. Two rules make this work:
- Plant one trap. Make sure the data has a quirk the model will get wrong, such as a category that was renamed halfway through the quarter. You are not testing whether they spot the trap. You are testing whether they check at all.
- Ask "how do you know?" once. Pick any claim in their output and ask how they know it is true. The answer tells you whether verification is a habit or a word.
Score the process, not the deliverable. A candidate who produces a slightly worse summary but caught the renamed category and can explain every number is the stronger hire.
What not to do
Do not ban the tools. Assessing someone without AI measures a skill the role will never use, and it selects for people who are good at a test rather than good at the work.
Do not rely on a certificate. A vendor certification proves someone studied a syllabus and passed a proctored exam. That is useful for some roles, and it is verifiable. It does not show scoping, verification or iteration, because multiple-choice exams cannot observe them.
Do not trust a portfolio you have not probed. Ask the candidate to change one thing in their own project, live. The process of making that change tells you more than the repository does.
Do not confuse fluency with capability. The candidates most likely to talk confidently about agents, RAG and context windows are often the ones who have read about them rather than used them. Vocabulary is the easiest thing to acquire and the least predictive.
What this looks like at scale
One interview at a time, the six tells are enough. Across a hiring pipeline, or across a team you are trying to upskill, you need the same observations recorded consistently, over repeated pieces of real work, so that you can see whether someone is improving rather than whether they had a good day.
That is the problem Square 1 AI is built around. Every project a learner submits is graded against a written rubric with criteria such as correctness, code quality and structure, documentation, git hygiene, evaluation and interpretation, and workflow and prompt craft. Every resubmission is kept, so the record is a trajectory rather than a single score. We do not yet publish outcome data, because we do not yet have enough of it to publish honestly, and we would rather say that than invent a number.
If you are hiring this quarter, you do not need any of that. You need twenty minutes, a real task from your own backlog, one planted trap, and the discipline to score the process instead of the answer.