You can fill a week reading resumes that all say the same thing. LangChain. RAG. PyTorch. A fine-tuned model “in production.” The screening call is fluent. Then the first real ticket arrives, and you get a notebook, an unbounded tool call, and no way to tell whether the answer is grounded. The person can talk about generative AI but they cannot own a feature.
I take a side here. If you want to hire AI developers who will still be useful after the demo, put a work sample that looks like the job in front of the resume. Do it before a charming narrative takes over the loop. Personnel selection research has pointed in this direction for decades. The current market makes it more urgent, because almost every candidate can now produce a polished chat transcript.
Demand is real, and it has flattened the signal on paper. LinkedIn’s Jobs on the Rise 2026 list of the 25 fastest-growing roles in the U.S., published 7 January 2026, opens with AI engineers (also called machine learning engineers). The most common skills LinkedIn lists for that title are LangChain, retrieval-augmented generation (RAG), and PyTorch. Those words on a resume are the baseline now. They are a weak way to rank two people who both want the job.
The 2026 Stack Overflow Developer Survey points the same way from the builder side. Among 17,464 respondents to the AI tool question:
83% use at least one AI tool. Coding assistants (66%) and general chat tools (63%) dominate; agents/workflows reach 26%.
The underlying table rounds to 65.9% and 62.5%. A candidate who “uses Copilot” is describing the default toolkit.
We already wrote about the resume-to-production gap in The AI Talent Shortage: What’s Actually Happening. This piece is the buying guide that follows: how to hire AI developers, how to hire generative AI engineers who can work past a demo, and how a U.S. product team can hold a serious bar when part of the work sits with an India delivery team.
What the research says about work samples
Frank L. Schmidt and John E. Hunter’s 1998 review, The Validity and Utility of Selection Methods in Personnel Psychology (Psychological Bulletin, 124(2), 262-274), summarized 85 years of findings on how well 19 selection methods predict job performance. Their Table 1 put the mean operational validity of work sample tests at .54, ahead of general mental ability at .51 and structured interviews at .51, and well ahead of unstructured interviews at .38. They defined the method in plain language:
Work sample tests are hands-on simulations of part or all of the job that must be performed by applicants.
Combined with a general mental ability measure, “GMA plus a work sample test” reached a “mean validity of .63.” They also wrote that “the average validity of the structured interview is .51, versus .38 for the unstructured interview.”
Those coefficients were the slide that hiring decks repeated for twenty years. They should not be repeated as if the debate ended in 1998. In Revisiting meta-analytic estimates of validity in personnel selection (Journal of Applied Psychology, 2022), Paul R. Sackett, Charlene Zhang, Christopher M. Berry, and Filip Lievens argued that common range-restriction corrections had overstated many of those validities. In a 2023 follow-up for practitioners, Revisiting the design of selection systems, the same authors reported that work sample validity fell by .21 relative to Schmidt and Hunter (from .54 to .33), cognitive ability by .20 (from .51 to .31), and unstructured interviews by .19.
Structured interviews emerged as the predictor with the highest mean validity.
That mean is .42, with an 80% credibility interval “ranging from .18 to .66.” The procedures that now sit at the top of their list are “those specific to individual jobs, such as structured interviews, job knowledge tests, work sample tests, and empirically-keyed biodata.”
Two practical conclusions survive the revision. First, a work sample is still a strong, job-specific method, and it should look like the work. Second, the work sample is stronger when you pair it with a structured interview and a written rubric. A take-home treated as a lone high-score contest is a thinner signal. Sackett and colleagues are explicit that means hide a lot of spread, and that “an investment in a custom-designed selection system” beats an off-the-shelf puzzle in the average case.
Google’s public hiring guidance lands in the same place. In a 2015 WIRED excerpt from Work Rules!, Laszlo Bock, then Senior Vice President of People Operations, wrote:
The best predictor of how someone will perform in a job is a work sample test (29 percent).
That “29 percent” is the squared validity of Schmidt and Hunter’s .54, presented as variance explained. He continued:
This entails giving candidates a sample piece of work, similar to that which they would do in the job, and assessing their performance at it.
And for Google’s own technical loop:
All our technical hires, whether in engineering or product management, go through a work sample test of sorts, where they are asked to solve engineering problems during the interview.
On interviews, Google re:Work is blunt:
Google uses structured interviewing, and we use the same interview questions, grade candidate responses on an identical scale, and make hiring decisions based on consistent, predetermined qualifications.
If you want the company-eye view before you redesign your loop, watch these two talks:
Schmidt and Hunter also noted a limit that still matters:
Work sample tests can be used only with applicants who already know the job.
That is the right constraint for experienced generative AI engineers. For earlier-career hires, keep the task smaller, drop any assumption of production war stories, and lean harder on a structured interview plus a short, supervised exercise. Do not replace the sample with a trivia quiz about model names.
A product-engineering bar, written down
Decide what “good” looks like before you read a single CV. I want this in a one-page bar, shared with every interviewer, in the U.S. and in India, for staff-aug and full-time candidates alike. If the bar lives only in one hiring manager’s head, you will hire the person who interviews like that manager.
For a product-facing AI role, I score four things.
Production judgment. Score whether they can take a messy source of truth (documents, an API, a log, a CRM object) and design a path that has a schema, a failure mode, and a stop condition. Evals, logging, permissions, and cost should show up in the design without being dragged out of them.
Product sense. Score whether they ask who the user is, what “done” means, and which error is worse (a missed answer, or a confident wrong one). A good candidate can cut scope and still leave a shippable slice.
Evidence. Score work you can inspect: a repo, a design note, a trace, a failing test they added. Fluency on a sales call does not count as evidence.
Collaboration under a clock. Score whether they can explain a tradeoff to a product partner, take feedback on the sample, and revise. The 2026 Stack Overflow survey’s work data is a reminder that the job is already broader than typing code: among 24,714 respondents, 89.9% write or generate code, 66.7% review changes, 63.4% make design or architecture decisions, and 51.8% deploy or release software. A loop that only tests LeetCode-style puzzles is sampling a slice of that week.
Write the bar in the same language you will use on a rubric. “Strong,” “adequate,” and “below the bar” for each attribute, with a short example of each. Google’s re:Work advice is to document “what a poor, borderline, solid, and outstanding answer would cover.” Steal that shape. Do not steal their questions blindly; write yours from the job you actually need to fill.
Give them a sample of the job
The work sample should be a smaller version of next quarter’s work, time-boxed, and scored before anyone falls in love with the resume. A good shape for a generative AI engineer or product-minded AI developer looks like this.
The brief (about half a page). A user-facing task with constraints: a messy document set or internal API, a quality bar (grounded answers, no silent tool writes), and a business limit (latency, cost, or a human pause on mutating actions). Tell them what “done” means. Tell them what they may invent and what they must not.
The artifact (four to six hours, or a paid longer take-home). A small running path or a tightly scoped prototype, plus a short design note: what they would test, what they would not ship, how they would monitor it, and what they would do with two more days. If the take-home will take more than an evening, pay for the time. Unpaid weekend projects select for people who can donate labor, which is a different filter than skill.
The live follow-up (sixty to ninety minutes, structured). Walk the sample. Ask the same questions of every candidate. What failed in your eval? What would you refuse to automate? How would a reviewer replay this request next month? Where did you use an assistant, and how did you check its work? Score with the rubric, in writing, before the debrief.
What I would fail on sight. A demo with no evals. A chain of tools with a shared secret and no owner. A note that only lists models and vendors. A refusal to discuss cost, logging, or a human pause. A beautiful notebook that cannot be handed to another engineer.
What I would forgive. A prototype that is honest about limits. A candidate who cuts a feature and says why. Missing a framework fashion, if the path is clear and tested.
Here is a composite of a pattern we see when a resume and a work sample point in opposite directions.
Illustrative example, details changed/composite.
The first packet looked finished before anyone opened the sample. Two pages of LangChain, RAG, and PyTorch. A line that said a "production assistant" had been shipped. A blog post on chunking. A GitHub full of notebook titles. The screening call was smooth. They walked a retrieval diagram on a whiteboard and named the usual failure modes without being asked. Then the brief went out: a support assistant over a messy policy folder, answers had to show a source, no silent writes, and a human pause on anything that changed a ticket.
What came back was a polished walkthrough video and a single notebook. The retriever kept returning the same three documents. There was no eval set. The "cite the source" step printed a filename that was not the file the model had used. A write tool sat behind a shared key, with no owner and no confirmation step. On the rubric, production judgment and evidence scored below the bar. Product sense was only adequate: they had cut the chat UI down to one flow, which was fine, and then stopped. In the structured walkthrough they could not replay a single request. The panel ended the loop there.
The second packet was thinner. Several years of backend work, one quiet line about "some LLM features," no conference talks. The sample was a bit ugly. A small service. A folder of failing cases they had written themselves. A short note that said they would not auto-close tickets. A log line for every tool call. In the follow-up they pointed at the two cases that still failed and what they would do with another afternoon. The brief and the rubric were the ones used for the first packet. That one moved forward.
A buying sequence you can run this month
- Write the bar and the brief in one sitting with the hiring manager and a senior engineer who will work with the person. If those two disagree, stop. You do not yet know the job.
- Source more widely than the usual four cities. LinkedIn’s 2026 list still concentrates advertised AI-engineer jobs in San Francisco, New York City, and Dallas. A hybrid model is how a team outside those markets reaches people who can pass the sample.
- Screen lightly, then sample. A short call to confirm interest, logistics, and a genuine look at one past artifact is enough before you invest in the take-home. Do not run five unstructured “culture” chats first.
- Score independently, then debrief. Each reviewer writes a rubric score before the group meeting. The debrief is for conflicting evidence. Keep it from becoming a contest for the loudest interviewer.
- Check the vendor the same way you check a candidate. If you are choosing an AI development company or a staff-aug partner in the U.S., ask for the work sample they use internally, a named owner for the people on your account, and a knowledge-transfer plan. Volume of CVs is easy to buy. Setting a product-engineering bar takes more work.
If you already have a specialist in the building, protect them from becoming the entire interview panel and the entire production path. The work sample is how you add a second person who can share the pager, or a partner team that can be left with the feature when the contract ends.
That is the filter. It is slower than a resume sort, and much faster than a hire you have to unwind in month two.
FAQ
What should I look for when I hire AI developers?
Look for production judgment, product sense, inspectable evidence, and the ability to explain tradeoffs under a time limit. Tool names on a resume are a weak ranking signal. A work sample that resembles the job, scored with a written rubric and followed by a structured interview, is a stronger basis for the decision.
Why use a work sample instead of a standard coding interview?
Schmidt and Hunter (1998) found work sample tests among the strongest predictors of job performance (mean validity .54 in their table) and defined them as hands-on simulations of the job. Sackett and colleagues (2022) later lowered many of those coefficients after correcting over-adjustments, and found structured interviews at the top of the revised list (mean .42), with work samples still among the leading job-specific methods. A puzzle that does not resemble the work is a weaker sample of the job you need done.
How do I hire generative AI engineers who can work in production?
Give them a time-boxed task with messy inputs, a quality bar, and a constraint on tools, cost, or human approval. Ask for a small working path and a short note on evals, failure modes, and monitoring. In the follow-up, score how they reason about grounding, permissions, and what they would refuse to automate.
How should a US company work with an India and US hybrid team?
Keep product ownership and the quality bar in one place, and apply the same work sample and rubric to every engineer, regardless of location. Be explicit about overlap hours, documentation, and who can halt a release. Treat knowledge transfer as a deliverable, especially on a fixed-length augmentation.
What should I ask an AI development company in the USA before I sign?
Ask who will actually join the work, how those people were assessed, whether you can review a redacted work sample or rubric, who the named owner is, and how knowledge will move to your staff. A U.S. entity with a hybrid delivery team can be a good fit when the bar is shared. A large pile of similar resumes leaves the assessment work still to do.
How long should a take-home work sample be?
For experienced hires, four to six hours is enough to see judgment if the brief is tight. If you need more than an evening, pay for the time. A sixty to ninety minute structured walkthrough of the sample matters as much as the files they upload.
Sources
- Schmidt, F. L., and Hunter, J. E. (1998). The Validity and Utility of Selection Methods in Personnel Psychology. Psychological Bulletin, 124(2), 262-274.
- Sackett, P. R., Zhang, C., Berry, C. M., and Lievens, F. (2022). Revisiting meta-analytic estimates of validity in personnel selection. Journal of Applied Psychology.
- Sackett, P. R., Zhang, C., Berry, C. M., and Lievens, F. (2023). Revisiting the design of selection systems. Industrial and Organizational Psychology.
- Bock, L. (2015). Here’s Google’s Secret to Hiring the Best People. WIRED, 7 April 2015. Excerpt from Work Rules!.
- Google re:Work. A guide to structured interviewing for better hiring practices.
- LinkedIn News. Jobs on the Rise 2026: The 25 fastest-growing roles in the U.S.. 7 January 2026.
- Stack Overflow. Developer Survey 2026, AI tool usage and work data.
- Life at Google. How We Hire at Google.
- re:Work with Google. Laszlo Bock on how Google is changing the nature of work.
