What Separates Strong Analysts from Average Ones
Technically, most analysts can write SQL and build dashboards. The differentiator is reasoning quality: do they notice when a dataset has a problem, do they ask whether the analysis actually answers the question being posed, and do they know when the data doesn't support a conclusion?
An analyst who produces confident outputs from flawed data is worse than no analyst at all. The first thing to evaluate is their instinct for skepticism.
Core Evaluation Dimensions
Data Skepticism: Given a real dataset with a known issue (duplicates, selection bias, measurement gap), do they find it? Ask them to QA an analysis before accepting it — not just run it.
Reasoning Quality: Can they explain causal reasoning vs. correlation clearly, without jargon? Can they identify when a trend is driven by a single cohort or segment hiding in the aggregate?
Communication of Uncertainty: Do they hedge appropriately, or do they present every result as definitive? The ability to say 'this data doesn't cleanly answer the question' is rare and valuable.
Business Translation: Analysis that doesn't connect to a decision is a report, not an asset. Look for evidence that their work actually changed what the business did.
Assessment Design
A more effective assessment than a standard SQL test: give a candidate a data question that requires both SQL and interpretation. Ask them to write the query, interpret the result, and then identify the limitations of their own analysis.
This three-part structure reveals SQL skill, reasoning quality, and epistemic honesty simultaneously — in about 45 minutes of asynchronous work.
Reference Check Focus
Ask references specifically: 'Did their analysis change decisions?' and 'Were they comfortable telling stakeholders when the data didn't support what they wanted to hear?' An analyst who always delivers validating results is either very lucky or is optimizing for approval over accuracy.