Nobody sets out to mislead themselves with a chart. It usually happens because a chart confirmed something someone already wanted to believe, and the natural next step, checking whether the chart actually supports that conclusion, got skipped in the rush to share good news. Here are the five questions I ask myself before trusting a chart enough to act on it, in the order I ask them.
1. Does the axis start at zero, and does it matter here
This is the most well-known chart trap for a reason: it works. A bar chart where the y-axis starts at 90 instead of 0 can turn a 3 percent difference into something that looks like a 30 percent difference at a glance. Line charts are more forgiving of this than bar charts, since the eye reads a line's slope reasonably well even on a truncated axis, but it is still worth a second look whenever the stakes are high enough to matter.
2. What is the sample size behind this percentage
"Conversion rate improved by 40 percent" sounds dramatic. If it means two purchases instead of one, out of a small test group, it is closer to noise than a finding. I try to have a habit of asking for the raw counts behind any percentage before reacting to the percentage itself. As a rough rule, I get more skeptical below 30 to 50 underlying events, since a handful of outcomes can swing a percentage wildly in either direction.
3. Was this date range chosen before or after looking at the data
A chart covering "the last 90 days" is a normal, defensible default. A chart covering "March 14th through June 2nd" should raise a small flag, because those specific boundaries were often chosen after someone already looked at the data and picked the window that told the best story. This is not always dishonest. Sometimes there is a real reason, like a product launch on March 14th. It is worth asking the reason out loud rather than assuming.
4. Is this comparing like with like
Year-over-year comparisons hide seasonal effects, holidays that fall on different weekdays, and one-time events, unless you check for them. Before trusting a comparison between two periods, I ask what else changed between them besides the thing being measured. A retail client of ours once flagged a scary 15 percent drop in a metric that turned out to be almost entirely explained by Easter falling in a different month that year.
5. What would this chart look like if the answer were "no change"
This is the question I find most useful and least commonly asked. Before looking at a chart's actual result, imagine what a chart showing no meaningful change would look like, given normal week-to-week variation. Then compare the real chart against that mental baseline, rather than against your hope for what it will show. If the "real" pattern is not obviously different from ordinary noise, it probably is not the finding it first appears to be.
The goal is not to distrust every chart you see. It is to reserve your trust for the ones that hold up once you ask a few basic questions, and to be honest with your team when one does not.
- Why do truncated axes make a chart misleading?
- When a bar chart's y-axis does not start at zero, small differences between bars get visually exaggerated, making a modest change look dramatic. Line charts are less affected by this than bar charts, since a line's slope is easier to read correctly even with a truncated axis.
- What is a reasonable sample size before trusting a percentage change?
- There is no single universal number, but as a rough habit, be cautious about percentage changes calculated from fewer than 30 to 50 underlying events, since small counts make percentages swing wildly from a handful of outcomes.