Statistical Significance: What Do Those Little Letters Mean?
I don’t pretend to be a statistics expert. In all honesty, if you came to this post looking for how to calculate statistical significance, you would be much better served here, or here, or here. However, if you were trying to figure out what those little letters mean next to the percentages in your survey results, and you didn’t want to ask your boss, this article is for you.
As you probably guessed by the title, the short answer is “statistical significance.” The long answer, and how to interpret the letters themselves, is below.
Statistical Significance, What Is It:
In very basic terms, statistical significance indicates that your results aren’t attributable to chance. In other words, if you did the same survey again, you would expect the results to be the same. I have found that in market research we most often use statistical significance to determine whether the difference between two responses are reliably different.
One way this is used is in concept testing. Statistical significance can be used to determine whether the difference in two concepts’ T2B or WPI scores are actually different or simply different by chance. Similarly, we may also look at whether the opinion responses of two different consumer target groups are meaningfully, repeatably different, or not. Using statistical significance testing we can be more confident and objective in our recommendations.
I Get Statistical Significance, But What Do the Letters Mean?
Alright, so let’s pretend you get your concept results back and they look something like this:
Concept A | Concept B | Concept C | Concept D | Concept E | |
Top-2-Box | 40 | 45 | 60AB | 50A | 65ABD |
In this case, the letters indicate that there is a statistically significant difference between two columns. In this example, the difference between Concept “C” and Concept “A” is significant. Concept “C” had the greater result than Concept “A.” This is signified by the “A” next to Concept “C’s” T2B score of 60. Concept “C” is also statistically significantly larger than Concept “B,” thus the “B” next to the “A” next to 60.
Conversely, the difference between Concept “C” and Concept “D” is not a difference of statistical significance. If you tested those two concepts again using the same methodology it’s possible their scores could be the same since the 5-pt difference between them is more likely to be due to chance. Therefore, you do not see a “D” displayed next to the 60 even though 60 > 50 as a raw number.
Great! I Get Statistical Significance With Capital Letters, But Some of Mine are Lower-Case!
Yep. Sometimes you get the data back and the results look like this:
Concept A | Concept B | Concept C | Concept D | Concept E | |
Top-2-Box | 40 | 45a | 60ABD | 50Ab | 65ABcD |
Now we have mix of capital and lower-case letters next to our data. The short answer is capital letters are best. The long answer is, it has to do with the confidence level of the test. In some cases your business may want statistical significance tested at a minimum confidence level and a desired confidence level. For argument, let’s say the minimum-level is 90% and the desired level is 95%.
The confidence levels simply indicate the percent likelihood that the difference being describe is chance. Nothing is certain when doing surveys. At a 95% confidence level, there is a 5% chance that the results are attributable to chance. Said another way, you can be 95% confident that the results are meaningfully and repeatably different. That’s pretty high. Depending on how you aligned your success criteria for the test, your business partners may be satisfied with a lower confidence level in some cases and you need to report both.
When you see a mix of capital and lower-case letters it means two different confidence levels are being reported. In the example above, the lower-case letters indicate a 90% confidence level and the capital letters indicate a 95% confidence level. Obviously, it is better to rely on the data with the highest confidence level.
Conclusion:
As we’ve mentioned before, market researchers enter the industry from a lot of different backgrounds and fields of study. If you came from a Liberal Arts background, you may not be familiar with this kind of notation. Good for you for noticing it, and trying to figure out its meaning. Netizen Insights is designed to be a safe place for new market researchers to go to learn the basics of the industry. We’re proud of you for searching for help, and glad you came to us. If there are other ways we can help, just add them in the comments below.
As soon as I found this internet site I went on reddit to share some of the love with them. aeedackgfcdd
This was so helpful! So does that mean that Concept A is not statistically significant because it doesn’t have a letter next to the number?
Thank you!
Hi Kara — not quite. It means that Concept A’s score of 40 is NOT statistically LARGER than any of the other concepts’ results. Concept C (60ABD) is statistically significantly larger than Concepts A, B, and D at 95% confidence. It is not statistically significantly larger than Concept E at either 90% or 95% confidence. Concept A, the lowest scoring concept, didn’t receive a statistically significant win v. any of the concepts in the example. Thus, no letters next to its score of 40. On the other hand, Concept A is statistically significantly SMALLER than Concepts C, D, and E at 95% confidence (which is indicated by the capital “A” appears next to those scores) and is statistically significantly SMALLER than concept B (which is indicated by the lower-case “a” that appears next to its score. Thanks for checking out our site and for reaching-out!