The importance of what it is not said

Scenario #1: the magic coin

Someone flips a coin four times and obtain the following values:

{Head, Tail, Head, Head}

Based on the outcome of the above experiment he reports:

My coin is magic: it has a 3/4 probability of Heads, and 1/4 of Tails.

 

Scenario #2: the voting visionary

Next weekend election will be held in your country. There are two candidates: A, and B. Someone interviews ten people and obtain the following voting intentions:

{A, B, B, B, A, A, A, A, A, B}

Based on the above he reports:

The result of the election will be A: 60%, B: 40%.

Scenario #3: the peak usage

You measure the system usage at the peak hour during three days. The measured values are:

{70%, 70%, 85%}

Based on the above you report:

The average system usage at the peak hour is 75%.

Think

Surely you consider that the man in the coin and the voting scenarios is silly, or utterly ignorant at least, because is making bold predictions based on a very small empirical evidence. However you probably accept with no objections what is telling the man (btw, you!) in the system usage scenario, but…

…THINK!

There is nothing significantly different in the three scenarios. So if there is something wrong with the coin and the voting case, it may be the same for the peak usage case.

 

The confidence interval

When you try to estimate a certain quantity based on a limited number of measurements the reported value is affected of what is called the statistical error. This is inherent to any product of sampling. This statistical error is not an “error” in the common sense of the word, but instead it express the precision or reliability of the reported figure. Instead of having a value, you have an interval, that is, you MUST say:

The value is likely to be between this and that.

The length of this interval corresponds to the imprecision / uncertainty. If the length is small, the precision is high. If the length is large, the precision is low. Clearly it is not the same to say

The head probability is likely to be between 0.2 and 0.8.

a very low precision determination, than to say

The head probability is likely to be between 0.499 and 0.501.

a much higher precision one.

The confidence interval is the official name, in maths or statistics, for such an interval.

The degree of “likelihood” (…is likely to be…) is quantified by what is known as confidence level. Typical values are 90% or 95%. The loose meaning of a 90% confidence level is is that there is a 90% probability that the true value –the one we are trying to determine- lies within the interval (and, consequently, that there is a 10% probability that it falls outside).

The confidence interval is centered around the sample mean, the arithmetic average of the data (the measured values):

Center of the confidence interval =  AVERAGE(data)

where data is the sample, or set of measured data points.

The confidence interval length is typically estimated with the following formula:

Length of the confidence interval =2* T.INV.2T(1-conf, size-1)*STDEV.S(data)/SQRT(size)

where size is the sample size (number of data points), and conf is the desired confidence level, and T.INV.2T(), STDEV.S() and SQRT() are worksheet (Excel) functions. (I’m not focusing here in the details,  I’m paying more attention to its consequences and dependencies than to the formula itself).

This length depends on the following factors: 

•  the sample size or number of data points: the length decreases, and the precision increases, when the number of data points increases.
•  the sample variability/dispersion: the length increases, and the precision decreases, when the data is noisy, erratic, highly variable. 
•  The confidence level: the lenght increases when the confidence level increases. Reason? If you want increased certainty in your report, more “safety” margin is needed.

To have a better idea on this, look at the following table showing the approximate confidence interval length versus the sample size for a variable that can go from 0 to 100, and for a confidence level of 90%. To achieve a precision of 3.5% you need around 1000 measurements!

Size	Length
5	64
10	38
20	25
30	20
100	11
1000	3.5

Let’s go back and revisit our scenarios, but now equipped with the above ideas and guidelines.

Scenario #1: the magic coin (REVISITED)

Someone flips a coin four times and obtain the following values:

{Head, Tail, Head, Head}

Based on the outcome of the above experiment it MUST be reported:

My coin has a head probability between 0.3 and 1 with a confidence level of 90%

If you want to increase the precision, that is, reduce the confidence interval length, you must increase the sample size, that is, the number of flips. With 100 flips you are going to obtain something like:

My coin has a probability of heads between 0.45 and 0.55 with a confidence level of 90%

Scenario #2: the voting visionary (REVISITED)

Next weekend election will be held in your country. There are two candidates: A, and B. Someone interviews ten people and obtain the following voting intentions:

{A, B, B, B, A, A, A, A, A, B}

Based on the above it MUST be reported:

The result of the election with a confidence level of 95% will be:

 A between 30% and 90% and B between 10% and 70%.

This has changed a lot from the initial bold prediction; it is much blurred now. Typical opinion polls to estimate the true percentage of vote with reasonable precision and a confidence level of 95% require a  sample size of around 1000 people. Have a look on the small letter next to the results when you see such a study in your newspaper.

 

Scenario #3: the usage peak (REVISITED)

You measure the system usage at the peak hour during three days. The measured values are:

{70%, 70%, 85%}

Based on the above you MUST report:

The average system usage at the peak hour is between 65% and 85% with a confidence level of 90%.

If you want to increase the precision, that is, reduce the confidence interval length, you must increase the sample size, that is, the number of dayly measurement. With 20 days you are going to obtain something like:

The average system usage at the peak hour is between 74% and 78% with a confidence level of 90%.

Things to remember

• There is an unavoidable uncertainty in your measurements and calculations.
• Any estimation based on sampled data or limited number of measurements must/should be accompanied with the precision/uncertainty
• Avoid too small samples. The sample size should be large enough to obtain a reasonable precision.
• What it is not said, the error, is usually as important as the reported value itself.
• There are many marketing tricks that do not tell the whole story and deliberately hide the estimation error. Do not act like a malicious or ignorant people creating those marketing messages.

In the next contribution I’ll have a closer look to the system usage case. Stay tuned.