What is Null Hypothesis Significance Testing?

One of the questions of the 2004 PS115 Exam was: The p-value for a Pearson correlation is a probability. What is it the probability of? The variety of answers
indicated that there is a lot of confusion about exactly what a test of significance is telling you. This is hardly surprising, as the logic of significance testing can be tricky
to understand. This is not helped by the fact that textbooks and other sources often incorrectly describe what can be concluded from a significance test (Gigerenzer,
1993). Nevertheless, you will benefit from reading about this in different sources, where the process of hypothesis testing is well described.
Why is this important? Several reasons. If you can understand the logic of testing for statistical significance you will be in a better position to understand and to criticise
the work that you will be required to read on your degree course. You will also be better able to report your own results. Many of the mistakes that students make in
reporting their results in a lab report would be eliminated if they had a better understanding of the process of significance testing.

The Statistical Process
1a. State the null hypotheses.
1b. State the alternate hypothesis.
2. Calculate the test statistic from data.
3. Calculate the p-value (associated probability)
for the obtained value of the test statistic
4a. If the p-value is less than .05, declare that
there is a statistically significant effect.
And reject the null hypothesis, and accept
the alternate hypothesis.
4b. If the p-value is greater than .05, declare that
there is no statistically significant effect.
And do not reject the null hypothesis.
An example t-test for independent groups
H0: There is no difference between the means
of the two groups.
HA: There is a difference between the means of
the two groups.
Obtain the t-value, which takes into account
the (1) difference between the means, (2)
standard deviations, (3) sample sizes.
Obtain the p-value for the t-value you have
obtained.
This is the probability of a t-value at least as
large as the one obtained if H0 is true.
If p < .05, there is a statistically significant
difference between the means.
i.e., Reject H0, and accept HA
If the p > .05, there is no statistically
significant difference between the means.
i.e., Do not reject H0
Commentary (specific to the example)
Assume that we have drawn two samples from the
same population.
(Both hypotheses are statements about what could
be true for populations of scores.)
The t-value evaluates the evidence for a difference.
The stronger the evidence, the greater the magnitude
of the t-value.
In general, the larger the t-value the smaller the p-
value.
The p-value tells us the probability of obtaining
evidence for a difference as least as strong as the
evidence that we have obtained if H0 is true.
Roughly speaking, this strength of evidence is
unlikely to be found if H0 is true. We have sufficient
evidence to decide that H0 is not a good hypothesis,
and are prepared to accept that there is a difference
between the groups.
Roughly speaking, this strength of evidence is not
uncommon if H0 is true. We have insufficient evidence
to declare that there is a difference between the
groups.

Ref: Gigerenzer, G. (1993). The Superego, the Ego, and the Id in statistical reasoning. In G. Keren & C. Lewis (Eds.), A Handbook for Data Analysis in the Behavioral Sciences: Methodological Issues. Hillsdale, NJ: LEA.
 Null Hypothesis Significance Testing - What it is NOT!

There are a number of incorrect beliefs (fallacies) or confusions about Null Hypothesis Significance Testing
that one hears quite commonly. Three are described below. Please avoid these errors!

The Replication Fallacy is the erroneous belief that the p-value of the significance test tells you something
about how likely it is that a finding will be replicated if the study were repeated.
The p-value does NOT tell you how likely it is that a finding will be replicated.
Therefore the following are all INCORRECT responses to the question: The p-value for a Pearson correlation
is a probability. What is it the probability of?
The probability of finding no relationship, if we were to examine a new sample of scores.
The probability that the same Pearson coefficient would be obtained, if we were to run the test again.
The probability of obtaining a correlation coefficient as strong, is another sample were taken from the
same population of scores.
At the .05 level, we are correct 95% of the time.

The Confidence Fallacy is the erroneous belief that the p-value of the significance test tells you the probability
that the null hypothesis is correct (or that it can be used to infer the probability of a true effect).
The p-value does NOT tell you how confident you should be that the null hypothesis is true (or false).
Therefore the following are all INCORRECT responses to the question: The p-value for a Pearson correlation
is a probability. What is it the probability of?
The probability that the correlation has been obtained by chance.
The probability that the null hypothesis is true.
The probability that there is no actual relationship.
The probability of rejecting the null hypothesis.
The probability of getting a correlation by accident.
The probability that the result is genuine.
The probability of a linear relationship between these variables.
The probability that there is some correlation between two variables.
At the .05 level, we are 95% sure that the two variables correlate.

Significance Confusion arises when the specialist terms associated with statistical inference are used incorrectly.
For instance, the phrase statistically significant refers to a cut-and-dried or definite decision about our data. If
we say that there is a statistically significant correlation, this means that our decision is that there is some
correlation between the variables. It makes no sense to say that the p-value tells us how likely the finding is to
be statistically significant.
Therefore the following are all INCORRECT responses to the question: The p-value for a Pearson correlation
is a probability. What is it the probability of?
The probability of rejecting the null hypothesis.
The probability of a non-significant relationship.
The probability that it is statistically significant.

Suggested Reading
Greer, B., & Mulhern, G. (2002). Making Sense of Data and Statistics in Psychology.
Read Chapter 6 skim through pages 71 to 88, then read pages 88 to 93 carefully.
Then read the sections on the t-test on pages 103 to 110, and on testing for significance (with
correlations) on pages 131 to 134.
Howell, D. C. (2004). Fundamental Statistics for the Behavioral Sciences (4th edition).
Read Chapter 8, pages 136 to 160 [fourth edition: pages 118 to 138].