Hypothesis Testing

As you continue to practice reading journal articles this week, you should begin to notice the basic format of these articles. Typically, they include an Introduction, Method, Results, and Discussion section. In the introduction, you will find the purpose of the study and from that you should be able to deduce the specific research question and corresponding hypotheses.

A clear link should be obvious between the purpose, the research questions, and corresponding hypotheses. The research question(s) is/are the specific question(s) that the researcher is attempting to answer by conducting the study. It is a narrowed focus from the broader problem or issue identified in the purpose. These are the actual questions that your methodology will help you answer. Research questions should clearly identify your independent and dependent variables for each specific question posed. For each research question there will be a null and alternative hypothesis statement.

The methodology of the study lays out the plan for testing your hypotheses. You always want to think about testing the null  hypothesis. The null hypothesis is the hypothesis statement that says that there is no difference between groups or no relationship between variables. Basically, null = nothing happened. Your method is designed to systematically test the hypothesis and collect accurate and reliable data that will tell us if we should accept or reject the null  hypothesis. The only time you ever consider your research (alternative) hypothesis is when the null hypothesis is rejected.

So, how do we know when to accept or reject the null hypothesis? The data collected using the method will be reported in the results section of an article. The results section is where you find the information you need to decide if the null hypothesis should be accepted or rejected.

It is important to understand the concept of the null & alternative hypothesis prior to interpreting inferential statistics, because the whole point of the statistical procedure is either accept or reject the null hypothesis. Most researchers use inferential statistics to assess differences between sample groups so that they can infer to a general population. Typically, when comparing data, the goal is to reject the null hypothesis so that the alternative hypothesis is supported. The statistical tests are simply comparing the means of the two groups you are comparing and using that information along with sample size and variability within the data to calculate the probability that any differences between group means are due to random error as opposed to an actual effect from the independent variable.

These results are often written in result summary statements such as t (20) = 3.135, p < .001. Another example is F(2, 40) = .232, p > .05. This is a standard format for written summaries of inferential statistics. The entire anatomy of these summary statements will be addressed later on. Today's lesson will focus on the p-value.

The p-value in the first example is < .001. The p-value in the second example is > .05. Sometimes the p-value will be indicated as equal to some number such as, p = .35. This number should be read as a probability. The p-value is the probability that the null hypothesis is true. (Memorize this statement.) The p-value is sometimes labeled "a" or "alpha" or "type I error". Remember that these labels all refer to the same probability: that the null is true. If the null hypothesis is true, then there is no significant difference between the group means you are comparing (Memorize this statement as well).

Suppose we conduct a study that compares student outcome scores on the STAR test between students who were taught using whole language versus students who were taught using phonics. Let's pretend we analyze the results and the computer tells us that we have an  F(3, 120) = .234, p = .56.

Just look at the value of p.

You should be thinking, "the value of p in this example is = .56."

Then you should convert this to a probability in your mind, "There is a .56 or 56% chance that the null hypothesis is true".

This is a fairly high probability that there is no difference between the group means. We can interpet this by saying there is no relationship between the variables. So we would accept the null hypothesis.

What does that mean? It is unlikely that there is any difference between achievement levels of students taught by whole language versus phonics as measured by group means obtained by giving the STAR test to all students. Any differences that are found between the two groups are likely due to random error.

What if our p-value had been = .02?

Then, you should have thought, "the value of p in this example is = .02."

Then you should convert this to a probability in your mind, "There is a .02 or 2% chance that the null hypothesis is true".

This is a fairly low probability that there is no difference between the group means. So we would reject the null hypothesis and claim that the alternative hypothesis was supported (never proven, just supported).

Type I Error

Type I error is the chance that you rejected a null hypothesis that was actually true. If you think about this, the Type I error rate and the p value are the same. They are the probability that the null hypothesis is true, even if you reject it. Type I error is the risk you are taking that you are wrong in claiming that the alternative hypothesis was supported.

Significance Cut-off Scores

Researchers decide a-priori, or prior to conducting the experiment, at which level they will accept or reject the null hypothesis. This is called a cut-off score and it sets a limit to the amount of risk you will take in claiming a false-positive result (Type I error). Typically researchers use arbitrary cut-off scores of .10, .05, or .01. The smaller the number, the more difficult it is to claim support for the alternative hypothesis, but also the less risk you take in making a false-positive mistake.

Lesson Activities

• Visit the Library Tutorial on scholarly journals versus popular magazines , and complete the online questionnaire to check your learning.  After you complete the quiz, save the results and upload the file to the dropbox for Week 5 in eLearning.  (Note: You can save the results by using the save as command on the file menu in your Internet browser.) If you are unable to save the results from your browser, copy and paste the completed quiz into a word document that can then be uploaded and save the confirmation message from your email program and upload that file to the Week 5 Lesson 1 dropbox dropbox as well.
• Go to the Student Papers link at the top of this page and read Sample 2.
  • Identify the Research Question, Null and Alternative Hypothesis.
  • Identify the statistical sentence in the results that addresses that question, and interpret the p value. (Focus on the MAIN Analyses)
  • Compare your answers with the model responses.  Correct any deficiencies.
  • Post your work in the Week 5 dropbox in eLearning.
• Go to the Student Papers link at the top of the page and read Sample 4.
  • Identify the Research Question, Null and Alternative Hypothesis.
  • Identify the statistical sentence in the results that addresses that question, and interpret the p value.
  • Compare your answers with the model responses.  Correct any deficiencies.
  • Post your work in the Week 5 dropbox in eLearning.

  Need Help?: This one is tricky because there are 2 tests going on. One that is a pretest versus posttest of 1 group. They do test the pre to post group means using a Paired Sample t-test and find a significant difference between the pre and post group means, rejecting the Null Hypothesis that the two group means were equal.

  You should know by now that this pre/post comparison by it self is not enough to have confidence in the intervention. Why? Because there is no comparison group. Therefore, the authors went further and compared their posttest data with the state average. However, instead of starting out with two equal groups and seeing if our intervention had an effect by making the group means significantly different from each other, in this study, the two groups start out different and the researcher's goal was to make them the same. Remember that the Null hypothesis ALWAYS tests the probablity of the two group means being equal. So, in this case we would want to accept the null hypothesis if the intervention had an effect, which is exactly what we do.

The importance of Research Designs and Validity

Internal validity has to do with our confidence that the manipulation we made in our independent variable caused the change we observed in our dependent variable. There are specific threats to internal validity that limit our ability to say that our manipulation directly affected the outcome. Some of these ideas are addressed in your Parsons & Brown text book. There are also specific techniques we can use to rule out these threats to internal validity such as using random assignment to comparison groups, using a comparison group, using a pretest and a posttest, etc.

External validity has to do with our ability to generalize our findings to others. If you conduct a study using one participant, you external validity will be low. It will be hard to make the case that what you found will apply to lots of people without also demonstrating the effect with more people. On the other hand, if you conduct a study using 1 million people, you would have high external validity because you demonstrated that the effect is present across a large sample of people and it will likely apply to others with similar relevant characteristics given the same set of circumstances.

Construct Validity has to do with our confidence that the measure we are using is really assessing the construct we set out to measure. For instance if you want to assess "paying attention in class" and you write an operational definition for remaining seated during class and another for looking at the assignments given during class, you may not really be measuring attention by using those two operational definitions. I could be sitting at my desk and looking at an assignment but not actually be reading the assignment or working on it. I may be writing a note to a friend in the corner of the worksheet, for instance.

Controlling for Threats to Internal Validity

The purpose of experimentation in general is to examine relationships between variables. The unique feature of experimentation is that it examines the direct influence of one variable (the independent variable) on another (the dependent variable). Drawing valid inferences about the effects of an independent variable requires attention to threats to internal validity. It is important to discuss threats to internal validity because they convey the reasons that carefully designed experiments are needed. 

Threats to internal validity include history, maturation, testing, instrumentation, statistical regression, selection bias, attrition, and diffusion of treatment.

Typically, the harder you work to increase your internal validity, the lower your external validity becomes, and vice versa. It is a balancing act and no study is perfect. However, if you don't have internal validity, then you don't have a study. So the priority of the two should always rest with maximizing internal validity. 

There are several strategies that can help boost validity such as using

• Randomly selected samples
• Comparison groups
• Random assignment to comparison groups
• Pretests and posttests
• Experimental designs

The links below offer examples of research designs that maximize validity. The research design sets up the basic map for how the experiment will be conducted.  It's important to think about the feasibility of using each design in any given applied situation to answer research questions. Some designs will be more helpful for different types of questions and in different settings.

Lesson Links

• Types of Designs
• Choosing a Design
• Experimental Designs
• Quasi-Experimental Designs
• Single-Case Research Designs (46 KB PDF file)

Need Help?  Have a question but can't find the answer? Here are some options:

• Check the list of frequently-asked questions.
• Email Dr. Barry*.

*Remember to include the course name and your name in your message.

To Page 3

Updated on September 22, 2010