The topics of this week's lessons relate to variables and measurements: what they are, what they represent, and how to ensure that they measure what they are intended to measure.

Constructs versus Variables and Scales of Measurement 

The dependent variable, what we've been calling your outcome variable, is what you measure. These lessons will focus on important aspects of the dependent variable and accurate measurement. Another issue that we have for this week is to begin searching for literature. The assignments for this week will help get you started in learning to navigate in the online library. 

A construct is an idea or abstract concept while a variable is something that is measured or observed. For instance, self esteem is a construct. To measure self esteem, you need to define it in terms that are measurable and/or observable, and there by, creating a variable which we can call self esteem.  For your research proposals that you write in subsequent courses, it will serve you better (read as, cause you less stress), if you choose to include variables in your study in which valid and reliable methods for measuring them already exist.

 For example, there are an ample number of valid and reliable measures of intelligence that exist. One of the best ways to know what measures exist for certain variables of interest is by reading empirical research that includes these variables. The articles that you find for your literature review should provide you with possibilities for how to measure your own variables if the research you find is close enough to the topic in which you are interested. As you begin to think about your research questions and identifying potential variables, however, it might behoove you to ask yourself how you are going to measure the constructs you are thinking about.  If you think measuring a construct will be an extremely difficult task then you might consider another construct that can be easily translated into a measurable variable of interest. 

The information you collect using measures of variables is called data. There are many types of data that can be collected. Data can be qualitative such as journal entries or interview transcripts. Data can also be quantitative such as the rate, duration, inter-response time, latency, percentages of whole or partial intervals, etc. Much of the data you will eventually collect will by based on a scale. The following are scales of measurement:

Nominal Data - Nominal data are categories of information that are grouped by name but have no hierarchical order to them (hence 'nominal').  For example, in the variable "gender" there are two categories, "male" and "female." Researchers will often assign the number 1 to all males and the number 2 to all females.  Similarly, for the variable "marital status" there are often 3 categories, "married," "divorced," and "single." A researcher might assign the number 1 to "single," the number 2 to "married," and the number 3 to "divorced." Even though you may have assigned a quantitative representation (number) to these qualitative categories, there is no empirical relationship between the numbers used in the nominal scale that actually corresponds to the actual mathematical relationship between the numbers used.  The numbers are simply used to identify members of a given category. No arithmetical procedures such as addition, subtraction, multiplication or division can be used with these numbers.  the only statistical procedures that may be used are those based on mere counting, such as reporting the number in each category (e.g., there were 24 males and 36 females participating in this study) or expressing the numbers as percentages of the total number of subjects (e.g., 40% of the participants were male and 60% were female).

Ordinal Data - The essential requirement for measurement to be considered ordinal is that it has a distinct order to it (hence 'ordinal'). These are categories of information that have names as nominal data does, but they also have a recognized order. For instance, movie rating G, PG, R, NC-17, X. These are named categories that also have an order to them. We know that an R rated movie will have more violence in it than a G rated movie, for instance. Numbers assigned are used to indicate the order of the observation (who has more or less of an attribute) and nothing more. For example, in a mile fun run in which the runners are not timed but are only ranked according to the order in which they crossed the finish line, we only know who came in first, second, third, etc.  We have no way of knowing how much faster one runner was than another.  The timed difference between the 1st and 2nd runner is not necessarily the same time difference between the 2nd and 3rd runner. The statistics appropriate for an ordinal scale are limited as they are for nominal scales.  Since the interval between the categories (ranks) is unknown, you cannot use procedures that assume equal intervals - addition, subtraction, etc.

Interval Data - Interval scales of measurement order objects or people according to the amount of an attribute they represent but they ALSO establish equal intervals between the units of measure (hence 'interval'). Numbers on an interval scale may be manipulated by addition and subtraction but not by multiplication or division because there is no absolute zero. For example, temperature as measured in Fahrenheit is an interval scale.  The difference between 40 and 50 degrees is the same as the difference between 60 and 70 degrees.  However, you cannot say that 80 degrees is twice as hot as 40 degrees because there is no true zero on this scale. You can have a temperature that is a negative number. Any statistical procedures based on adding may be used with interval level data along with any procedures appropriate for lower level data (nominal and ordinal)

It is important to point out that it has become common practice to treat many educational variables such as classroom tests and grades as if they were interval data, even when the assumption of equal intervals cannot be justified. It would be difficult to argue that the difference in achievement between an F = 0 and D = 1 (on a GPA) represents the same difference in achievement between C = 2 and B = 3.  Because this type of data lies somewhere between ordinal and interval it should be treated as ordinal data.  

Ratio Data - Only a few variables of interest in education and other social sciences are ratio in nature. With a ratio scale it is possible to multiply or divide values by a certain number without changing the properties of the scale.  For example, you can multiply 6 feet by 12 to change the unit of measurement from feet to inches, but it does not change the actual length of what was measured. You can say that someone that weighs 50 pounds weighs twice as much as a person who weighs 25 pounds. All types of statistical procedures are appropriate for ratio scale data.

Lesson Activities

• Visit the Library Tutorial on Basic Library Orientation , when asked what you would like to do, choose "Search for Articles." Complete this tutorial before you move on to the next assignment.

Operational Definitions

Operationally defining your variables is a critical step in planning your research.  Simply put, an operational definition of a variable includes both a clear description of what the variable "looks like" in behavioral terms as well as a clearly stated means of measuring that variable.

For instance, if I want to measure the construct "paying attention" then I need to create specific measurable variables that represent the construct paying attention. One such variable might be "staying on task." There will likely be many variables that could illustrate "paying attention", but we'll just use "staying on task" as an example for now. Once you have your measurable variables, you have to operationally define those variables. The operational definition will include two parts: 1 Clear description: student is staying on task when they are seated at their desk, with face turned toward paper, pencil in hand, and continuously working toward completing the assignment (not drawing or daydreaming or other). 2 Means of measurement: teacher will measure staying on task behavior by recording the number of whole 5 second intervals in which the student exhibits the description above. Then, a percentage of 5 second intervals can be calculated by dividing the number of successful staying on task intervals by the total number of intervals observed.  Observations should be done for at least 60 consecutive minutes a day.

Many students forget to include both parts of an operational definition. You will not be able to collect data if you don't know exactly what you are looking for AND exactly how to measure it. This is a very important step!

 I cannot tell you how many people say they want to study the construct "motivation" and will use that term throughout a proposal and in their research questions without ever giving a thought to the idea of the wide array of ways in which motivation is defined in the literature.  Motivation may be identified as self-efficacy in one study or internal/external in another.  Some see locus of control as motivation while others look toward self-concept.  Do not fall into this trap.  It is important to say what you mean and mean what you say.  Hopefully, the brief reading, exercise and case studies here will help you think about how you will need to operationalize the variables you choose for your own research.

Lesson Activities

• Following the example in Table 3.1 of your text book, think of 3 constructs you might be interested in measuring and then create operational definitions of variables that could be used to measure that idea.  Post your constructs and operational definitions of corresponding variables in the "Week 3 Lesson 1 " dropbox in the eLearning system.

Please note: For this course, please upload ALL files to Elearning as Rich Text Format (RTF) rather than word docs.

When you save an assignment in Word, you have the option of choosing "Save As" from the file menu. When you do that, you can select the "File Type" in the window that pops up. Please use the drop down menu under "File Type" to choose Rich Text Format (RTF) before you upload any files to Elearning.

Reliability

Reliability is the concept of knowing how consistently a measure can assess a variable. There are many different types of reliability like test-retest reliability which tells you the agreement between two administrations of the same test over time if everything else is held constant and inter-rater reliability which tells you the percent agreement between independent observers collecting data using operational definitions, and more. You will have to deal with the idea of reliability in your measures regardless of the design you choose. This section is dedicated to reliability in research that employs behavioral observation to collect data, specifically inter-rater reliability.

If I am the only observer collecting data, how do we know the data I record is not subjective, based on my opinion? In other words, how do we know the data is scientific, systematic, and objective? One place to start is by using operational definitions to define what data I should be recording. This is a very good start, but even with that operational definition, there is sometimes room for interpretation. Was that time spent rubbing his eyes considered off task, for instance? One observer might score that as on task while another might say that it was off task. To make sure that data are reliable, we need a second observer to independently confirm our findings.

Inter-observer reliability is the percentage of agreement between two independent observers, who use the same operational definitions during the same time frame to collect data.  The requirement to establish reliability is to agree more than 80% of the time on at least 33% of the trials.

As an example, if I wanted to calculate inter-observer agreement for the variable "staying on task" that we defined above, I would collect data using the operational definition every 5 seconds for at least an hour. During at least 20 minutes of that very same hour, a second independent observer (that means they can't see how I'm scoring the behavior), independently records their interpretation of the behavior also using the same operational definition every 5 seconds. Then, I would take the 20 minutes of data that I collected during the same time that the second observer collected their data and I would compare the findings. I would total the number of intervals in which we agreed. Let's pretend we agreed 238 times. Then I would count the total possible number of intervals in the 20 minute period (240 five second intervals in this example). Finally, I would divide the number of agreements (228) by the total possible number of intervals (240) and then multiply by 100 to yield a percentage of inter-observer agreement = 95%

Established measures of constructs such as a depression inventory or an IQ exam will have established rates of other types of reliability like test-retest reliability, and for those of you who use these established measures that do not rely on direct observation, this exercise in calculating inter-observer reliability will not be necessary. Many of you will use direct observations as a means of measuring variables. Those of you who do, will need to use inter-rater reliability to establish the reliability of your direct observation measures.

Lesson Activities

• Go to the John C. Pace library online and find an empirically based journal article (one that used data to answer a research question) that uses a single case design. Journals such as the Journal of Applied Behavior Analysis and Journal of Positive Behavioral Interventions are good sources.
• Skip to the Method or Results section and find where they discuss reliability. Read about how that was calculated and what percent of reliability the authors report.  How reliable are their measures and how does this affect your interpretation of this study? Do you have more or less confidence that their results are accurate based on the reported reliability? Think about the importance of reliable measurement is making good decisions. How confident would you be in your decision making if the measurement tools used to collect data to help you make that decision were not reliable?
• You don't need to post your thoughts, but make sure you think about this concept. It's an important one to  understand before we move on.

Need Help?  Have a question but can't find the answer? Here are some options:

• Check the list of frequently-asked questions.
• Email Dr. Barry*.

*Remember to include the course name and your name in your message.

To Page 3

Updated on January 3, 2010