Educational Research
Learning Objectives
At the end of the workshop, participants will be able to:
A. Explain the concept of reliability.
B. Explain the concept of validity.
C. Identify two types of reliability.
D. Identify two types of validity.
Introduction
Overview
The focus of this website is to acquaint you with: a) the basics of scholarship of teaching and scholarship, b) explain some of the basic terms found in educational research, and c) provide examples of both a qualitative study and a quantitative study.
Beyond Bullet Points (BBP)
There has been an increased attention on a particular educational technology used when giving presentations. Beyond Bullet Points (BBP) is an alternate presentation format that has been replacing the traditional PowerPoint (PPT) format. Beyond Bullet Points consists of using an image with a headline and two lines of text on the slides. This replaces the traditional bullet points of PPT. This approach was developed by Cliff Atkinson (Atkinson, 2011). An example of how a BBP slide would look like is provided for you below.
Importance of BBP in my Career
Presentation technologies are an important element in my career as a professor. As a professor, I often will need to incorporate lectures into my classes and may choose to use a presentation technology to aid myself in conveying the materials to my students. The traditional approach used for presentations in a Business School/College, where I teach, has been PowerPoint (PPT). However, as new technology appear, it is important to evaluate the benefits and downsides of these new technologies, especially by examining the existing research available.
Since I have had the fortune to use BBP in classes that I have taken as a student I felt it was an important technology to evaluate for adoption in my own courses.
Beyond Bullet Points (BBP)
There has been an increased attention on a particular educational technology used when giving presentations. Beyond Bullet Points (BBP) is an alternate presentation format that has been replacing the traditional PowerPoint (PPT) format. Beyond Bullet Points consists of using an image with a headline and two lines of text on the slides. This replaces the traditional bullet points of PPT. This approach was developed by Cliff Atkinson (Atkinson, 2011). An example of how a BBP slide would look like is provided for you below.
Importance of BBP in my Career
Presentation technologies are an important element in my career as a professor. As a professor, I often will need to incorporate lectures into my classes and may choose to use a presentation technology to aid myself in conveying the materials to my students. The traditional approach used for presentations in a Business School/College, where I teach, has been PowerPoint (PPT). However, as new technology appear, it is important to evaluate the benefits and downsides of these new technologies, especially by examining the existing research available.
Since I have had the fortune to use BBP in classes that I have taken as a student I felt it was an important technology to evaluate for adoption in my own courses.
Measurement
Measurement is the assignment of values to outcomes (Salkind, 2o17). There are two very important concepts to understand when working with measurement. These concepts are are reliability and validity. Each will be discussed below.
Reliability
Reliability is the consistency of scores over time or it can also be described as the degree to which the scores are free from error (Salkind, 2017). When an individual completes a test, a score is obtained, which is is composed of three parts.
The error score is made up of two components. The first component is method error which is attributed to the situation in which the testing occurs (Salkind, 2017). An example would be that the instruction were not clearly provided for the test. The second component is the trait error which is attributed to the person taking the test (Salkind, 2017). An example would be forgetting your glasses which lead to impaired vision when completing the test. Therefore, the goal would be to minimize the error score, so the reliability will increase (Stone, 1978).
Reliability is the consistency of scores over time or it can also be described as the degree to which the scores are free from error (Salkind, 2017). When an individual completes a test, a score is obtained, which is is composed of three parts.
- Observed score which reflects the obtained score from the measure.
- Observed score is made up of a true score, which reflects the true value of the dimension being measured and this is never actually known (Stone, 1978).
- Observed score is also made up of an error score, which reflects those factors that cause a discrepancy between the observed score and true score (Salkind, 2017).
The error score is made up of two components. The first component is method error which is attributed to the situation in which the testing occurs (Salkind, 2017). An example would be that the instruction were not clearly provided for the test. The second component is the trait error which is attributed to the person taking the test (Salkind, 2017). An example would be forgetting your glasses which lead to impaired vision when completing the test. Therefore, the goal would be to minimize the error score, so the reliability will increase (Stone, 1978).
Types of Reliability
There are various types of reliability, such as parallel-forms reliability and inter-rater reliability. Parallel-forms is when different forms of the test are developed and given to the groups of participants (Stone, 1978). The scores of each group are then correlated with each other. We should see equivalence among the different tests. For example, if I am teaching a large course a give a pre-test and need to test everyone at the end of the semester then I may use an alternate form of the test used. This will eliminate students relying on recalling the questions from the first test.
There are various types of reliability, such as parallel-forms reliability and inter-rater reliability. Parallel-forms is when different forms of the test are developed and given to the groups of participants (Stone, 1978). The scores of each group are then correlated with each other. We should see equivalence among the different tests. For example, if I am teaching a large course a give a pre-test and need to test everyone at the end of the semester then I may use an alternate form of the test used. This will eliminate students relying on recalling the questions from the first test.
A second form of reliability is inter-rater reliability. Inter-rater reliability is when you measure the consistency across raters. This is done by looking at the number of agreements among raters and dividing it by the total number of rating (Salkind, 2017). For example, if a group of students is giving a presentation and is being rated by multiple instructors for their final grade you would want to this type of reliability to ensure the grades are meaningful and accurate.
Validity
Validity is determining the degree to which we are measuring what we purport to measure (Stone, 1978). Reliability is considered a necessary, but not a sufficient condition of validity (Salkind, 2017; Stone, 1978). In other words, a measure may be reliable but not valid. A measure can also be highly reliable and highly valid, which is what we often desire in measurement.
Types of Validity
There are different types of validity, such as content validity and criterion validity. Content validity is when a test represents the domain from which the sample of test items are drawn from (Salkind, 2017). This is established by having a subject matter expert assess the items pulled from the domain.
A second type of validity is criterion validity. This type of validity examines how well a test estimates current performance or future performance (Salkind, 2017). There are two sub-types so if it is the future performance this is predictive validity while the current performance is concurrent validity. Criterion validity measures the relationship between the test and the selected criterion (Salkind, 2017).
Validity is determining the degree to which we are measuring what we purport to measure (Stone, 1978). Reliability is considered a necessary, but not a sufficient condition of validity (Salkind, 2017; Stone, 1978). In other words, a measure may be reliable but not valid. A measure can also be highly reliable and highly valid, which is what we often desire in measurement.
Types of Validity
There are different types of validity, such as content validity and criterion validity. Content validity is when a test represents the domain from which the sample of test items are drawn from (Salkind, 2017). This is established by having a subject matter expert assess the items pulled from the domain.
A second type of validity is criterion validity. This type of validity examines how well a test estimates current performance or future performance (Salkind, 2017). There are two sub-types so if it is the future performance this is predictive validity while the current performance is concurrent validity. Criterion validity measures the relationship between the test and the selected criterion (Salkind, 2017).
Importance of Reliability & Validity in my Career Field:
Reliability and validity are important in my career as a Human Resource (HR) faculty in two different ways in my teaching and in my own research. For this discussion, I will focus on my teaching.
Reliability
The one form of reliability that is essential in my teaching is the use of parallel forms. I often teach multiple sections of a HR course and want to gauge how well the learning outcomes have been met. Therefore, I often will create parallel forms of the pre-test (and post-test) so the students are not compromising the question items between the sections. I find the use of parallel forms is very common in academia. Reliability is important here because you want to make sure the scores correlate between the two forms of the exam you are using to ensure equivalence.
A second example would be inter-rater reliability. In the field of HR it is not uncommon to have two people interview a job applicant. In class, when I cover the topic of selection, I bring up inter-rater reliability and demonstrate how important it is when organizations use such an approach in their interviewing. We discuss how important agreement between the raters are and solutions if this is not the case. This leads to a discussion that organizations may have the need for training to make sure that the potential interviewers understand the attributes they are evaluating and how to score such attributes using a score sheet. This concepts also applies to the discussion of performance appraisal in class as well.
Validity
One form of validity that is essential in my teaching is the use of criterion validity. In teaching HR you often cover the topic of selection or talk about various selection devices (e.g., interviews, work samples, ability tests). Organizations often have job applicants complete different selection devices (e.g., interviews, work samples, IQ test, etc.) during a selection process. These scores are then used to infer who would be successful on the job. This is predictive validity. Student need to understand this type of validity if they will be working in HR, especially if they have staffing responsibilities.
A second example would in class I talk about how selection systems may need to be changed. I describe the scenario of the organization that does not hire a lot of people a year but are considering adding a test (e.g., IQ, typing test) to their selection process. Since their job applicant sample would take a long time to have enough people to use, the organization could give the new test to the current employees in the chosen positions. They could then correlate the test scores with existing performance measures which they would have for their current workforce. This is concurrent validity. Based on the results, the HR professionals could decide if they should add the test to the existing selection system.
Overall, measurement is very important in HR so my HR students truly need to have an understanding of reliability and validity so they can become successful HR professionals.
Reliability and validity are important in my career as a Human Resource (HR) faculty in two different ways in my teaching and in my own research. For this discussion, I will focus on my teaching.
Reliability
The one form of reliability that is essential in my teaching is the use of parallel forms. I often teach multiple sections of a HR course and want to gauge how well the learning outcomes have been met. Therefore, I often will create parallel forms of the pre-test (and post-test) so the students are not compromising the question items between the sections. I find the use of parallel forms is very common in academia. Reliability is important here because you want to make sure the scores correlate between the two forms of the exam you are using to ensure equivalence.
A second example would be inter-rater reliability. In the field of HR it is not uncommon to have two people interview a job applicant. In class, when I cover the topic of selection, I bring up inter-rater reliability and demonstrate how important it is when organizations use such an approach in their interviewing. We discuss how important agreement between the raters are and solutions if this is not the case. This leads to a discussion that organizations may have the need for training to make sure that the potential interviewers understand the attributes they are evaluating and how to score such attributes using a score sheet. This concepts also applies to the discussion of performance appraisal in class as well.
Validity
One form of validity that is essential in my teaching is the use of criterion validity. In teaching HR you often cover the topic of selection or talk about various selection devices (e.g., interviews, work samples, ability tests). Organizations often have job applicants complete different selection devices (e.g., interviews, work samples, IQ test, etc.) during a selection process. These scores are then used to infer who would be successful on the job. This is predictive validity. Student need to understand this type of validity if they will be working in HR, especially if they have staffing responsibilities.
A second example would in class I talk about how selection systems may need to be changed. I describe the scenario of the organization that does not hire a lot of people a year but are considering adding a test (e.g., IQ, typing test) to their selection process. Since their job applicant sample would take a long time to have enough people to use, the organization could give the new test to the current employees in the chosen positions. They could then correlate the test scores with existing performance measures which they would have for their current workforce. This is concurrent validity. Based on the results, the HR professionals could decide if they should add the test to the existing selection system.
Overall, measurement is very important in HR so my HR students truly need to have an understanding of reliability and validity so they can become successful HR professionals.
Qualitative Study - Sample
In the presentation below, you will see an example of a research question and how it can be studied through a qualitative study. In a qualitative research, you are examining more exploratory issues using various techniques such as interviews, surveys, and focus groups. The research question in this example is "What is the influence on the satisfaction levels of business students when Beyond Bullet Points is used?" I hope you enjoy the presentation.
Qualitative Study Design Workshop (Prezi)
Click to set custom HTML
Quantitative Study - Sample
In the presentation below, you will see an example of a research question and how it can be studied through a quantitative study. In a quantitative research, you are examining a more focused research issue using various techniques that allow for more control and power in testing cause and effect. Such approaches include quasi-research designs and true experimental designs. The research questions in this example are: a) What is the influence on the learning outcomes of business students when Beyond Bullet Points is used in comparison the traditional PowerPoint?"and b)"What is the influence on the satisfaction levels of business students when Beyond Bullet Points is used in comparison the traditional PowerPoint?" I hope you enjoy the presentation.
Quantitative Study Design Workshop (Prezi)
Assessment
References
Atkinson, C. (2011). Beyond bullet points: Using Microsoft PowerPoint to create presentations that inform, motivate, and inspire. Redmond, Washington: Microsoft Press.
Salkind, N. J. (2017). Exploring research (9th ed.) Boston: Pearson Education.
Stone, E. F. (1978). Research methods in organizational behavior. New York:Goodyear Publishing Company.
Salkind, N. J. (2017). Exploring research (9th ed.) Boston: Pearson Education.
Stone, E. F. (1978). Research methods in organizational behavior. New York:Goodyear Publishing Company.