Training in Educational Research and Statistical Analysis for Teachers - Session 3: Designing Educational Research

Today’s lesson is about the design of educational research.

There’s a lot of content that has to be memorized conceptually, so the volume itself is overwhelming compared to before.

So I just brought the lesson itself as it is.

1. Diverse Research Questions in Education

Research in education presents empirical evidence through design and analysis for various questions like these, and then draws out useful educational implications.

For example, suppose we have the following questions in the field of education.

• Which is more helpful for learning, preview or review?

• How is the learning motivation of gifted students different from that of typical children?

• Has the Free Learning Semester system affected the decline in students’ grades?

• What factors influence school dropout?

Then how should we design a study?

2. Quantitative Research Methods

Quantitative research has the following definition and characteristics:

• The process of collecting data, quantifying it, and finding answers to research questions

• If it is not quantified, statistical treatment is impossible

• Generally includes statistical analysis procedures, uses many cases rather than a few, and aims to draw conclusions about overall trends

Quantitative research is not appropriate for every study, but for the 3rd and 4th questions above, quantitative research is more suitable.

This is because the structures of these two research questions are similar.

This kind of structure is a typical structure of educational research.

Then, once we have framed the research question like this, what elements should we consider?

3. Validity

Validity refers to how valid the conclusion that the researcher wants to claim actually is.

Every study has a direction intended by the researcher.

For example, in the paper from Session 1, breakfast was related to grades, and in the paper from Session 2, assessment was related to memory.

Validity is about whether one can block approaches in directions other than the one intended by the researcher.

Suppose we have a conclusion like the one below.

When I was in elementary school, I did a lot of preview, and my grades went up a lot.
So we can say that preview has a greater learning effect than review.

The reasons why the above statement is not valid can be thought of as follows:

• If the researcher did that, might other classmates have found review more helpful, or might other classmates have come to different conclusions?

• Since it was in childhood, might it have no effect on current elementary school students?

• Is it not simply because review was not done?

• Regardless of preview, wasn’t the private tutoring started in elementary school what actually helped?

These kinds of possibilities reduce the validity of the conclusion.

When a study has high validity, it must exclude such various possibilities for interpretation.

1) Internal Validity and External Validity

(1) Internal validity

The degree to which, within the scope in which the research was conducted, the conclusion is explained in line with the researcher’s intention and not in some other way.

• Have you ever tried review? Isn’t it that you only did preview and never even attempted review?

• Regardless of preview, wasn’t the private tutoring you started in elementary school what actually helped?

(2) External validity

The degree to which the conclusion of the study can be validly applied beyond the specific participants and conditions of the study.

• Was it just me? Is it the same for other classmates?

• Will preview also help current elementary school students, not just when I was little?

2) Which type of validity should researchers prioritize?

Basically, researchers should devise a research design with high internal validity, and if it also has high external validity, its usefulness is great.

4. Research Design

1) Example of a research design

Suppose we have the following research question.

Research question – Which is more helpful for learning, preview or review?

Now suppose we designed the study as follows.

Design 1 – Have the students in my class do preview for a week, then take a test; then have them do review for the next week and take the same test.

First, let’s say we interpret the results as follows: if the first test score is higher than the later test score, preview is effective; if the first test score is lower than the later test score, review is effective.

In that case, the following interpretations would threaten the internal validity of the conclusion.

• If they take the same test again, wouldn’t the later score naturally be higher because of the test itself?

• After doing preview first, their overall learning attitude might improve so that grades go up regardless of review.

To increase the internal validity of the study, we can revise the design as follows.

Design 2 – Have the students in my class do preview for a week, and have the students in the neighboring class do review for the same period, then give them the same test.

In this design, if my class’s test scores are higher than the neighboring class’s, preview can be interpreted as effective; if my class’s test scores are lower, review can be interpreted as effective.

However, even in this design, there are other possible interpretations that threaten internal validity.

• What if the students in my class originally had better learning attitudes than those in the neighboring class?

• What if the neighboring class’s teacher was progressing through the curriculum faster than my class?

The best research design is one that can block these various kinds of alternative interpretations and leave only the interpretation intended by the researcher.

2) Various experimental designs

If we only compare the experimental group’s scores after the treatment, it is hard to know what the treatment means.

We cannot tell whether the outcome is due to the treatment, the prior characteristics of the learner group, or their natural growth.

Therefore, it is necessary to check scores both before and after the treatment.

Alternatively, we can divide into experimental and control groups after the treatment.

An excellent design considers all these aspects and compares the experimental and control groups both before and after the treatment.

The broader the range of data in terms of time points and groups, the more we can mitigate learning effects and group differences.

5. Factors Threatening Internal Validity

• History – When a coincidental event that occurs influences the experiment

• Maturation – When results change due to the natural growth of the research participants

• Testing – When a kind of learning effect occurs from repeatedly using the same type of test

• Instrumentation – When the test instrument itself or the measurement method changes

• Selection – When comparing two groups, participants with different characteristics from the start are selectively assigned to each group

6. One More Thing? – Operational Definition

There can be a gap between the concept a researcher wants to study and what is actually implemented in the study. This gap is called construct validity.

• What should we regard as “preview”? Should we also consider what is commonly referred to as “prior learning” as preview?

• What should we regard as “learning”? Better memory of the learning content? Providing creative answers? Or just test scores?

7. Summary

• Researchers want to interpret research results in a way that fits their research hypotheses.

• However, for every study, there is always the possibility of explaining the results in ways other than the researcher’s intention.

• Researchers must identify such third-variable explanations and devise a research design that can block room for alternative interpretations.

• Such a design is one with high validity.

8. Afterthoughts

The process of experimental design struck me as somewhat similar to the process of writing paper-and-pencil test items.

Isn’t it possible to interpret it this way?

Or isn’t the conclusion too universal, or could there be another possible conclusion?

In the end, it seems to be a process of connecting the experimental procedure, results, and interpretation based on human deductive reasoning.

If you want to write a paper quickly, it seems you need deep thought and reflection from the experimental stage onward.