America’s Test Kitchen (https://www.americastestkitchen.com) provides a scientific analysis of cooking various foods (meats, dishes, fish, breads, etc.) as well as cooking products in a series of seasons and episodes. In a recent review of baking stones and steels for making pizza that is just like the professionals, Lisa McManus describes how they tested several products. In this short video, she describes the experiment, controlling for time, temperature, and various dimensions of the stone or steel for making a great pizza dough. She explains the principles behind the baking process and how the variables come together into conclusions and recommendations. Most importantly, she uses specific words to describe their study and make a recommendation.
In educational science, the same idea of research is used, whether a literature review, descriptive study, or controlled experiment. And it’s the Method section that provides the details for interpreting the results. In this section, the process for collecting data need to be explicitly described so replication is possible. The setting is an important context for collecting information with small variation possible that can lead to different conclusions. The sample of participants is equally important, whether children, teachers, young adults, parents, or community members. Usually, this aspect of a study leads to limitations in generalizing the results from samples to populations, an important issue in educational research.
Data can be collected in a number of ways, all of which may influence the outcomes: surveys (digital or paper-pencil), individually or collectively (think of the difference between focus groups versus an on-line surveys), proctored or done with no observation, etc. The measures themselves need to withstand the criticism that the outcomes are not reliable (e.g., they might change with different items, forms, samples, occasions, or judges). Data also need to be compiled with quality assurance so that they can be analyzed appropriately with various checks conducted to ensure common sense interpretations.
Finally, the entire enterprise of various research efforts need to be assembled and fit into a holistic manner that reveals consistencies and inconsistencies. Such patterns can then be tracked back to the Method section and conjectures made that lead to more research, the usual conclusion (and not made just to keep researchers employed). Though the variables and their variation may seem endless, it is helpful to consider the field of medicine, with all the amazing advancements made in the past 100 years. These strides have come about with systematic research, both controlled descriptive studies as well as experiments with controls implemented in trials. And all done with humans who vary as widely in incredible ways.
The moral of this blog: Use specific words in either describing educational studies or conducting one (or for that matter, in anything you write or speak). Think about all the variations possible in words like settings, participants, data, measures, collection, analysis, and interpretation.