Session F08: Mini-Symposium: Emerging Research Methods in Physics Education Research

Emerging Methods in PER: Observations from Working on the Statistical Modeling Review Committee

Physics Education Researchers have been embracing increasingly more sophisticated statistical methods to gain greater insights into the factors that most impact student learning and growth. These techniques can provide important opportunities for guiding advancement in our field. However, they also provide challenges as it is easy to misinterpret results from these analyses. After a brief overview of the opportunities, and challenges, I will share some of the most prevalent issues that we see in papers sent to the statistical modeling review committee. These examples will provide context and motivation for a discussion of a draft set of guidelines being developed to help researchers and referees maintain consistent publication standards for quantitative research. Overall, these review guidelines aim to ensure that authors provide justified interpretations of their studies and share enough information about their studies to encourage replication, extension, and meta-analysis.   

Network clustering of students based on answers to a questionnaire on refraction

Students’ answers to conceptual questionnaires are mostly analysed at the cohort level. However, most studies have not considered clusters of students within the cohort. This study uses newly published data [0] to analyse students' (N=1368, across five continents) answering patterns to a refraction questionnaire. Other student data include age, courseID, and type of programme. This study uses a network approach to find clusters: (A) we calculate the similarity [1] of each pair of student responses. This results in a weighted adjancency matrix, forming a 'similarity network'. (B) A backbone network is extracted [see: 2] to remove spurious links. (C) Clusters are found using community detection [see: 2]. The presentation will show results of analyses on how students clusters can be characterised in terms common answers and other student data. This addresses research aims:

1. How can student similarity clusters be characterised in relation to the context in which they study?

2. How can student similarity clusters be characterised in terms of particular answering patterns?

The results of this study could further inform instructional strategies developed in PER.

[0] Linder, C., et al. (2024). Relationship between semiotic representations and student performance in the context of refraction. PR-PER.<br type="_moz" />
[1] Lin, D. (1998, July). An information-theoretic definition of similarity. In Icml (Vol. 98, No. 1998, pp. 296-304).

[2] Brewe, E., et al (2016). https://doi.org/10.1103/PhysRevPhysEducRes.12.020131   

Mapping expert knowledge of quantum sensing and measurement

Quantum systems and their unique behaviors will lead to significant improvements in computing, communication, and sensing. Of these three application areas, quantum sensing has received little attention within curriculum development. Quantum computing curricula are accessible to many levels (including high school) because the computers are described at a higher and more abstract level that avoids details about widely varying hardware implementations. However, quantum sensors' primary function is to interact with their environment, and the hardware designs are diverse and integral to their operation, which may necessitate additional concepts and mathematical formalism (e.g., continuous time evolution). Unlike many areas of undergraduate physics education, quantum sensing is at the forefront of modern scientific research. Introductory textbooks, which are often used to identify and organize key ideas, do not exist. As a first step in improving quantum sensing education, we are conducting semi-structured interviews with leading experts to understand how they describe the major ideas of quantum sensing and measurement. The interviews employ a concept-mapping activity where experts graphically connect those ideas to each other and to topics in other courses (e.g., undergraduate quantum mechanics or introductory quantum computing). Experts also suggest models of minimum complexity that could be used for teaching and developing exercises around clusters of concepts. We will highlight findings from a small number of interviews to describe our concept mapping method and some preliminary results.   

Redesigning the Commonly Used Legacy Instruments: Using Evidence Centered Design to Develop Valid, Equitable, and Flexible Items Assessing Conceptual Understanding of Kinematics and Dynamics for Introductory Physics

Commonly used legacy instruments, such as the Force Concept Inventory (FCI) and the Force and Motion Conceptual Evaluation (FMCE), have shown some serious flaws including substantial psychometric problems which threatening their reliability and validity, and demographic biases which make them inaccurate for some underrepresented student populations. Even with these flaws, these instruments have been vital to the development of Physics Education Research (PER) as a discipline; however, while the teaching of physics has evolved, the formative assessments researchers, instructors and, more broadly, physics departments use to evaluate their classrooms have not. It is critical that these instruments be redesigned in order to continue to advance the teaching and learning of physics for all students. In this talk, we will discuss the project goals and development strategy of our new NSF award (#2235518) intended to design a set of valid, fair, and flexible tool assessing conceptual understanding of kinematics and dynamics for introductory physics courses. Constructed within Evidence Centered Design (ECD), the new assessment items will be grounded in learning and measurement theory, based on a construct-centered and community-based approach. The flexible instrument, the Kinematics and Dynamics Assessment (KDA) will include multiple scales measuring foundational topics within kinematics and dynamics organized into subscales which instructors can utilize to build their own classroom assessment. The library of items will be extensively validated to ensure superior psychometric properties and instrumental fairness for women, underrepresented minority students, and first-generation college students.   

Exploring Physics departments' ecosystems

We studied the ‘physics experience’ of students and faculty in two physics departments, one in Europe and one in North America, to understand the socially constructed criteria and norms that shape how students and faculty perceive the culture of university level physics education, with the goal of empowering physics departments to understand and respond to social and structural issues that lead to gender inequity in Physics. To minimize researcher bias, an a priori framework was formulated to construct the interviews and the analysis. Since individuals experience their environment in different ways, semi structured interviews were used with techniques that elicit various responses and provide triangulation, to uncover the experiences of participants by stimulating them to discuss issues for which they did not have a readily formulated response. The data will be analyzed following principles of grounded theory and a reliability index will be established. This presentation will discuss the need for and the process of formulating an a priori framework, the methods used for the collection and analysis of qualitative data and for establishing the reliability index, and some initial findings in examining the interplay of the various axes of identity of the participants and the collective culture of the physics department.   

Elevating Black Women's Experiences in Physics Graduate Programs through Photovoice

Black women are severely underrepresented among physics PhDs awarded. Several programs have been implemented to support students from underrepresented racial and ethnic groups to earn physics PhDs as well as to support women in physics. Black women live at the intersection of marginalization in physics through both race and gender. Thus, programs designed to support individuals along one of these dimensions without an intentional intersectional focus may leave Black women's needs unknown and unmet. In this project, supported by the American Physical Society Innovation Fund, we leverage photovoice methodology to partner with Black women enrolled in physics PhD programs to describe their journeys towards a physics PhD. In photovoice, participants as co-researchers are provided a prompt and time to take photos and generate photo captions that respond to the prompt. We conducted individual interviews with participants using Wang and Burris's SHOWED approach, where specific prompts are provided to aid participants in discussing the root causes of the experiences captured in their photos as well as identifying lessons to share with others and potential allies to work towards change. This talk will focus on why we selected and how we adapted photovoice for our research goals.   

Conceptual engineering in physics education

A hallmark of engagement in scientific inquiry is students’ pursuit of coherent, mechanistic models of natural phenomena (Hammer & Van Zee, 2006; Russ *). An articulation of the construct of mechanistic by Russ et al. (2008, p. 512) emphasizes the importance of identifying entities, their properties, and their organization such that they can be said to causally produce the phenomenon being modeled. However, the entities and phenomena that populate scientific models are rarely physical objects that are objectively noticed, categorized and employed in causal stories (e.g., the first billiard ball strikes the second billiard ball), but theoretical constructs (e.g., work transferred kinetic energy). Such theoretical constructs are all but inescapable in scientific models: “entropy causes solubility to rise,” “boojum nucleation … stabilizes the metastable configuration,” and “penguin diagrams generate a local strangeness-changing interaction.” Entities in scientific models, then, are not so much physical objects that are readily identified as they are theoretical objects that are carefully engineered to meet the demands of emerging theories, and their development, implementation and acceptance by the scientific community represents a significant accomplishment in the development of scientific models. 

Over the past few years, this activity has received focused attention under the term “conceptual engineering” within philosophy. The goal of this talk is to describe the concept of conceptual engineering and its relevance to physics education. The second goal is to show that, far from being rare, conceptual engineering can be common (almost inescapable) in student thinking and discourse, and that students - at least in the context of an undergraduate inquiry course - are not only capable of such engineering, but that it arises almost spontaneously and meaningfully.