Session B16: Mini-Symposium: Physics Education Research Incorporating Big Data and Data Mining

Opportunities and challenges of large language models in physics education

Recent advances in large language models (LLMs) such as ChatGPT demonstrate compelling performances in complex tasks such as writing coherent essays, reasoning, and even performing well in untrained tasks. While these models also hold a great promise for physics education, there are concerns about biases and output reliability as well as impacts on learning willingness, exam fraud, and misuse. Thus, it is crucial for their integration into physics learning and teaching to provide scientific evidence regarding their effectiveness and safety. In this contribution, we demonstrate how LLMs may support physics teachers and learners and conclude with a perspective of how AI may transform physics education.  

Specifically, we report on a comparison of pre-service teachers creating physics tasks using either ChatGPT or a textbook. The correctness of tasks was similar across both groups and both groups adjusted the difficulty level well for their target audience. Moreover, both groups struggled with task specificity and often omitted key solution information. However, those using textbooks created clearer tasks and embedded them more effectively in contexts. While ChatGPT users praised its user-friendliness, they faced issues with output quality.

Additionally, we show how LLMs are sensitive to students’ misconceptions and can aid in developing and validating Concept Inventories. Concept inventories are crucial for checking learners' understanding of physics concepts, but the creation and validation of such inventories are often resource intensive. In this context, we show how ChatGPT enables the creation synthetically generated empirical data, significantly accelerating the creation and validation of concept tests.

Overall, LLMs offer novel perspectives on the intersection of AI and physics education. However, their integration into teaching and learning should be carefully managed, considering their limitations and ensuring they complement traditional teaching methods.   

Integrating Generative AI as a Tool for Formative Feedback in Large Enrollment Physics Courses

We investigated ways to integrate AI into our introductory physics classical mechanics course. We utilized ChatGPT to offer constructive written feedback to students during an optional pre-exam review activity. Given students' struggles with developing systematic problem-solving approaches and instructors' time limitations in large classes, we created prompts to connect course concepts and gather ChatGPT feedback. Data collected encompassed student responses, ChatGPT feedback, criteria provided for feedback and instructors' assessments. Through thematic analysis of feedback received from ChatGPT and instructors' assessments, our preliminary results suggest that ChatGPT can be used to provide timely constructive written feedback if provided with detailed criteria. In this talk, we present both the encouraging similarities between human instructor feedback and ChatGPT's feedback as well as some of the limitations that emerged in our study. In the future, it will be important to repeat this study with a larger sample size. This will help us confirm ways AI could potentially be incorporated in physics courses or suggest additional unforeseen subtleties about the potential effectiveness of AI as a feedback tool in physics courses.   

Analyzing unsupervised approaches to coding motivation of women in physics.

Machine learning, along with its applications to written language, is rapidly developing. PER researchers are curious to what extent these computational methods can be shown to have validity within qualitative research. We present our investigation of the use of an unsupervised machine learning algorithm called Latent Dirichlet Allocation in creating a coding scheme to characterize women's motivation for joining physics. We analyze over 2,000 survey responses from Conference for Undergraduate Women in Physics participants that were previously hand coded by Franklin et al. using established self efficacy and expectancy value theories. We test what motivational codes emerge from an unsupervised natural language processing approach, and compare the results from Latent Dirichlet Allocation topic modeling to the established code scheme. Comparison of the theoretical coding and the topic modeling show points of similarity but fail to fully reproduce the human motivational codes.   

Methods for trustworthy application of Large Language Models in PER

Within physics education research (PER), a growing body of literature investigates using natural language processing machine learning algorithms to apply coding schemes to student writing. The aspiration is that this form of measurement may be more efficient and consistent than similar measurements made with human analysis, allowing larger and broader data sets to be analyzed. In our work, we are harnessing recent innovations in Large Language Models (LLMs) such as BERT and LLaMA to learn complex coding scheme rules. Furthermore, we leverage methods from uncertainty quantification to help understand the trustworthiness of these measurements. In this talk, I will demonstrate a successful application of LLMs to measure experimental skills in lab notes and apply our methodology to evaluate the statistical and systematic uncertainty in this form of algorithm measurement.   

Predicting persistence of women in physics with machine learning

In this work, we use a variety of machine learning tools to predict retention of women in physics. Previously, we used data collected at the Conference for Undergraduate Women in Physics, along with a follow-up survey, to study which factors correlated with long term persistence in physics. The factors we studied were sense of belonging, sense of community, interest, physics identity, perceived recognition, and performance competence. In this study, we build on our previous results by comparing the machine learning methods of support vector machines, neural networks, random forests, and logistic regression to best predict which women are most at risk of leaving the discipline. 

The end goal of this study is a tool that can effectively predict whether an undergraduate woman may need more support to remain in physics. Using this, professors could provide targeted interventions to increase overall retention, supporting gender equity in physics. Though we focus on gender equity in this work, the principles of our machine learning approach can be used for other predictive measures in physics education. This work aims to highlight the uses of each machine learning technique for other potential work.   

Utility of pre-instruction diagnostic tests for estimating probabilities of final course grades in introductory physics

We have examined the relationship between various pre-instruction assessment measures and final course grades for students enrolled in introductory general physics courses at five campuses of four universities; the total sample included 25 separate classes and over 2000 students. The three assessments were the Force Concept Inventory, the Lawson Test of Scientific Reasoning, and a mathematics diagnostic test that we have developed and tested over the past seven years. We find, with nearly 90 percent consistency, that top-quartile scorers on the pre-instruction assessments have double or greater probability of receiving high (top quartile) course grades, and half or less probability of receiving low (bottom quartile) course grades, compared to students who scored in the bottom quartile on the assessments. Predictor variables have some inter-correlation but models incorporating two or more predictors generally have somewhat more predictive power than single-variable models, although the most successful sets of predictors vary from course to course.