Principal Investigator: Matt Silberglitt
CoPrincipal Investigator(s): Mark Loveland, Daniel Brenner, Andrew Grillo-Hill, Matt Silberglitt
Organization: WestEd

Abstract:
The Discovery Research K-12 (DRK-12) program seeks to significantly enhance the learning and teaching of science, technology, engineering, and mathematics (STEM) by preK-12 students and teachers through research and development of innovative resources, models, and tools. Projects in the DRK-12 program build on fundamental research in STEM education and prior research and development efforts that provide theoretical and empirical justification for proposed projects. This project is a four-year design and development study submitted to the assessment strand of the program. It will focus on understanding how educational games, designed according to research-based learning and assessment design principles, can better assess and promote students’ science knowledge, application of science process skills, and motivation and engagement in learning. The project will develop a new genre of games to serve as formative assessment resources designed to collect evidence of science learning during gameplay, provide feedback and coaching in the form of hints, and reinforce middle grade (6th-8th) students’ life science concepts and investigation practices about ecosystems described in the Next Generation Science Standards (NGSS) (Achieve, 2013). The games will build on the designs of the simulation-based, curriculum-embedded assessments developed in previous NSF-funded efforts, which include student progress reports and reflection activities that allow teachers to provide feedback to students and adjust instruction. The design of the games will draw from multiple lines of research, such as cognition, particularly model-based learning; principled assessment design; and motivation. Intended to provide engaging activities for understanding and investigating the system components, roles, interactions, and population dynamics of ecosystems, the project will produce two sets of comprehensive games: (1) Organisms and Interactions, and (2) Emergent Population Levels: Managing an Ecosystem. Each game will consist of progressively advanced mini-games. Twenty-four California Bay Area middle school teachers will participate in the study. Teacher professional development (PD) will include face-to-face sessions and an online platform that permits a wide range of interactions among participants and the facilitators. The PD will emphasize the alignment of the ecosystem simulation-based curriculum modules with their state standards, instructional materials, and the new games.

The project will address six research questions: (1) How well do the games align with the ecosystem crosscutting concepts, core ideas, and inquiry practices in the NGSS?; (2) How well do game components meet quality standards?; (3) How well do the games integrate with the existing simulation-based curriculum modules and the teachers’ existing instructional sequence?; (4) What effect does the use of the games have on students’ understanding of the science concepts, scientific practices, and collaboration skills?; (5) How does success in gameplay relate to improved performance on the external outcome measures comprised of the simulation-based benchmark and the pre/posttest?; and (6) How does the use of the games affect students’ engagement in science learning? In a Year 1 usability study, the project will test, analyze, and revise alpha versions of the games. In Year 2, a classroom feasibility study of beta versions will inform further revisions. In Year 3, six teachers will pilot-test the games. A second pilot test in Year 4 will examine the effectiveness of the games by comparing student performance in classes using the existing simulation-based curriculum-embedded assessments and reflection activities with classes using the curriculum-embedded assessments plus the new games. Data collection and analysis strategies include: (a) alignment reviews; (b) focus groups and usability testing; (c) cognitive labs for construct validity and usability; (d) game reports (badges); (e) pre/posttest of American Association for the Advancement of Science (AAAS) items; (f) benchmark assessment data; (g) student interest in the games and science; (h) teacher surveys; (i) case studies; (j) game quality analysis; (k) differential item functioning; (l) analysis of covariance; and (m) analysis of variance on posttest scores (outcome variable) to compare the means across student groups (by intervention mode) and their prior science achievement levels.