CIRCLS was designed to support researchers funded by NSF EXPs, as well as non-funded researchers doing research in emerging educational technology. To identify relevant NSF projects outside of NSF EXPs, we trained a machine learning classifier on NSF EXPs project abstracts that were labeled with a list of project topics, or “tags,” from the CIRCLS PI surveys. We then applied it to projects awarded by other NSF programs across the EDU directorate (see Figure 1). We identified 172 projects funded through NSF programs other than the NSF EXPs between 2017 and 2024 that focused on emergent learning technologies. CIRCLS included researchers from those projects in communications related to CIRCLS resources and future opportunities.

Figure 1

To understand the extent of the cross-disciplinary research contributions of this community, the CIRCLS team developed a tag-based framework consisting of 49 project tags, hierarchically arranged within 10 major categories of interest. While the tags have been modified over time as new projects and topics have been added to the portfolio, they still provide important insights about the evolution of topic trends in the portfolio, laying the foundation for the analyses presented in the remainder of this section. PI surveys contribute to the majority of data we have on tagged projects. NLP techniques were used to tag projects for which we did not have survey data.

Once all exploratory research projects were tagged, we were able to analyze the data and identify the top 10 tags that emerged (see Figure 2). The most frequently detected tags were ‘‘AI’’ (8.5%), ‘‘Machine Learning” (7.7%), ‘‘Modeling and Simulation,” “Formative Assessment,” and “Augmented/Virtual/Mixed Reality” (all 5.7%). The equity- and social justice-related tags, “Broadening Participation of Historically Marginalized Groups” and “Accessibility and Technology,” were also in the top 10.

Figure 2: Top 10 tags depicting top research interests in 2022

Another area of interest was the interdisciplinary expertise of the researchers engaged in exploratory research, as one goal of NSF EXPs was to bring together interdisciplinary teams at the intersection of technology and learning sciences research. We manually cataloged research expertise for 145 researchers (awarded under RETTL in the year 2022-2023), visiting their websites and CVs. Using these data we created a tree map visualization of the expertise of the community (see Figure 3). The colors of the tiles represent the high-level categories, which are further divided into smaller tiles or codes that represent fine-grained, specialized, and self-identified expertise of the researchers in the community. The size of the tiles represents the relative number of people identifying that topic as their expertise. As you can see, AI approaches account for 20% of the expertise (refer to the orange block in top left corner in Figure 4; dig into the interactive tree map here). Uniquely, due to the nature of the research, this community includes researchers from various domains outside of educational technology research (see the expanded inset of other domains in Figure 5). These include a pool of researchers with expertise in AR/VR/MR, and game-based environments. Sub-expertises like ethical AI and broad social justice-related codes were also represented (the yellow box at the center of the visualization). The tree map visualization helped us capture the interdisciplinarity within the community, which extended beyond the prerequisite computing and learning expertise required for the NSF EXP projects awarded under RETTL. The map helped to demonstrate how multidisciplinary project teams are and helped guide our selection of topics for this report.

Figure 3: NSF EXP expertise for projects awarded under RETTL based on online websites and CVs

Figure 4

Figure 5

Using the above method of detecting relevant keywords in project abstracts, we tagged all NSF EXPs projects awarded between 2017 and 2021, and used them to study the changing trends of research interests (represented by project tags) within the community. We visually represented the number of projects and the top 10 dominant broader categories representing the fine granular tags in a race-chart. The race-chart is an animated version of individual yearly bar charts that depicts the changes in the positions of the bars, showing evolving trends in the research interests among the community.

We identified 10 broad research categories, which include AI approaches, learning environments, learning processes, social justice, and capacity building, among many more. As you can see in Figure 6, in 2017, AI was one of the top topics being addressed by projects, with learning processes and theories taking precedence. But starting in 2018, and inspired by the big push by NSF on AI at the end of 2017, exploratory research became more AI focused. The race-chart also shows that very few projects focused on categories like social justice and equity in 2017, but that over time these topics rose closer to the top (see Figure 7). This trend indicated that the community is interested in creating equity- and social justice-driven technologies for learning, making them important topics to focus on in this report.

Figure 6

Figure 7

To further characterize the research contributions and work of this community, we constructed a social network depicting the project collaborations and conducted an analysis that allowed us to (1) visualize investigator relationships and research collaborations, and (2) map the interdisciplinary contributions. Specifically, we were able to highlight the diversity of expertise within the project teams and the intensity of a research topic across the community.

The network connected the PIs and Co-PIs of NSF projects between 2011 and 2022 to visualize the research community (Figure 8). The project (blue nodes) is connected to the investigators (red nodes) and the topic areas they work on (tan nodes) via the “has expertise” edge (see Figure 8, access the interactive network by Aditi Mallavarapu). This network depicts 238 researchers connected to 87 projects and represents 116 expertise areas that form the “building blocks” for interdisciplinary research in this community.

Figure 8

Drilling down further on the identified patterns of expansive interdisciplinary reach of the community, we used a three-year portfolio of NSF EXPs projects awarded under RETTL from 2021 to 2023 to examine the intensity of research topics. We identified “hotspots,” or high intensity research topics within the community, which included HCI, AI, cognitive psychology/cognitive science, neuroscience and educational technology, as well as “cold spots,” or low intensity areas that we identified as potential “entry points” for integrating knowledge from other domains and experts. Several areas of expertise related to equity and accessibility, including ethical AI, equity in education, working with special populations, and broadening participation for underrepresented communities, were identified as cold spots, with less than two projects through the three years. These cold spot topics were identified as areas of potential growth for the community and informed our decision to focus on accessibility and learning in this report (see Mallavarapu et al., 2023 for more details on the devised metrics).