Portfolio and Analysis
Overview and Methods
Download the Report (PDF)CIRCLS regularly reviews and analyzes the NSF EXPs portfolio, as well as NSF’s emerging learning technology portfolio, more broadly. This is done to increase our understanding of the research topics being addressed by this community, as well as to help NSF understand whether this exploratory research is pushing the envelope forward and achieving the intended level of interdisciplinarity and innovation. We use a wide range of approaches, from more traditional methods, such as analyzing survey data using descriptive statistics, to more innovative methods, including machine learning algorithms. These approaches allow us to uncover emerging and interesting trends across exploratory learning technology projects. This includes trends related to the topics addressed by projects, as well as trends in partnerships, interdisciplinarity, and the demographics of the researchers and other members of the CIRCLS community. Analyses also allow us to examine how trends have evolved over time and which project topics are consistently the focus of exploratory projects in this portfolio.
In order to identify focal topics and PI interviewees for this report, we drew on findings from prior reviews and analyses of the portfolio with a goal of identifying exploratory research topics that met at least one of the following criteria: 1) common/prominent across the portfolio, 2) trending or popular, 3) long-term trajectory/evolved across many years, 4) an NSF priority, 5) emergent/up and coming in the portfolio that we believe will become more prominent in the next 5-10 years).
Multiple data sources and analysis methods have been used to understand the breadth and depth of exploratory learning technology research, including:
- Data Sources:
- PI surveys
- Human coding of expertise from professional websites and online CVs
- Project information pages
- Publication datasets
- Methods:
- Identifying: Use of machine learning algorithms to identify projects funded by NSF programs beyond the NSF EXPs that were thematically similar to NSF EXPs projects and should be included in the CIRCLS community
- Tagging: Natural language processing to “tag” project abstracts
- Mapping: Visualizations to map the expertise of researchers in the community
- Analyzing:
- Quantitative/longitudinal analysis of survey data
- Social network analysis to reveal patterns of (1) diversity of expertise and (2) intensity of inter-expertise collaboration within the community
- Qualitative analysis of project information pages
- Bibliometric analysis of interdisciplinarity and novelty of project publications
These data sources and methods have led to rich visual representations of the CIRCLS community. For example, the tree map shows the different expertise in our community, the social network analysis shows the kinds of collaborations that bring diverse expertise together (Mallavarapu et al., 2023), and the animated project tag race-chart shows changing trends over time in the research undertaken by the community.
