Capturing Team Cognition: A Multimodal Dataset for Adaptive Collaborative Interfaces (2023 – 2026)

Although working on teams has the potential to fuel creative synergy during open-ended problem solving, teams are also vulnerable to process loss, or breakdowns in coordination or communication that arise naturally from multiple people trying to work together. To combat this phenomenon, it is necessary to facilitate the social and cognitive processes required for successful collaboration (coordination, planning, conflict management, etc.) as well as respond and adapt intelligently when teams experience these breakdowns.

 

While several digital collaboration tools such as Miro, Slack, or Trello exist to support teams by improving communication and task management, most do not offer any form of analytics or adaptability, and provide little visibility into the underlying cognitive and affective states that shape team performance. Assessing the quality of collaboration has traditionally relied on subjective self-report measures, or time-consuming annotation of audio or video data, which might not scale to the needs of large organizations.

 

In contrast, measuring brain activity via non-invasive recording techniques such as electroencephalography (EEG) or functional near-infrared spectroscopy (fNIRS) can be done continuously in real time while teams are working, and provide insights into the group and individual processes shaping the quality of collaboration.

 

However, a current major limitation to the development of such a system is the dearth of ecologically valid datasets: most data is collected from pairs of co-located users in controlled lab environments, despite the fact that real-world teams often collaborate on open-ended problem-solving tasks in larger, geographically distributed groups, shifting their attention between multiple different tools and sub-tasks. Very few open-access datasets exist from teams of three or more working together to find creative solutions to a problem or achieve a common goal, and none involve teams where members were distributed across multiple locations.

 

To fill this need, my colleagues from the University of Bremen and I introduce a rich multimodal dataset and experiment setup supporting the development of adaptive collaborative systems. We examined naturalistic collaboration in multi-human-agent teams, recording synchronized EEG, audio transcripts, screen activity, and behavioral annotations from participants while they worked together remotely across two continents to design a virtual escape room, with help available from ChatGPT.

 

We derived several measures from our neural and behavioral data—such as task engagement, neural synchrony, and interaction patterns—to model team processes, and demonstrated that we could detect factors relevant to collaboration with brain sensing, as well as replicate some findings from prior work.

 

By providing a shareable dataset, robust sensing infrastructure, and techniques for modeling distributed collaboration, this work enables future interactive systems that sense and support distributed teamwork in real time.

 

Paper (CHI ’26): https://doi.org/10.1145/3772318.3791607

Dataset: https://osf.io/p2u85/ (More coming soon!)

​

Team Brain-Computer Interfaces: Charting Opportunities, Concerns, and Guidelines for Future Collaborative Work (2023 – 2026)

Brain-driven interfaces have the potential to adapt in real time to the needs of teams by sensing mental states and processes relevant to collaboration. However, non-invasive brain-computer interfaces (BCIs) are still an emerging interaction modality, and support systems for teams leveraging brain activity from multiple users do not yet exist. Proceeding directly with design and implementation risks creating a system that users do not find useful, surfaces sensitive neural data in ways users are uncomfortable with, or enables harmful downstream applications such as discrimination based on brain activity or tying cognitive metrics to employee performance reviews.

 

Therefore, to ground the development of team-facing BCIs in stakeholder needs rather than technical possibility alone, I conducted a two-phase study with team members and leaders. First, I employed user-centered techniques (generative card sorting and semi-structured interviews) to investigate stakeholder needs, challenges, and aspirational capabilities during collaborative work, as well as compensatory strategies or tools used (such as generative AI, organization software, etc.) to probe gaps in current offerings without priming participants toward any particular tech solution. Second, I presented speculative storyboard scenarios depicting potential BCI implementations inspired by participants’ own ideas, probing their reactions, perceived utility, and concerns. This allowed me to examine the same population’s enthusiasm for brain-sensing tools alongside their apprehension, yielding a grounded picture of both the opportunity space and the risks.

 

Participants identified several unmet needs that brain sensing could plausibly address: detecting when teammates are confused or disengaged, balancing workload across a team, and surfacing the quality of interpersonal dynamics over time. When presented with BCI storyboard scenarios, participants recognized their potential to meaningfully support collaboration, but were equally vocal about risks, including concerns over data accuracy, the possibility of gaming metrics, privacy violations, and how cognitive data might be misused by employers or misattributed across contexts. Drawing on these findings, I propose a set of six design principles for responsible, user-centered BCI development for teamwork, emphasizing configurable consent, purpose-bound data collection, transparency, support-oriented (rather than evaluative) feedback, optional participation, and collaborative decision-making about how data is used.

 

Paper: In preparation

​​

Integrating Non-Invasive Neuroimaging and Educational Data Mining to Improve Understanding of Robust Learning Processes (2023 – 2024)

In ideal circumstances, teachers would have enough one-on-one time with each student to tailor instruction and assessment to their individual learning needs, though in practice this is often not possible. Intelligent tutoring systems (such as the ASSISTments platform) are increasingly capable of supplementing teacher interactions by harnessing tremendous amounts of behavioral data to provide more personalized learning experiences. However, because the data captured is event-based, these systems can fail to capture critical information about student learning during pauses in this log data: sometimes, pauses are indicative of task-related cognitive states supportive of robust learning, such as reflecting on errors and confronting one’s misconceptions, while at other times they can indicate students are engaging in off-task behavior, daydreaming, or simply focusing on the wrong problem features.

 

Different kinds of pauses warrant different system responses; however, existing work examining the relationship between pauses and subsequent learning outcomes has been inconclusive. The continuous nature of physiological measurements allows us to compensate for these pauses with insights from brain data, with the goal of differentiating different types of pauses to make better predictions about learning outcomes and the student’s cognitive state. In addition, the detailed tutoring log data provides contextual information about what events occur before and after the pauses that will enable better interpretations of brain activity.

 

This work is an ongoing collaboration between colleagues at the University of Pittsburgh, Lehigh University, and the WPI HCI Lab, and is part of the NSF’s Integrative Strategies for Understanding Neural and Cognitive Systems (NCS) program. While at WPI, I coordinated data collection and analysis for the final portion of this study, in which fNIRS data was collected from a cohort of N = 15 students from local community colleges while they completed realistic learning tasks using ASSISTments, as well as a series of validation tasks to quantify relevant cognitive processes, such as rule learning, rule following, goal maintenance, and goal updating.

 

By measuring these states as well as capturing self-reported periods of confusion during the learning tasks, we developed a proof-of-concept adaptive system able intervene and provide hints when students’ brain activity indicated that they were stuck. Future work will refine this system, further investigate students’ cognitive states during learning, and build a foundation for combining lightweight neuroimaging, machine learning, and personalized learning environments for the development of new adaptive learning systems.

 

Interpersonal Neural Synchrony BCI for Adaptive Multi-Agent Human-Robot Collaboration (2021 – 2023)

When humans work closely together, they can pick up subtle cues from their team members and adapt their behavior appropriately. However, in contexts with robot team members, robots may not be able to detect these cues and respond accordingly, and the presence of a robot may impact the performance and cognitive states of human collaborators. Interpersonal neural synchrony (INS), where the brain activity of multiple people over a period of time is similar, has been observed in neurophysiological recordings of pairs of participants engaging in cooperative and social activities. This project investigated how measurements of INS acquired using functional near-infrared spectroscopy (fNIRS) might be used to assess and augment teamwork in complex multi-agent human-robot team tasks.

​

Examining the Impact of Digital Jury Moderation on the Polarization of US Political Communities on Social Media (2020 – 2022)

As polarization among political officials has increased dramatically in recent years, the social media landscape has followed suit. The increased prevalence of disinformation, inflammatory rhetoric, and harassment online has augmented polarization in turn, propelling a feedback loop resulting in the erosion of democratic norms. Effective moderation of social media platforms can help solve this problem.

 

My MS thesis work explored how implementing a democratic, peer-based "digital jury" moderation system for social media platforms would impact polarization online, compared to traditional, "top-down" moderation that is conducted by employees of the platforms themselves. While the peer-based system did not significantly impact polarization, our moderators on average viewed the system as just, legitimate, and effective at reducing harmful content. Additionally, end users noticed no difference between the two systems, indicating that implementing such a peer-based moderation system has the benefit of providing users agency in platform governance without adversely impacting user experience.

​

Paper (IWC): https://doi.org/10.1093/iwc/iwae036

​

BrainEx: Interactive Visual Exploration and Discovery of Sequence Similarity in Brain Signals (2020 – 2022)

As brain sensing technologies have become cheaper and more widespread, researchers and developers have designed systems able to leverage the unique insights into users’ mental states these devices offer to create more intelligent and adaptive user experiences, with applications in communication, marketing, human factors, entertainment, education, and more. However, developing these systems has traditionally required integrating expertise in several different domains: making a working BCI requires acquiring useful brain signals (containing the mental state or process of interest) via controlled experiments or public datasets, preprocessing the data to remove artifacts and isolate these components of interest, and analyzing it to extract a control signal (e.g., the detection of a cognitive state such as workload or engagement) via statistical or machine learning approaches, which is used to drive an adaptive intervention.

 

While tools exist to assist with these tasks, most are geared toward experts and have rigid constraints on the types of analysis available. These tools typically assume that users have an a priori understanding of the dataset being used and which features are important, and are tailored to a specific data modality (e.g., EEG or fNIRS). There are few tools available which prioritize interactive exploration to gain a broad sense of a dataset, or that enable browsing the relationships and structures within them—both unmet needs of non-experts working with real-world data or publicly available datasets.

 

To support interactive exploration and discovery within brain signal datasets, my colleagues in WPI’s HCI Lab and I developed BrainEx, a web-based analytics and visual exploration platform for brain data. BrainEx takes advantage of distributed computing algorithms that enable fast exploration of complex, large collections of time series data, allowing for performance orders of magnitude faster and more accurate than other leading approaches while being easy to use and learn. This system allows non-expert researchers to perform similarity search, explore feature data and natural clustering, and select sequences of interest for future searches and exploration, via an intuitive visual interface.

 

A user study conducted with experts in data visualization, neuroscience, and human-computer interaction demonstrates BrainEx’s effectiveness at meeting these functional requirements for real-world use cases, allowing users to find occurrences of signal patterns of interest, uncover latent relationships, extrapolate labels for sequences of unlabeled data, and more.

 

Paper (EICS ’22): https://doi.org/10.1145/3534516

​

Understanding HCI Practices and Challenges of Experiment Reporting with Brain Signals: Towards Reproducibility and Reuse (2019 – 2022)

The ability to interface directly with the human brain has been an alluring possibility in popular culture for several decades, and advancements in cognitive neuroscience and brain sensing technologies have made this frontier more accessible than ever before. With the prospect of more widespread adoption of this emerging technology, HCI researchers have taken an increasing interest in integrating brain sensing into interactive systems, as well as leveraging the insight it offers into user psychological states for user experience evaluation.

 

However, the specialized knowledge and integration of research from several different domains required to develop such systems considerably hinder innovation. While researchers would typically attempt to build on each other’s prior work to overcome these challenges, differing norms and expectations regarding which methodological information to report and the types of contributions that are prioritized across disciplines and publication venues limits the reproducibility and reuse of HCI research using brain signals.

 

In this work, collaborators at the University of Bremen and I analyzed 110 papers using brain signals published in HCI venues to better understand current reporting practices. We reviewed their domains, applications, modalities, mental states and processes, and more to map the heterogeneity of approaches, revealing a variety of different reporting considerations depending on factors such as the type of contribution and research direction. To capture these variations and facilitate comparisons, we constructed a formal experiment model categorizing the different attributes occurring in these publications (experiment flow, task description, recording setup, data processing approach, etc.), and refined it using exploratory factor analysis to uncover redundant elements, as well as the confirm differences in reporting practices we observed in a data-driven way. Finally, we derived several recommendations for the community from our exploration (leveraging and learning from the diversity of researchers doing this work, awareness of the breadth of acceptable contributions, sharing data and code, adoption of the experiment model) to help ensure future HCI research using brain signals is more reproducible and reusable.

 

Paper (TOCHI): https://doi.org/10.1145/3490554

​

Postbaccalaureate Research (2017 – 2019)

After completing my undergraduate studies, I joined Dr. Hal Blumenfeld's lab at the Yale University School of Medicine as a postgraduate research associate. I contributed to several projects that aimed to understand the neuromechanisms of normal and disordered consciousness, but my main project (under the mentorship of postdoctoral fellow Dr. Kate Christison-Lagay) explored the mechanisms of auditory perception. I helped record and analyze intracranial EEG signals from patients with medically intractable epilepsy who performed an auditory threshold perception task.

 

Observations indicated that a switch-and-wave phenomenon similar to what had previously been observed during visual perception was also present during auditory perception, suggesting the presence of a common perceptual network spanning multiple sensory modalities.

​

Paper (Neuroimage): https://doi.org/10.1016/j.neuroimage.2025.121041

​

Undergraduate Research (2014 – 2017)

BCI-based Brain-to-Brain Communication (Summer 2016)

In the summer before my senior year, I attained a competitive Vredenburg Scholarship to conduct research abroad at Tokyo Institute of Technology under Dr. Tohru Yagi. There, I pursued a project (with then-PhD student Theerawit Wilaiprasitporn) that explored how an EEG-based brain-computer interface (BCI) could be used to facilitate bi-directional communication using brain signals. Although just a proof-of-concept, results indicated that by synchronizing participants' signals using steady-state visually evoked potentials (SSVEP), we could increase potential information content being communicated directly between individuals' brains.

​

Paper (BMEiCON ’16): https://doi.org/10.1109/BMEiCON.2016.7859615

​

Effects of Exercise on Mouse Cerebral Vascular Structure (2014 – 2017)

My first research experience was in Dr. David Linden’s neuroscience lab at the Johns Hopkins University School of Medicine, supervised by postdoctoral fellow Dr. Robert Cudmore. We sought to determine if exercise influenced cerebral vascular structure of adult mice; vascular plasticity has been shown to occur in young mice, but whether adults also exhibited this phenomenon was unknown. I used customized software to convert time series of vascular image stacks into a collection of undirected graphs, and developed an open-source browser using Python to make the analysis publicly accessible. I discovered no systemic change in capillary diameter but did observe that pruning occurred for a small number of vessels that were part of short cycles in the vascular network, potentially indicating that cerebral vasculature selectively removes vessels that make the local network inefficient.