Blog

Recognizing Gesture: A Vital Feature in Multimodal Communication

Amy Corbitt — Mon, 07 Apr 2025 18:20:06 +0000

Recognizing Gesture: A Vital Feature in Multimodal Communication Amy Corbitt Mon, 04/07/2025 - 12:20

Hannah VanderHoeven and Nikhil Krishnaswamy

Hannah VanderHoeven is a Ph.D research student at Colorado State University (CSU) who holds a MS in Computer Science from CSU. As part of iSAT, Hannah works with Dr. Krishnaswamy on automatic gesture recognition, multimodal processing, and system development to help extract context from group work and process it in an effective and meaningful manner.

Nikhil Krishnaswamy is an Assistant Professor of Computer Science at Colorado State University. His expertise lies in natural language processing, including multimodal methods, human-AI collaboration, and embodied cognition. He works to research and develop AI systems that complement and support human expertise and effort.

When working together to achieve a goal, especially in collaborative problem solving (CPS) tasks, students are likely to use many different forms of communication. These include not only language but also actions, gestures, and other nonverbal cues. Gestures, in particular, are rich in meaning and often complement spoken language. Certain concepts like pointing to something in physical space is usually expressed through gesture(s) alone. Because of this, recognizing gestures accurately and in real time is crucial for multimodal language understanding. This is especially important when designing well-rounded AI systems that can interpret the state of the collaboration for a given task.

When combined with speech, nonverbal behaviors such as gestures can provide crucial context information by helping an AI interpret how a person interacts with others and their surrounding physical space. Deictic gestures, such as pointing, allow participants to indicate locations in their workspace, helping speakers specify a referent or target of interest in their environment. These targets of interest can align directly with demonstrative pronouns in speech (e.g., when you say: "this one" or "those") and can be used to add clarity to otherwise pretty vague statements. By incorporating this information, an AI agent can gain a more complete understanding of a scene than it would from speech alone.

Hand detection tools like MediaPipe, an open-source library developed by Google, aid in gesture recognition. MediaPipe provides joint locations (landmarks) of detected hands on a frame-by-frame basis. These landmarks can be used to train gesture recognition models for a variety of applications. By leveraging this data, we can more efficiently train lightweight classifiers to detect a wide range of gestures.

Most vision-based gesture recognition projects focus on detecting hand poses in a single video frame, which is useful for human-computer interaction, such as recognizing static commands to control a user interface. However, there is less focus on recognizing the distinct “phases'' of more complex gestures that are more likely to occur throughout interactions between humans. As participants work through small group tasks, there is likely to be more emphasis on complex multiframe gestures, such as pointing, that can accompany multiple words spoken.

Unifying gesture and language can result in enriched natural language, providing a text-only large language model (LLM) with a much more complete depiction of a scene. Imagine a scene where someone points to a green block and says, “I think that one weighs twenty grams.” From the speech transcription alone, an AI agent powered by an LLM wouldn’t know what “that one” refers to. However, by integrating gesture recognition and object detection, these two channels can be unified into a clear, unambiguous description: “I think [the green block] weighs twenty grams.” In this way, gesture recognition and other forms of multimodal interpretation serve as a means of giving “blind” systems “eyes.”

Below are some additional resources for you to check out!

References

Kendon, A. (1997). Gesture. Annual review of anthropology, 26(1), 109-128.

Lascarides, A., & Stone, M. (2009). A formal semantic analysis of gesture. Journal of Semantics, 26(4), 393-449.

Lugaresi, C., Tang, J., Nash, H., McClanahan, C., Uboweja, E., Hays, M., ... & Grundmann, M. (2019, June). Mediapipe: A framework for perceiving and processing reality. In Third workshop on computer vision for AR/VR at IEEE computer vision and pattern recognition (CVPR) (Vol. 2019).

McNeill, D. (2008). Gesture and thought. In Gesture and thought. University of Chicago press.

VanderHoeven, H., Blanchard, N., & Krishnaswamy, N. (2023, July). Robust motion recognition using gesture phase annotation. In International conference on human-computer interaction (pp. 592-608). Cham: Springer Nature Switzerland.

Off

Traditional

White

Collaborative Learning: How Does Learning Happen During Collaboration and How Can We Support It?

Amy Corbitt — Thu, 20 Mar 2025 14:47:22 +0000

Collaborative Learning: How Does Learning Happen During Collaboration and How Can We Support It? Amy Corbitt Thu, 03/20/2025 - 08:47

Indrani Dey & Sadhana Puntambekar

Indrani Dey is a PhD student in Learning Sciences at the University of Wisconsin-Madison, studying under Dr. Sadhana Puntambekar. At iSAT, she works with Dr. Puntambekar to investigate how students’ verbal and nonverbal interactions during group work reflect their engagement and learning, and explore how these insights can guide the design of AI technologies for classrooms.

Sadhana Puntambekar is the Sears-Bascom Professor in Educational Psychology at the University of Wisconsin-Madison. Her expertise lies in scaffolding and Design-Based Research. She investigates how middle students learn science, through the design and use of interactive technologies.

Collaboration and collaborative learning are key 21st century skills that increase classroom engagement, participation, and learning. But what does it actually mean for students to collaborate, and do so well? Does putting students in a group mean that they will work together effectively? The short answer is no. While some groups may have rich, engaging discussions, others often struggle. One or two students may be doing all the talking in a group while others stay silent; some just go along with their group’s decisions, or discussions may leave members dissatisfied. Does being silent indicate that a student is not participating? Does disagreement mean that the group is not collaborating well? These are common examples of group dynamics observed in real classrooms, which then leads to the question: What does effective collaboration look like, and how can we help students get better at it?

Collaboration and Learning: How Are They Related?

Learning is an inherently social process. When students collaborate on a joint activity or problem, multiple things may happen. Group members can share their ideas and hear other ideas that they may not have thought of on their own. They can ask each other questions to clarify their understanding. Explanations prompt students to reason through their thinking, which, in turn, helps them refine their own ideas and build a deeper understanding. Adding to or challenging the ideas of other peers can strengthen critical thinking skills. Refining ideas can deepen overall understanding and thereby improve learning. Collaboration can also nurture socio-emotional, self-regulatory, and communication skills—skills that are valuable not just in school but beyond the classroom as well.

Why Does Collaboration Break Down?

Effective collaboration demands a lot from students. They need to have rich discussions that move their thinking forward, consider multiple perspectives, and unify those perspectives in order to find common ground to reach shared goals. This requires a high cognitive demand that can be challenging to sustain over long periods. Students also need to stay engaged and regulate themselves and their group to stay on track and complete activities. Given all of these demands, collaboration can sometimes break down or may not be as effective for learning.

Researchers have developed different strategies to support peer interactions during collaboration. For example, assigning roles to individuals in a group may help the group work proceed more smoothly; Think-Pair-Share encourages students to discuss ideas in pairs before sharing with the class; Jigsaw facilitates discussions by having each student first become an “expert” in a topic and then share their unique knowledge with their group to solve a problem. These strategies were designed to facilitate more structured discussions, encouraging all group members to engage and participate, and therefore collaborate. But even with this structure, students may still struggle. To better understand why collaboration can still be challenging, even with these embedded participatory structures, we need to examine student interactions during collaboration.

Verbal and Nonverbal Interactions During Collaboration

In both characteristics of effective collaboration and its support strategies, there is often an emphasis on verbal participation. Educational research on collaborative learning often emphasizes analyzing students’ verbal interactions to understand their collaboration and learning. However, nonverbal interactions also play a fundamental role. For example, imagine a student shares an idea but their group members ignore them by avoiding eye contact or otherwise indicating that they are not listening—the student might stop contributing. They may then work on their own or disengage from the task altogether through body language, for example, such as not working on the task, looking around but not focusing on the group activity, or leaning away.

Students may not feel comfortable speaking up for many reasons, including differences in personality, culture, or language. Does that mean they are not engaged in the work or are not learning from their peers? Not necessarily. Students show engagement by sharing attention on joint tasks with their group members, making eye contact or nodding, leaning in, or performing the task, even if they do not speak as much. But simply showing behavioral engagement does not indicate whether students are learning or not.

How Can We Support Collaboration?

To support collaborative learning, we need to provide support not only for what students say but also do. Observing students’ verbal and nonverbal behaviors to assess their engagement and learning before providing necessary support may help create an environment where all students can participate.

Pedagogical strategies such as assigning roles or jigsaw to scaffold students’ verbal participation and thinking need to be complemented by supporting students’ nonverbal engagement. For example, when assigning roles to students, teachers could have students rotate between roles (e.g., a leader/decision-making role versus a note-taker) that play to their strengths or help them develop areas where they may not be as confident. Similarly, using tools or technologies that allow students to share and build on ideas through digital formats may also help students think through their ideas before sharing with their group.

AI-based tools are an emerging method for supporting collaboration. These tools collect data and provide analytics, often through visual displays (e.g., teacher or student dashboards). At iSAT, our Community Builder or CoBi helps students improve their communication and self-regulation skills, while the Jigsaw Interactive Agent (JIA) supports students’ interactions and knowledge building during a jigsaw activity. Both contribute to helping students build collaboration skills.

By recognizing challenges students face during collaboration, and identifying areas where it may break down, we can better support students with strategies and tools they need to collaborate effectively and learn together.

References:

Barron, B. (2003). When smart groups fail. The journal of the learning sciences, 12(3), 307-359.

Bereiter, C. (2005). Education and mind in the knowledge age. Routledge.

Dillenbourg, P., & Jermann, P. (2007). Designing integrative scripts. In F. Fischer, I. Kollar, H. Mandl, & J.M. Haake (Eds.), Scripting computer-supported collaborative learning (pp. 275-301). Springer, Boston, MA.

Engle, R. A., & Conant, F. R. (2002). Guiding principles for fostering productive disciplinary engagement: Explaining an emergent argument in a community of learners classroom. Cognition and Instruction, 20(4), 399-483.

Hmelo-Silver, C. E., & Barrows, H. S. (2008). Facilitating collaborative knowledge building. Cognition and Instruction, 26(1), 48-94.

Järvelä, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48(1), 25-39.

Kollar, I., Fischer, F., & Hesse, F. W. (2006). Collaboration scripts: A conceptual analysis. Educational Psychology Review, 18(2), 159-185.

Kolodner, J. L., Camp, P. J., Crismond, D., Fasse, B., Gray, J., Holbrook, J., ... & Ryan, M. (2003). Problem-based learning meets case-based reasoning in the middle-school science classroom: Putting learning by designä into practice. The Journal of the Learning Sciences, 12(4), 495-547.

Kreijns, K., & Kirschner, P. A. (2002). Group awareness widgets for enhancing social interaction in computer-supported collaborative learning environments: Design and implementation. Conference on Frontiers in Education (FIE 2002), pp. 14–20.

Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge University Press.

Mercer, N., & Littleton, K. (2007). Dialogue and the development of children's thinking: A sociocultural approach. Routledge.

Molenaar, I., & Knoop-van Campen, C. A. (2018). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347-355.

Puntambekar, S., & Kolodner, J. L. (2005). Toward implementing distributed scaffolding: Helping students learn science from design. Journal of Research in Science Teaching, 42(2), 185-217.

Resnick, L. B., Michaels, S., & O'Connor, M. C. (2010). How (well-structured) talk builds the mind. In R. J. Sternberg, & D. D. Preiss (Eds.), Innovations in educational psychology: Perspectives on learning, teaching, and human development (pp. 163-194). New York: Springer.

Schnaubert, L., & Bodemer, D. (2019). Providing different types of group awareness information to guide collaborative learning. International Journal of Computer-Supported Collaborative Learning, 14, 7-51.

Tabak, I., & Baumgartner, E. (2004). The teacher as partner: Exploring participant structures, symmetry, and identity work in scaffolding. In Investigating Participant Structures in the Context of Science Instruction (pp. 393-430). Routledge.

Vygotsky, L. S. (1978). Mind in society: Development of higher psychological processes. Harvard University Press.

Off

Traditional

White

Fine-tuning a Strong Language model to Enable Classroom Speech Recognition

Amy Corbitt — Tue, 25 Feb 2025 18:45:34 +0000

Fine-tuning a Strong Language model to Enable Classroom Speech Recognition Amy Corbitt Tue, 02/25/2025 - 11:45

Viet Anh Trinh

Postdoctorate Viet Anh Trinh led a project within Strand 1 to develop a novel neural network architecture that can both recognize and generate speech. He has since moved on from iSAT to a role at Nvidia, where he is continuing his work on multimodal Large Language Models.

Understanding and processing speech in classrooms can be difficult because it comes with its own set of challenges such as background noise, people’s unique speaking styles, changes in pitch, and differences in the content of speech. Kids also don’t talk like adults, which means that existing speech recognition models don’t really work well for classroom speech, especially when there isn’t a lot of labeled data available for model training. To tackle this, we use unsupervised machine learning–-a type of machine learning where a computer learns patterns from data without being given explicit instructions or labeled examples. This helps reduce the need for large amounts of labeled training data while at the same time better capturing how kids speak and communicate. This approach holds lots of potential for future applications in education, healthcare, and more—enabling more inclusive AI systems tailored to younger users.

Our speech processing team has been working on a Discrete Multimodal Large Language Model (DMLM), which is capable of flexibly translating data across modalities to perform various speech processing tasks. DMLM is one of the first discrete token-based decoder-only models; it can translate to and from text, speech and images. Combining audio, images, and text helps the model better understand speech context. To improve its performance, we fine-tune a strong language model by blending unsupervised learning with multimodal data to advance speech recognition technology.

What we have found so far is that our LLM-based approach, by harnessing multimodal inputs, can outperform state-of-the-art models of similar size. We achieve significantly more accurate speech recognition in both noisy and quiet environments as well as clearer and more reliable understanding in real-world scenarios.

LLMs can help address the complexities of children’s speech recognition. Looking ahead, we aim to expand this work by exploring multilingual capabilities ( including English-to-Spanish speech translation) and visual information (including equations, slides, and images from lecture materials in video conference formats); leveraging lip movements in classroom settings could also further enhance ASR performance in noisy environments. All of these advancements will improve educational tools and enrich children’s interactions with AI systems, fostering meaningful progress in inclusive and accessible technology.

Off

Traditional

White

Tackling Bias in Automatic Speech Recognition - Two Examples From Our Ongoing Work

Amy Corbitt — Wed, 22 Jan 2025 20:01:43 +0000

Tackling Bias in Automatic Speech Recognition - Two Examples From Our Ongoing Work Amy Corbitt Wed, 01/22/2025 - 13:01

By Wayne Ward and Rosy Southwell

Rosy Southwell is a postdoc research scientist at CU Boulder who holds a PhD in Cognitive Neuroscience from University College London, UK and an MS in Natural Sciences from University of Cambridge, UK. As part of iSAT, Rosy works on automatic speech recognition and processing to help extract as much information as possible from noisy audio recorded in the classroom.

Dr. Wayne Ward is a Research Professor at CU Boulder whose research involves applying supervised machine learning to the tasks of automatic speech recognition, dialog modeling and extracting semantic representations from speech and text. His recent focus has been on applying these technologies to questing answering and virtual tutoring systems.

AI systems that are designed to offer real-time classroom support need to be able to understand what students are saying—and do so with high accuracy. This requires Automatic Speech Recognition (ASR), which is the process where spoken language is automatically converted into text.The text can then be used by an AI to understand how students are working together.

A key consideration when developing an AI system is how it is trained and the data it learns from. In the context of speech recognition, the AI is trained on a large collection of audio recordings from many different speakers. These systems have become a lot more accurate in recent years, especially for adults from particular demographics (native English speakers, white, US accent), but this does not reflect the diversity of speakers in the world, of course.

The question is: how will an AI perform in a classroom setting where it is mostly children and teenagers who are talking? They may come from diverse backgrounds, speak in a variety of accents, and use gen Z slang. In our work, we have found this domain to be significantly challenging for existing speech recognition systems—in part because it is still very uncommon for children's speech to be used for training an AI system. Let’s discuss two variables in our data where ASR shows its weaknesses: age and race.

First, let's look at how we can adapt ASR to work better for students of all ages. We have a lot of training data from adults and elementary school students, and a small amount of test data from 9th graders. The word error rate (WER), which is the percentage of words that get transcribed wrongly by the model, can help us figure out what’s going on. For models that have been trained on adult speech, WER is 8% for adult speakers, but on our evaluation set of 9th grader speech, WER reaches up to 56%. In other words, the ASR gets it wrong more than half of the time! The WER for elementary school kids’ models when tested on kids of the same age is about 9%, but for 9th graders it jumps to a whopping 46%. This shows that models trained on one age group do not really generalize well to a different age group.

We can make improvements by starting with adult models and using a process called “fine-tuning” where a model goes through additional training on different data to adapt it to a new domain. Fine-tuning on elementary school kids’ speech did improve the WER slightly to 41%. To address the scarcity of training data for specific age groups, we are working on new techniques to adapt models to different age groups using very small amounts of age-specific data.

Second, there is concern within the AI community about “accuracy biases” in AI systems that can disadvantage certain demographics such as non-white speakers. As a team we have often discussed bias in AI models, identifying places where AI could be affected by bias, and how we can mitigate this. In some of our recent work, we found that a popular ASR tool on which we base automatic feedback for tutors is 24% less accurate for Black speakers when compared to white speakers because the acoustics of their voices are not as well understood by the AI. Without access to the data they used for training the AI, one likely reason for this accuracy bias is that the model was not shown enough speech from Black speakers when the AI was trained. If an AI can't "hear" individuals accurately, then this has consequences for its ability to provide helpful feedback! We used fine-tuning to reduce the accuracy gap between Black and white tutors by around a third, and also improved the ASR accuracy for both groups of tutors. But from just these two examples, it is clear that there is still a lot more work that needs to happen to overcome these bias issues in the future!

Off

Traditional

White

Meeting the Triple Challenge for AI Implementation – New Pedagogies, New Technology, New Topics

Amy Corbitt — Sun, 08 Dec 2024 20:30:31 +0000

Meeting the Triple Challenge for AI Implementation – New Pedagogies, New Technology, New Topics Amy Corbitt Sun, 12/08/2024 - 13:30

Mon-Lin Monica Ko

Mon-Lin Monica Ko is an Assistant Research Professor at the Institute of Cognitive Science at CU Boulder and a team lead for Strand 3 at iSAT. Her work focuses on promoting and studying classroom interactions that support students' engagement in disciplinary practices. At iSAT, she investigates how inclusive co-design processes can empower teachers and students with diverse identities to better understand how AI learning technologies can be used for good in their schools and communities. In addition to her role on iSAT, Ko is also currently a Research Assistant Professor at the Learning Sciences Research Institute (LSRI) at the University of Illinois Chicago.

Over the last couple of years, the topic of AI has catapulted into the public sphere, fueled in part by the release of ChatGPT. Numerous headlines that we encounter focus on the doom and gloom of AI. Others boast its amazing capacity to mimic human intelligence. In the U.S., we’ve seen many school districts BAN the use of AI. The explosion of AI-powered tools also surfaces important questions for learning scientists, curriculum developers, and teacher educators such as: “what do we think students need to learn in order to use AI safely, ethically, and responsibly ...and what does it look like for teachers to model these practices?” Clearly, we are at an interesting juncture in conceptualizing exactly what it is that students need to learn about AI.

Our iSAT team has been aiming to prepare students to be critical consumers and builders of ethical AI systems. Rather than simply supporting students in learning about AI systems, we want them to interrogate how these systems work, what purposes they serve (and for whom), and what societal impacts these technologies can and should have. This vision for student learning requires us to examine and reflect on what it is that students need to know and be able to do when it comes to AI. It also requires us to think about how it is that we position students as the ones who envision, build and critique the AI systems of the future.

Moving toward this vision for student learning has led us to encounter what we’re calling the triple innovation challenge. First, AI is an emerging field and there are no standards yet for what should be taught – and when – in K-12 classrooms. Teachers are also new to AI and are learning about it as they are teaching it in their classrooms. Second, our work in schools is part of a movement that seeks to promote student-centered learning through curriculum-based professional learning. All of our units are created using a so-called storyline approach. This, too, is novel to teachers. Our third challenge involves integrating our AI partners into existing socio-technical infrastructures of our partner schools and districts, each of whom have different capacities for integrating these emerging technologies into classrooms. These three innovation challenges present both challenges and opportunities to our work.

Over the past four years, we’ve been partnering with community organizations, teachers, and school districts to figure out how we can address this triple innovation challenge. These partnerships have led to modifications and revisions to our curriculum materials. Our conversations with teachers and students have resulted in a deeper understanding of how we can better integrate our AI partners into curricular routines. Our work inside classrooms has sharpened the need to better understand what teacher-student-AI partner interaction can and should look like, and how these interactions can best support collaborative learning inside classrooms.

What we’ve learned from our repeated engagements with various stakeholders is that we need to cultivate a dynamic learning environment to sufficiently prepare students to understand, critique and build ethical AI systems. We need rich curriculum materials that tackle questions and phenomena that students are interested in – something that really invites and builds on their everyday experiences. We also need to create AI partners that are not just a black box – but something that can be interrogated and questioned and opened up for inquiry. We also need to support teachers in enacting these AI-partner embedded curriculum materials – and provide ample learning opportunities for teachers to do their own learning about AI. Lastly, we need to leverage and build on student experiences in order to support their own learning.

This is really important – the AI partner we’ve developed can’t stand on its own to reach our goals for students. We need to think about how all these pieces fit together to enable meaningful student learning. As we look to Year 5 of the iSAT project, we are continuing to rely on our work inside classrooms, alongside teachers and students, to better understand how we can create this dynamic learning environment.

Off

Traditional

White

What is Human-AI Teaming in Three Levels of Complexity in Learning Environments?

Amy Corbitt — Mon, 11 Nov 2024 22:58:05 +0000

What is Human-AI Teaming in Three Levels of Complexity in Learning Environments? Amy Corbitt Mon, 11/11/2024 - 15:58

By: Ray Hao , Lucrezia Lucchi and Jamie Gorman

Ray Hao is a PhD student and Fulton Fellow in Human Systems Engineering at Arizona State University, studying under Dr. Jamie Gorman.

Lucrezia Lucchi is a Psychology PhD student in the Dynamics of Perception, Cognition, & Action Lab at Arizona State University. Lucrezia has a background in Exercise Physiology and Human Movement Sciences.

Professor Jamie Gorman is an expert in modeling and measuring coordination dynamics in human and human-machine teams in The Polytechnic School at Arizona State University.

In today’s rapidly evolving digital world, parents are faced with growing questions about how to provide the best education for their children. One increasingly important factor in education is Human-AI teaming where students collaborate with artificial intelligence (AI) technologies to enhance learning. But what does it actually mean for students and AI systems to collaborate as teams, and just how complex can this process be?

What is Human-AI Teaming in Learning Environments?

Human-AI teaming in learning environments refers to the collaboration efforts of humans (teachers, students) and AI systems. It aims at enhancing educational outcomes by combining the unique strengths of both. In these settings, AI systems do not replace humans; instead, they work side by side with teachers and students to support and improve the learning process. Human-AI teaming can vary in complexity, starting with basic AI assistance and evolving into more collaborative teamwork and community, each offering distinct opportunities to enhance learning.

Level 1: Basic AI Assistance – Personalized Learning

At the foundational level, AI helps students by scaffolding personalized learning experiences and guiding students through customized learning experiences. This approach focuses on individual optimization by helping students progress at their own pace. Many tools on this level offer feedback, hints, and explanations tailored to the student’s needs but without direct group collaboration in the learning process.

Example: Duolingo, for instance, provides an adaptive and interactive experience where users progress through language lessons tailored to their learning pace. It adjusts the difficulty of lessons based on user performance, offering targeted hints and explanations to address specific challenges. Similarly, Khan Academy personalizes learning in subjects like math and science, suggesting exercises that align with the learner's current understanding. The platform offers immediate feedback, allowing users to correct errors in real-time and supporting a structured, individualized learning journey.

Level 2: Collaborative AI Partners – Learning Together

At this level, AI not only tutors students but also encourages collaboration in the learning process. It works alongside students in group projects, providing real-time insights, asking guiding questions, and even learning from interactions. By encouraging critical thinking and teamwork, AI makes learning more interactive and engaging. The Institute for Student-AI Teaming (iSAT) envisions classrooms where AI and students collaborate on problem-solving tasks, helping each other through challenging concepts while developing critical thinking skills (D’Mello et al., 2024).

Example: Our AI partner CoBi (Community Builder) is a great example of this. Designed for classroom use, CoBi helps groups of learners improve their collaboration skills by focusing on how they interact and work together. Another example is Kahoot! Team Mode, which supports AI-powered analytics to adjust in real time according to the group's collective performance. This platform has become a popular tool among educators to enhance students' collaboration and teamwork skills.

Level 3: AI-Enhanced Communities – Knowledge Building

At this level of complexity, AI is deeply integrated into the classroom environment, working as both a facilitator and a teammate for collaborative learning. It assists teachers in managing class-wide discussions, offers insights into student participation, and highlights key moments of critical thinking or engagement, nurturing an inclusive learning community where every voice is valued (Langer-Osuna, 2017).

Example: iSAT’s AI partner JIA (Jigsaw Interactive Agent) is designed to enhance collaborative learning by supporting student interactions and promoting effective group dynamics through real-time prompts and interventions. JIA encourages students to actively listen, share ideas, and build on each other's contributions in jigsaw activities, fostering deeper engagement and understanding. IBM Watson Education Classroom also offers an additional approach to personalized learning by helping teachers monitor and manage content, providing insights into student participation and needs to support learning outcomes for the entire class.

Potential Challenges & Concerns

As technology continues to evolve, we can expect even greater collaboration between humans and AI to enhance the way students learn, making education more personalized and effective than ever before. Despite the potential benefits, such as tailored learning experiences, improved collaboration skills, and increased accessibility to educational support, there are also challenges and concerns associated with Human-AI teaming.

One major concern, according to Alrazaq and colleagues (2023), is the risk of over-relying on AI, which could hinder the development of critical thinking and creativity in both teachers and students if not used in a balanced way. For example, students may become accustomed to relying on AI for quick answers or guidance, which can impede their capacity for thorough research and independent insight formation, potentially diminishing critical faculties. Similarly, teachers might depend on AI gaining insights for student assessment and feedback, potentially bypassing deeper observation and reflection on individual learning needs. This reliance can deter students from developing skills that are crucial for academic and professional success (see the studies of Koos & Wachsmann, 2023, and Zhai and colleagues, 2024, respectively). Navigating the use and deployment of AI in education also poses significant challenges, as integrating AI systems for personalized learning and automated assessments can lead to inconsistencies when evaluating students' progress compared to traditional methods. Additionally, privacy concerns arise as AI systems collect and analyze student data in various ways, raising questions about how this data is used and protected. Lastly, there is the issue of equity. The National School Boards Association defines educational equity as “the intentional allocation of resources, instruction, and opportunities according to need, requiring that discriminatory practices, prejudices, and beliefs be identified and eradicated.” Not all students have equal access to technology, and if Human-AI teaming becomes central to education, it could widen the digital divide between those with access to high-quality AI tools and those without. Ensuring that these tools are appropriately integrated as school resources requires widespread education on their availability and growing relevance to academic curricula.

Why Human-AI Teaming Matters for Our Children

Incorporating AI into learning environments isn't just about optimizing test scores – it is about preparing students for the future. As AI continues to evolve, the ability to work alongside AI partners will be a crucial skill. Educational research highlights that interactive and collaborative approaches to learning are the most effective, supporting the goal of incorporating AI to help students adaptively solve real-world problems, rather than focusing solely on individual mastery of narrow topics, while also developing critical thinking and teamwork skills that are vital for success in today’s workforce (see NASEM, 2018, Fiore and colleagues, 2018, and D’Mello and colleagues, 2024, for more information). By teaming up with AI at multiple levels of complexity, students have an additional platform for learning to collaborate effectively, think critically, and creatively problem-solve. These collaborative experiences empower students to succeed in the classroom and beyond, equipping them with the skills they need to navigate an increasingly complex and technology-driven world.

AI has entered the mainstream in classrooms, and there are different visions for how AI should be used to educate students. One vision, according to Vee (2024), is to replace human teachers with bots that are subject matter experts, capable of teaching any subject. Another approach, which we embrace, is human-AI teaming, in which students and teachers team with AI to enable new concepts of learning. We feel this latter approach may better engage learners with each other and their teachers by supporting collaboration, rather than students learning to interact primarily with closed, AI-based systems that may lack the richness and creativity of human interaction.

References

Abd-Alrazaq, A., AlSaad, R., Alhuwail, D., Ahmed, A., Healy, P. M., Latifi, S., ... & Sheikh, J. (2023). Large language models in medical education: opportunities, challenges, and future directions. JMIR Medical Education, 9(1), e48291. doi:10.2196/48291

D'Mello, S. K., Biddy, Q., Breideband, T., Bush, J., Chang, M., Cortez, A., ... & Whitehill, J. (2024). From learning optimization to learner flourishing: Reimagining AI in Education at the Institute for Student‐AI Teaming (iSAT). AI Magazine, 45(1), 61-68. https://doi.org/10.1002/aaai.12158

Fiore, S. M., Graesser, A., & Greiff, S. (2018). Collaborative problem-solving education for the twenty-first-century workforce. Nature Human Behaviour, 2(6), 367-369. https://doi.org/10.1038/s41562-018-0363-y

Koos, S., & Wachsmann, S. (2023). Navigating the Impact of ChatGPT/GPT4 on Legal Academic Examinations: Challenges, Opportunities and Recommendations. Media Iuris, 6(2). https://doi.org/10.20473/mi.v6i2.45270

Langer-Osuna, J. M. (2017). Authority, identity, and collaborative mathematics. Journal for Research in Mathematics Education, 48(3), 237-247. https://doi.org/10.5951/jresematheduc.48.3.0237

National Academies of Sciences, Division of Behavioral, Social Sciences, Board on Science Education, Board on Behavioral, Sensory Sciences, ... & Practice of Learning. (2018). How people learn II: Learners, contexts, and cultures. National Academies Press.

National School Boards Association. (n.d.). Center for Public Education: Equity. Retrieved November 6, 2024, from https://www.nsba.org/Services/Center-for-Public-Education#:~:text=Equity,beliefs%20be%20identified%20and%20eradicated.

Vee, A. (2024). AI pioneers want bots to replace human teachers - here’s why that’s unlikely. The Conversation. Retrieved 10/10/2024 from https://theconversation.com/ai-pioneers-want-bots-to-replace-human-teachers-heres-why-thats-unlikely-235754.

Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students' cognitive abilities: a systematic review. Smart Learning Environments, 11(1), 28. https://doi.org/10.1186/s40561-024-00316-7

Off

Traditional

White

Considering Learning and Evidence of Impact in Evaluating the Potential of AI for Education

Amy Corbitt — Tue, 29 Oct 2024 16:10:09 +0000

Considering Learning and Evidence of Impact in Evaluating the Potential of AI for Education Amy Corbitt Tue, 10/29/2024 - 10:10

William Penuel

William R. Penuel is a professor of learning sciences and human development in the School of Education at the �鶹��Ƶ. His current research examines conditions needed to implement rigorous, responsive, and equitable teaching practices in STEM education. At iSAT, he is a Co-Principal Investigator and Co-Lead of Strand 3 - which focuses on inclusive co-design processes to empower stakeholders with diverse identities to envision, co-create, critique, and apply AI learning technologies for their schools and communities.

As school and district leaders, you are used to building planes while flying them. But the advent of AI—specifically Generative AI—in classrooms has caught many of us off guard and not sure what airspace we’ve entered. Generative AI is the technology behind popular tools like ChatGPT, as well as tools today that use AI to help teachers build lesson plans and assessments for use in their classrooms. It’s a specific kind of AI that learns from the data it’s been fed (such as text, video, or images) to create new content. If you’ve tried it out, you may be impressed both by its capabilities to simulate human interaction, as well as its limitations.

As an education leader, Generative AI presents many interrelated challenges to you, to teachers, to parents, and to students pertaining to safety, transparency, and ethics. In this blog post, we want to focus on two other central issues that Chief Academic Officers, district technology leaders, principals, and instructional coaches should keep in the foreground when evaluating the potential integration of AI into schools: learning and evidence of impact. Learning has to do with both our goals for learning and how we support them. Evidence of impact has to do with the power and limits of tools to achieve those learning goals. Good evidence also involves evidence of what’s required of teachers to implement tools well, to achieve benefits for students. Both these considerations are important in evaluating Generative AI and other tools, but often they live in the background of discussions about Generative AI.

Take the discussion of the potential of Generative AI for personalization and differentiation of learning. This is chief among the advantages that advocates of AI tout. The questions to consider are: What kinds of learning goals can Generative AI support? What do we know about the potential of Generative AI for supporting these goals?

Intelligent Tutors Help Personalize Individuals’ Mastery of Discrete Knowledge and Skills

There is more than 50 years of research on intelligent tutoring systems (ITSs) that we can draw on to give us a sense of what learning goals AI for personalization can support. ITSs are trained when their developers subdivide knowledge to be taught into smaller components—skills, abilities, and concepts—allowing ITSs to recommend tasks based on a student’s mastery level. There’s a large body of evidence of impact that suggests that for the kinds of problems ITSs are used to help students with, they do as least as well as human tutors do in supporting learning.

However, while AI excels at guiding students toward specific, well-defined learning goals (like solving a math problem), it struggles with more open-ended tasks where multiple solutions exist, or where collaboration and dialogue are essential. Further, it may limit deeper engagement and valuable experiences like productive struggle or peer collaboration. The evidence base applies only to well-designed ITSs, as well. Many of the Generative AI tools today can’t achieve the results of the best ITSs. While they are good at handling requests in everyday language, many of these tools still give inaccurate answers to math problems students encounter in schools.

This is not to say that Generative AI won’t become more capable of solving math problems or helping support critical thinking, teamwork, and real-world problem solving in the future, but there is not strong evidence of impact for achieving these learning goals. There is even less evidence related to what’s needed to prepare teachers to use these tools well. There’s reason to be skeptical, then, about claims that the current class of tools of Generative AI can support these goals.

AI Can Support Collaborative Problem Solving in Inquiry-Rich Environments

There’s an equally rich body of evidence of impact for a set of AI tools that support collaborative learning. For more than two decades, the field of computer-supported collaborative learning has created and tested different tools focused on fostering group awareness and giving students feedback on small groups’ cognitive and social dynamics. A review of these kinds of group awareness tools show improvements to students’ knowledge and skill, as well as group task performance and social interaction in collaborative learning. The relevance of these findings for K-12 schools, though, is not as clear, because many of these tools were designed for online environments in higher education.

Here’s where emerging research comes in – the kind designed to build evidence of impact grounded in a robust vision for teaching and learning. The Institute of Student AI-Teaming is developing AI partners—the Community Builder (CoBi) and the Jigsaw Interactive Agent (JIA)—that perform the key functions of group awareness tools. These tools are intended to be integrated with rich curricula that focus on collaborative problem solving in STEM. These tools do something very different from what Generative AI tools as currently used to plan instruction or support personalization do: they help students learn to collaborate more effectively and equitably. They support a different kind of learning, too, one that is focused on students figuring out ideas and solving problems together, using disciplinary practices from STEM that are targeted in today’s standards. And while we are still gathering evidence of impact, we already know that students are using some collaborative solving skills more when they are using an AI partner to support their learning. We aim to make these partners—and the instructional materials to teach about AI—available to schools for free in the coming year.

Questions to Ask �鶹��Ƶ Learning and Impact

AI is here to stay, and as a leader, you know you have an obligation to approach how to use AI responsibly and ethically to achieve your vision for teaching and learning. No doubt, AI may now or in the future be useful for increasing efficiency in how teachers plan and how students develop discrete knowledge and skill. As vendors continue to rush to offer generative AI products to schools and districts, it’s important to ask three questions:

What kind of learning does this tool support?

What kind of preparation do teachers need to use the tool well?

What evidence of impact is there for the claims being made about Generative AI?

Integrating AI into classrooms is likely to lead to changes in how teachers teach and how students learn. Teachers will need support in learning how the AI works, and how to use AI tools to support teaching and learning that is consistent with what we know about how students learn. A generative AI chat bot doesn’t understand how people learn, no matter how skillful its interactions seem. That leaves it as your responsibility as a critical consumer of AI tools to ask tough questions of vendors about their ideas about teaching and learning and to demand they present evidence of bold claims about the power of AI.

Now is a moment when we are all particularly open and keen to learn about AI, and it is as imperative as ever to create opportunities where educators and leaders can learn together about the potential and limits of Generative AI and other tools that support learning goals for collaborative problem solving. We not only have to be “in the loop”: as decision makers about teaching and learning, we need to stay at the center, working at a pace that protects both our children and takes care of our visions for teaching and learning and that follows evidence more than hype.

Off

Traditional

White

A High Level Overview of Building the iSAT MakeCode Activity Logging Platform

Amy Corbitt — Wed, 23 Oct 2024 20:53:07 +0000

A High Level Overview of Building the iSAT MakeCode Activity Logging Platform Amy Corbitt Wed, 10/23/2024 - 14:53

Sachin Rathod

Sachin Rathod is a Software Engineer working on full-stack development of iSAT’s AI Partners with the Institute-wide team. He is also a graduate student pursuing a master’s degree in computer science at CU Boulder. His interests and expertise are in Machine Learning, Distributed Systems, and Cloud Computing.

The iSAT MakeCode Activity Logging Platform provides a robust system for tracking and analyzing user coding activity in the Microsoft MakeCode environment. It is an expanded version of the Microsoft MakeCode blocks/JavaScript code editor for the micro:bit, designed to log and track user coding activity.

We leverage AWS services to ensure that every user coding action is logged efficiently and can be used for both real-time analysis and future learning. Whether users are looking to monitor coding sessions or integrate this data into machine learning models, this platform offers an extensible solution to track and store all the necessary data.

In this blog post, we provide the technical architecture for the iSAT MakeCode and how it has been implemented.

Figure 1. The iSAT MakeCode block editor.

The iSAT MakeCode Activity Logging Platform provides a deeper insight into how users interact with the micro:bit block-based editor, logging every code edit into a NoSQL database AWS DynamoDB for future analysis enabling both real-time monitoring and post-session review of coding activities.

The platform is built around two key components: the Extended Web-Based Micro:bit Block Editor and the Back-End Data Logging API Server. The web-based editor serves as the main interface, allowing users to interact with a blocks/JavaScript editor. We started with the open source Micro:bit Block Editor code base and enhanced it to track and log coding activities, including block additions, deletions, and code modifications.

The back-end server is responsible for logging each coding activity to a database, allowing developers and researchers to analyze user behavior and feed the data into machine learning pipelines. The server operates in the AWS cloud and receives logging messages sent by the user’s browser as they interact with the block code editor. Each edit version is logged to the database in MakeCode’s JavaScript format along with a timestamp and the action performed on the block (create, delete, modify), providing a transcript of the progression from the start to finish of their coding session. Each JavaScript version can be analyzed, and researchers can paste the JavaScript back into the block code editor to see a visual representation of the code blocks at the given time in the user’s coding session.

iSAT uses a variety of key technologies to support these functionalities including:

Microsoft MakeCode for Micro:bit: Used as the base platform for coding in blocks and JavaScript.
AWS DynamoDB: Stores all the coding activities.
AWS ECS (Elastic Container Service): Hosts both the frontend and backend services.
AWS Fargate: Provides the infrastructure for running containers and services.
Node.js: Used for backend server logic and communication with DynamoDB.

How It Works

The MakeCode Activity Logging Platform is built to be scalable and efficient, leveraging cloud-native technologies on AWS. The below points cover typical user actions and data flow within the system.

User Action Flow:
- When a user requests for iSAT MakeCode Activity Logging Platform (front-end application), an API Gateway (shown in the top left of Fig 2. Architecture diagram) manages and routes the request to application load balancer. The load balancer then routes the request to the next available AWS Fargate tasks (containers) that serve the front-end application.
- Upon successful response, a user logs into the MakeCode system and joins a study session by entering their Study ID and Session Code.
- The user proceeds to work on coding tasks within the MakeCode editor, usually provided as tutorials.
- All actions, such as block additions, deletions, and modifications, are sent across to the Back-End Data Logging API Server deployed on AWS ESC (shown in the top right of Fig 2. Architecture diagram). The backend-server logs these actions to AWS DynamoDB (shown in the bottom right of Fig 2. Architecture diagram) in real-time.
Logging System:
- Each code modification is stored in its MakeCode JavaScript format, meaning you can both analyze the logs and re-input them into the MakeCode editor to visualize the user’s coding process step by step.
- All events are timestamped, making it easy to track the coding session's progression.
Deployment:
- The frontend and backend services are deployed on AWS ECS clusters. AWS Fargate containers for both services are managed using ECS, ensuring high availability and ease of scaling.
- An API Gateway manages and routes incoming requests to the appropriate service, whether it’s a frontend action or a backend log storage.
- AWS Auto Scaling Groups and Application Load Balancers ensure that the system can handle varying loads, ensuring both scalability and reliability.

Figure 2. Architecture Diagram

Installation and Setup

To use this platform, developers need:

An AWS Cloud Account to set up and deploy the services.
GitHub to access the platform’s code and deploy it onto AWS ECS.
Basic knowledge of AWS services (like DynamoDB, and ECS, and Fargate) and the Microsoft MakeCode editor.

Please contact our team at Info.AI-Institute@Colorado.edu for more information and to schedule a consultation.

The iSAT MakeCode Activity Logging Platform provides a robust system for tracking and analyzing user coding activity in the Microsoft MakeCode environment. It is an expanded version of the Microsoft MakeCode blocks/JavaScript code editor for the micro:bit, designed to log and track user coding activity.

Off

Traditional

White

The iSAT Blog in Year 5

Amy Corbitt — Mon, 07 Oct 2024 21:43:02 +0000

The iSAT Blog in Year 5 Amy Corbitt Mon, 10/07/2024 - 15:43

As we enter our fifth year as an Institute, we’re excited to expand the iSAT blog. One of our principal goals is to address the central challenge of how to promote deep conceptual learning via rich socio-collaborative learning experiences for all students. We are pursuing this challenge through the development of AI partners that are intended to provide real-time classroom support and to augment collaborative learning. We are going to use this space to look at different topics in the area of Artificial Intelligence in education (AIEd) and to provide a behind-the-scenes look to show how we conduct our work.

From October through April, our blog will feature around 25 posts covering a range of topics. These will include technical discussions such as “Multimodal Large Language Models: iSAT's work on building discrete multimodal language models to facilitate multiple speech processing tasks” and “Gesture Detection: Identifying key moments when a gesture occurs and determining what the gesture was” as well as broader topics in AI like “Where Does my Data Go? A discussion on security measures for participants’ data” and “AI to Support Collaboration vs Generative AI (ChatGPT)”.

These almost weekly posts will cater to different audiences including parents, school administrators, teachers, developers, students, researchers and policy makers. Each post will be authored by team members from our various research “strands.” These strands are integral to the development of the AI Partners, each focusing on a specific area of research. In addition to covering the different strands, many blog posts will feature cross-strand collaboration, showcasing how these different research areas intersect and come together at iSAT. The diverse range of topics will provide a comprehensive behind-the-scenes look at the work of our Institute.

Off

Traditional

White

iSAT Curriculum Series: Forward to the Future: The Self-Driving Car Curriculum Unit for Middle School STEM Classrooms

Anonymous — Tue, 16 Jul 2024 21:05:34 +0000

iSAT Curriculum Series: Forward to the Future: The Self-Driving Car Curriculum Unit for Middle School STEM Classrooms Anonymous (not verified) Tue, 07/16/2024 - 15:05

By Jeff Bush

Jeff Bush is an Assistant Research Professor at the Institute of Cognitive Science at CU Boulder. He is also a theme lead at iSAT. His research focuses on the intersection of technology, STEM teacher learning and professional development with sub-topics of mathematics education, computational thinking, physical computing, formative assessment, complex instruction, Artificial Intelligence, user experience research, compassion, and equity.

In today's AI obsessed technological landscape, the Self-Driving Car (SDC) Unit puts students in the fast lane for learning innovative and responsible AI skills. Aimed at giving students technical proficiency, ethical judgment skills and hands-on collaborative skills, this unit dives deep into the complexities of programming autonomous vehicles while integrating cutting-edge AI-embedded technologies.

What is the Self-Driving Car Unit and How Does it Work?

The Self-Driving Car Unit immerses students in the exciting world of autonomous vehicles, putting them in the driver’s seat with an interdisciplinary approach. It begins with an engaging launch phase, featuring videos and discussions that highlight the real-world challenges and ethical dilemmas associated with self-driving cars. Students explore scenarios where a self-driving car must make split-second decisions, such as navigating around obstacles or deciding when to hand control over to a human operator.

As the unit progresses over 12-15 classes (typically spanning 3-4 weeks), students steer into the fundamental concepts of AI and robotics. They learn about data collection, training classifiers, neural networks, and the ethical implications of AI decision-making. Practical sessions involve programming their own miniature SDCs using platforms like the BitCar, where they implement features such as line-following, obstacle avoidance, and mode switching between autonomous and human-controlled operation.

How is This Unit Helping Kids in Classrooms - Specifically with Collaboration?

Central to the success of the Self-Driving Car Unit is its emphasis on collaboration. Students are organized into groups, each specializing in different aspects of SDC functionality like line following or object avoidance. This structure encourages teamwork as students share knowledge, brainstorm solutions, and troubleshoot challenges collectively. They then come together into a mixed group with one expert from each group; students teach about their feature to others and learn about the other two features from their peers. This peer-to-peer teaching not only reinforces understanding but also promotes effective communication and collaboration skills essential for future careers in STEM fields.

How is This Unit Tied into Our AI Partner CoBi?

Our AI partner, CoBi, plays a crucial role in enhancing the learning experience. Throughout the unit, CoBi provides support for collaboration and meta-reflection on how to best work in groups. Students collaborate in small groups and then CoBi gives them examples of how they did a good job upholding their co-negotiated class community agreements. This helps students develop these critical collaboration skills and be more adept at applying those skills in new contexts. The positive reinforcement and noticings help prevent a surveillance relationship and keep pushing students’ thinking by using actual examples from their class.

The integration of an AI partner such as CoBi aligns seamlessly with educational standards such as AI4K12 and CSTA, emphasizing computational thinking, problem-solving, and the societal implications of technology. This holistic approach prepares students not only to understand the mechanics of self-driving cars but also to critically analyze and contribute to the ongoing development of AI technologies.

In conclusion, the Self-Driving Car Unit represents a paradigm shift in STEM education, leveraging hands-on learning and AI-driven support to cultivate a new generation of innovators and problem-solvers. By exploring the three way intersection of robotics, AI, and ethics, students not only gain technical skills but also develop the collaboration and critical thinking abilities necessary to put the pedal to the metal in a technology-driven world. As we continue to build more AI-powered curricula, this unit stands as a testament to the power of integrating cutting-edge technology into educational curricula.

Off

Traditional

White