Beyond the Hype: Measuring the Real Effectiveness of AI Learning Tools
By Jessica Smagler, Head of Research and Outcomes, Kyron Learning
Proving that students are learning – especially in new and innovative programs – is harder than it sounds. And the rapid proliferation of AI tools has made this more urgent, not less. Most AI tools promise transformative outcomes but often provide little evidence to back them up. For institutions trying to make responsible decisions about what to adopt and who to trust, the question isn’t just does this work – it’s how would we even know?
As an AI learning company working with institutions across higher education, we’ve had to think hard about what meaningful evidence looks like and how to build toward it when rigorous outcome data takes time to accumulate.
What we’ve found is that measuring the impact of a genuinely new kind of educational technology isn’t a single leap to a finish line. It’s a progression from early signals to deeper evidence, and each stage has real value if you know what it can and can’t tell you. We think of it as four stages: Engagement & Confidence, Formative Signals, Persistence & Achievement, and Sustained & Verified Outcomes.
This is a framework built from practice, developed alongside institutions doing this work in real conditions. We offer it as an approach that can help any institution navigate the evidence question more clearly, whatever tools they’re evaluating.
Engagement & Confidence: Early Signs That Something Is Working
Engagement and confidence are not learning outcomes, but they are valuable prerequisites. This is especially true when introducing new modalities. Before you can measure what students have learned, you need to know whether they are showing up, staying engaged, and experiencing the instruction as credible and useful. Research in educational psychology is consistent on this: time-on-task and perceived relevance are preconditions for learning. Students who are disengaged aren’t learning, regardless of how good the content is. And students who feel confused or unsupported tend to disengage.
Early confidence data at Kyron was encouraging: more than 80% of learners reported feeling more confident after a Kyron lesson and wanted to see more of them in their courses. When a learning tool builds confidence, students are more likely to keep engaging with it. Meanwhile, at one fully online partner university, students were spending over 22 minutes on each Kyron module compared to roughly 3 minutes for traditional video content. Students who spend seven times longer with content are, at a minimum, giving learning a chance.
Formative Signals: Seeing Inside the Learning Experience
Formative signals start to tell you whether learning is actually happening. And this is where AI tools, if designed well, have a meaningful advantage over many other modalities.
A textbook can’t tell you where a student got confused. A video can’t surface a misconception. But an AI tutor, by its very nature, is witnessing student thinking in real time – the questions students ask, the reasoning they attempt, the points where they struggle, and the moments where something clicks. The question is whether a given tool is designed to make that visible and actionable.
Institutions should be asking this directly of any AI learning tool they evaluate: what formative insight does your platform generate, and how does it get into the hands of instructors?
At Kyron, formative insight is central to how the platform works. Learner misconceptions are surfaced to instructors at both the individual and section level. Instructors can access full transcripts of student interactions, seeing exactly how each learner reasoned through a problem, where they needed scaffolding, and how their understanding evolved. And we use those same interaction patterns internally to continuously improve the learner experience.
This kind of data is the bridge between early engagement signals and the outcome measures that ultimately matter. It won’t tell you whether students passed – but it will tell you a great deal about whether they’re on track.
Persistence & Achievement: Proof That Learning is Happening
Persistence and achievement are where the framework starts to deliver on its promise. Persistence – whether students stay enrolled, continue engaging, and complete what they started – is one of the most consequential measures in higher education, particularly for the populations most at risk of stopping out. Achievement measures whether they actually learned: grades, pass rates, competency demonstrations.
These are the outcomes institutions care most about. They take time to accumulate, but when they do come, they are the most direct answer to the question this whole framework is designed to answer: is this working?
The evidence across our institutional partners is compelling. At one partner institution, students who engaged with Kyron showed statistically significantly higher grades on case study assignments and stronger persistence rates across multiple health information technology courses. At a community college partner, the pass rate for a gateway English course rose from 68% to 72% — breaking the 70% threshold for the first time in the institution’s history. And at a non-profit workforce development organization, Kyron’s integration into an HR track led to a 15% increase in course completion and a 20% increase in learner retention.
These are not anecdotes. They are the validation that the earlier signals were pointing in the right direction.
Sustained & Verified Outcomes: Building Evidence That Holds
Programs that can offer solid persistence and achievement data are in a great position to start thinking about even more sophisticated evidence. That might mean longitudinal tracking – following the same cohorts over time to see whether gains persist and compound. It might mean quasi-experimental designs that allow for more rigorous comparisons across sections or populations. Or it might mean pursuing independent, third-party validation that makes findings credible beyond a single institutional context.
At Kyron, this is exactly where we are headed. In the coming months, we will be incorporating an in-app assessment tool that will allow institutions to measure learning gains over time directly within the platform. We’re also deepening our understanding of dosage: how much Kyron does a learner need to see meaningful gains? And we’re pursuing third-party validation to ensure our findings are as rigorous as students deserve.
AI in higher education will only be as good as our willingness to hold it accountable. That means building measurement frameworks that are honest about what early signals can and can’t tell us, and patient enough to follow the evidence all the way to outcomes that actually change student trajectories. The institutions that do this well won’t just make better decisions about technology. They’ll be better positioned to serve the students who need them most.
Transcript
Wes Smith (00:04.091)
All right. Here we go.
Well, welcome to the President’s Forum podcast today, Nestor. Thanks for being with us.
napereira (00:13.199)
Thank you, I’m happy to be here.
Wes Smith (00:14.639)
Hey, before we get into some of the questions about AI and about how Miami Dade is using AI, tell us a little bit about Miami Dade and your role there.
napereira (00:25.806)
Sure, absolutely. So Miami Dade College is one of the largest and most diverse institutions of higher ed in the United States. And we serve over 120,000 students annually across eight campuses and MDC Online. And really our mission is really to provide accessible high quality education to a predominantly first generation, low income and minority student population.
really the communities that need higher education the most and have historically had the least access to it. And you know it’s my privilege as Vice Provost for Academic and Learning Technologies to lead the college’s digital learning infrastructure, overseeing MDC Online, which again serves those 120,000 students, along with our Canvas Learning Management System, our AI Student Success Platform, and really
trying to move our students forward in whatever chosen path they would like to take in their careers, in their professional lives. and and I just ensure that the tools and the platforms and the experiences that we build aren’t just innovative, but that they’re effective for the MDC students and really founded to serve them.
Wes Smith (01:29.231)
Right.
Wes Smith (01:40.612)
Yeah, we talk about that a lot at the presence forum. The idea there’s a difference between, you know, innovation and accountable innovation. So, you know, you y there are a lot of cool things you could be doing, but do they decrease cost for students or do they, you know, provide a better pathway to success for students? That’s the ultimate question, right?
napereira (01:48.555)
Right.
napereira (02:01.494)
Yeah, absolutely. And that’s something I’m happy to to talk with you about today.
Wes Smith (02:05.667)
Yeah, so I know Miami Dade has been out in front on on putting AI in cr classrooms and and you’re looking at at things that a lot of schools you’re doing things a lot of schools are, you know, still talking about looking at. They they would love to get to a position to execute on it. But tell us a little bit about what first got you interested in AI in in higher education and the problems you were trying to solve for your students.
napereira (02:32.223)
Yeah, so really the you know, the honest answer is that we weren’t really chasing a trend because there’s so many trends now in AI trying to to steer you in a particular way. We were trying to solve a persistence problem. So at Miami Dade College, we we serve this student population again, where the majority are first generation college students and many are working adults and balancing jobs and family. and really a significant number of them also facing language barriers or gaps in academic preparation.
So, really, the the traditional model of office hours and email didn’t really apply and wasn’t really designed for this type of student. They really needed you know, support at 11 o’clock at night, right before something was due, maybe in the language that they think in, and really calibrated to where they actually were in their understanding and not necessarily where we assume them to be in their understanding of the material in a particular course. That’s what really
drove us toward AI and and we started with the question, you know, where are the students most likely to stop persisting? Right? What what would have to be true for that moment to go differently? And you know, the answer kept pointing to access for us. And really to this idea of timely personalized support that scales across thousands of students without proportionally scaling that cost. And that was very important to us as well.
you know, and today we’re solving for that through Lucy. And and Lucy is our live user coach for you. She’s an AI agentic platform that that we co-created. and it really functions as an AI student success coach that’s embedded right in our learning management system. she brings together proactive outreach, real-time faculty alerts, and connected tutoring and advising services so that the right intervention reaches the right student at the right moment.
And really we’ve moved from this curiosity in AI to really making a part of our infrastructure.
Wes Smith (04:33.955)
Yeah, I love it. And and you had a a specific use case for that. You wanted to work on persistence. How can we leverage AI to to advance persistence or keep our students engaged and and progressing? Is that does that sound about right?
napereira (04:50.024)
That sounds about right. And and you know, we’re starting with our online modality where students can sometimes feel a bit disconnected just because of the asynchronous nature of how these these courses run. so that’s where we started and we’re expanding that across all the modalities at D C.
Wes Smith (05:05.081)
Yeah, I love it. So you teamed up with Kyron learning on this, is that right? So walk me through that a little bit. How did you actually get it into your courses? Like how how did you take it from an idea of we need more persistence to students have Lucy or students have other assets that that they can use to stay more connected?
napereira (05:09.254)
Yes, that’s correct.
napereira (05:30.319)
Right, so Chiron’s a great partner and it really functions like an interactive lecturer inside the course itself, right? And it engages students in real-time conversational dialogue, which was very important to us. Again, because of that asynchronous nature of of online courses, we thought that this kind of lecture would be really beneficial. But the real reason is that you know the technology for us identifies and corrects misconceptions.
As they’re forming. So as the students are having this conversation with the AI, the AI is kind of understanding what are the misconceptions that these students are having and how can I help them? How can I ask deeper questions and steer them in the right direction? And this was important to us in terms of persistence. This this data, this analysis of students and their misconceptions comes about you know before a quiz or an exam might reveal them, right? So it’s
prior to that happening. And that’s really a a different and a and a meaningful different model than traditional content delivery. And the way that we introduced it, we introduced it kind of deliberately and and not all at once. And we started with our ENC 1101, which is our introductory English composition course at MDC, and really made sure that our instructional design team was trained and ready to support our faculty really before they even touched the technology and the platform. So
For us, the infrastructure and the support really existed on day one and not really as an afterthought. you know, yeah. Absolutely, yeah. and really from there we expanded course by course using what we learned in each rollout to really refine you know what we saw next.
Wes Smith (07:03.097)
Yeah, that seems really smart.
Wes Smith (07:13.839)
How long does it take you to embed within a single course? Like is that like a a quarter long process or or are you being able to rapidly scale that at this point?
napereira (07:24.963)
We’re we’re able to rapidly scale that at this point. Initially we took our time because the platform was new to us. but once we were able to train you know, internally with our instructional designers and with our faculty, we found that that was actually a rather quick platform to be able to embed this in the courses, right? And and all the technical things in the background as to where how it gets inserted into the course and all that, that’s part of what my technical team does. As far as faculty, they just need to understand how the platform works.
Wes Smith (07:30.554)
Mm-hmm.
napereira (07:54.094)
and really serve as the subject matter experts in in their given course. and we found that that does not necessarily take that long. You know, it could be weeks where you can embed this technology in there and really train the AI is what what faculty are doing.
Wes Smith (08:08.613)
So it sounds like faculty pretty supportive, they’re moving it, they’re advancing it. Tell us a little bit about how students are responding.
napereira (08:15.905)
Yeah, so students have responded really well to this and we’ve seen we do take surveys you know after each session to kind of see where students are at, how they feel about the technology, and it was overwhelmingly positive. One of the one of the things that I thought was an important indicator was kind of what I talked about before. Like d do students have that a better connection now because they have these lectures with the AI and the AI is is kind of picking up on on
you know, what they need help on and that the AI is there twenty four seven for them. and all of our surveys showed a kind of positive reaction to that to that connection and the explanation of course material inside of of the courses through the use of Chiron technology.
Wes Smith (09:01.581)
Nice. So I mean, we are living in an age where we’ve been talking about personalized education for a long, long time and the time and resources for one individual human to, you know, to provide individualized care for each student is it’s almost pro it’s prohibitive in almost every instance. But with AI, we can expand, we can we can see students get that personalized education at scale.
napereira (09:07.798)
Right.
Wes Smith (09:29.207)
And you’re you’re watching it right now, and what I’m hearing is students are reacting positively to that. Is that
napereira (09:35.829)
Yeah, absolutely. And we have fantastic faculty at at Miami Dade College. like you said, we’ve been talking about personalized experiences and learning for a long time, but sometimes it’s it’s kind of f it’s physically difficult to be able to do that you know, at scale with so many students where we see the AI as really as another tool for our faculty. You know, faculty are able to see those misconceptions in real time. They’re able to understand, better understand what specific students
are having issues with at a specific time. and the AI can kind of remember that across you know across the the course of of of that class that they’re taking and really keep probing the student to figure out ever if there are any other misconceptions.
Wes Smith (10:19.865)
Yeah, I love it is as a tool for faculty to be able to personalize their efforts more to each individual. So yeah, I mean it’s a win, it’s a win for faculty, it’s a win for students. And I know that a lot of education leaders that aren’t at Miami Dade and and they’re at different institutions out there watching and they’re thinking, okay, yeah, we love this, we’re we’re curious about it.
that they’re in that curiosity stage. They’re they’re interested though in making the leap and actually doing it. You’ve you’ve made the leap. Tell us lessons learned for for any of those education leaders out there watching. What would you recommend as far as taking the next step towards actually adopting and making it a reality for their students?
napereira (11:10.227)
Yeah, I think really the the biggest lesson learned is you have to start with the student problem and not necessarily a specific tool, right? So you kind of have to work your way backwards from from the results that you want to see and then go backwards. And it’s really easy to, I think, to get pulled into what can this technology do or that one or that AI platform or that one, instead of what is actually getting in our students’ way right now from persisting and from s for succeeding.
So really every piece of what we’ve built from Lucy to the Chiron integration really traces back to a specific point of friction that we could name before we ever evaluated a vendor. Right. and then secondly, you know, we really built the infrastructure before you built the rollout. and we trained our instructional designers ahead of faculty, like we discussed before, to just make sure that this new tool was
was easy to understand, it was accessible, and then had that faculty buy-in. I’d also say, you know, a third thing is try and treat it as a phased rollout and not necessarily a huge launch event. you know, we’ve had a a real roadmap, you know, what’s live now, what’s coming next quarter, what’s what’s further out. and that phasing really let us prove the value at each stage.
instead of you know betting everything on one big rollout and also gives us an opportunity to hear feedback in real time from our faculty and students and make adjustments in the platform as as we see you know as we saw necessary. And then finally I’d say bring leadership real data, you know, not just a demo of what it can do. And institutional buy-in I think from you know what was actually happening with students and and not just what the technology could theoretically do, but what
changes and what positive effects did we actually see.
Wes Smith (12:57.401)
Right. I love I love your focus on data and I like, you know, this idea of starting with a student problem that you want to solve as opposed to you know, flashy new technology that you think looks cool. No, let’s let’s let’s figure out the problem that we’re working to solve. So so speaking of that, tell us what you’re seeing about you know, on your persistence issue at Miami Dade. Are you seeing the data?
napereira (13:06.13)
Right.
Wes Smith (13:24.335)
That is encouraging to say, okay, what we’re doing here is advancing our goals and persistence.
napereira (13:31.482)
Yes, very very encouraging data we’re seeing, you know, particularly from our spring semester that just finished you know several weeks ago. We’re seeing students that have actively engaged with the AI where those misconceptions were detected. We’ve seen them have measurably higher pass rates and persistence than students who were kind of on the same level but did not necessarily interact with the AI. It was important for us to kind of see that. we’ve also had some faculty that taught the exact same course for us.
one using the AI model and one not, and then doing a comparison between them. And we did see higher persistence and higher pass rates in the course that had the AI. And then some qualitative data that we received from from students about how they felt. How did you feel about having that help, that assistant, that that guide there for you 24-7? And that was very positive. We also wanted to to have qualitative data as well. And that was very positive from both the students and the faculty. And
You know, one other thing that we that we developed in conjunction with the AI is kind of this AI dashboard that lives in the LMS for faculty to have the data at their fingertips. They don’t have to dig through and try to find out, you know, how are the students interacting with the AI, what has it identified as misconceptions. This is all in one location for faculty and they really appreciated that.
Wes Smith (14:47.065)
Yeah. I love it. I l I love that you’ve got the data to to show you now and that you can now start to refine your systems and improve. That’s that’s a great stage to be at. Okay, so finally, l I I want to pick your brain on one last thing and then we’ll let you go. That is, you know, a a lot of this series is intended to inform policymakers. People in Washington, DC or in the state houses,
Or or you know, those in in education that are building the future system for our students about innovation and technology and specifically about AI. Is there any advice that you would give to policymakers who are building that, you know, next next generation of policy that will facilitate higher education? What do you want them to understand about AI and what AI can do for students?
napereira (15:45.881)
Yeah, it’s a great question. the first thing I I guess I’d want poly policymakers to understand is that AI in education is not primarily about automation, right? It’s really about access. And for generations, you know, we’ve been discussing personalized academic support, and we’ve had that in the past. We’ve had tutors and advisors and we’ve attempted to have real time feedback and all that. that’s been available to students maybe who could afford it or who attended well resourced institutions.
you know, for example, the students at at Miami Dade College, most of whom are again working adults and many of whom are are the first in their families to pursue a degree, they’ve had those access opportunities at at MDC, but what AI lets us do is really scale that access further and extending that personalized support to every student at a scale that really wasn’t possible before.
I I’d also ask, you know, and advise policymakers to consider specifically what do those funding mechanisms look like that reward AI implementation? and tie those to measurable results. again, completion rates, credential attainment, workforce readiness, things like that, and not just adoption metrics, right? you know, yeah, absolutely. And and really
Wes Smith (17:03.311)
Yeah, I love that. Well look let’s let’s tie it to outcomes, not not implementation. Yeah. It’s not it’s not about the the fad of, you know, we adopt it. It’s about it’s really about persistence and completion.
napereira (17:08.927)
Yeah.
napereira (17:18.923)
Yeah, absolutely. And you know, I think that the the the recognition, particularly to colleges and community colleges and and other universities around the country that that enroll the largest student populations in higher ed, we really think that those students that are balancing their lives, their work, their family, school and all that, they really need to be central to this conversation about about AI. because if AI driven innovation doesn’t reach
these institutions and those students that really need it the most. I think we’ve missed the the you know, the students who need it and really the workforce pipeline that the economy depends on.
Wes Smith (17:56.112)
Yeah, I love I love what you’re doing down there at Miami, Dave, Nestor. Thank you so much for sharing your college’s approach and and just your lessons learned. I think it’s really helpful for everybody else that that’s watching and that’s thinking about, hey, we we should probably be moving in the same direction. So this is this is fantastic.
napereira (18:15.317)
Than thank you so much, Wes. I really appreciate you inviting me on.