Okay. Let's get started. Hello, everyone. Welcome to today's webinar, which is called AI in hiring, promise pitfalls and practical solutions. My name is Michelle. I'm your host from Criteria, and we're really excited to dive into an incredibly timely and relevant discussion about the role of AI in the world of hiring and talent acquisition. And just as a quick reminder, we are Criteria. And as many of you know, we are a leader in the talent assessment and development space with science backed tools that help organizations make better talent decisions. And so this webinar will be about an hour. We'll have time for q and a at the end, and everyone will receive a link to the recording after the webinar as well. But without further ado, I'd love to introduce our two incredible speakers. And first up is Chris Dayden, the chief technology officer at Criteria, where he leads the development of cutting edge AI powered hiring solutions. With a background as an entrepreneur and product visionary, Chris has built and scaled transformative technologies used by organizations worldwide. He's also a respected voice in the AI and tech community, having spoken on panels at SHRM and leading tech conferences. Welcome, Chris. Hello. Thanks for having me. And we're also thrilled to be joined by Jillian Phelan, chief people officer at Criteria and a seasoned global HR leader with a track record of guiding companies through rapid growth and transformational change. From venture exits to global expansion, Jillian has led people strategies that keep pace with evolving business and technology landscapes, much like today's AI driven shifts. She's passionate about building inclusive, high performing cultures that scale sustainably even in the most complex environments. Welcome, Jillian. Good morning. Thank you for having me. But with that, let's jump into the first of our panel style questions by setting the stage a bit. So, obviously, we know, that AI and hiring has been getting a lot of traction recently. And so one question I have for both of you is, what are the big biggest trends you're seeing in this space? I think the, the hiring world is experiencing really a massive inflection point right now. I think that intelligence systems that are, of course, fueled by the advancement of AI are no longer just automating repetitive tasks in our HR life cycle, but they're helping us reshape the definition of what qualified means when candidates are applying to a role. And I think that instead of relying on static signals like resumes or job titles or companies they've been at, Organizations are starting to prioritize adaptability. They're prioritizing problem solving and real time potential. I think that's a trend that I I absolutely observe. And these intelligent tools, of course, again, fueled by AI and the advancement we have there, have given us the ability to reduce bias and human noise from the early stages of hiring. And that ultimately means that we have better alignment with the role we're looking to hire for and more fairness at a scale that we haven't seen before. Ultimately, I think we're just seeing a major evolution in talent today. We're, again, moving from simply automating hiring steps to amplifying talent signals, and those, are creating and designing stronger and more resilient teams. I'll echo a bit of what Chris just said, which is AI in HR is unequivocally shifting from automating processes to enabling data driven decision making. And, again, that's a something HR has not always had, so it's quite a unique moment in time. We certainly are seeing more tools like skills intelligence platforms, talent coaches, and predictive insights that can help shape how we hire, how we develop our employees, and how we go about workforce planning. Great. And, question for Jillian specifically. From your perspective as a chief people officer, what do you think is driving the adoption of AI in hiring and recruiting today? AI adoption in HR and recruiting is driven by a combination of a few things, a little bit of pressure, a little bit of promise, but also practicality. When it comes to pressure, there's pressure in two buckets, one on resources and one on the use and adoption of AI. So resources, we're all doing more with less since teams are leaner, budgets are a little bit tighter, and that's combined with rising expectations for speed and execution. So AI is stepping in to automate repetitive tasks like resume screening, candidate scheduling, employee support, and it's helping to free up HR teams to focus on higher impact work. The second point that I mentioned is really, like, promise. There's is a promise out there that AI will enhance decision making capabilities, like the ability to analyze hiring patterns, predict attrition, and tailor learning paths, and they're giving us more data informed ways to support the employees while remaining aligned to the business strategy. But in the end, it's really about is this practical? So organizations are realizing that thoughtful, secure, ethical AI integration is not necessarily a nice to have anymore. It's becoming a competitive necessity. It's here to stay, and it's changing how we go about attracting, supporting, and growing our talent. Great. Chris, a question for you. From a technology standpoint, why do you think now is the right time for AI to make a meaningful impact in the hiring process? I think Julian's highlight of the pressure that we're all under as organizations to deliver at this new level of efficiency is a really, really good one. I think there's a convergence between the technical maturity that we've had from these, AI advancements like large language models. You know, those are the technologies that power tools like ChatGPT that we all have learned to love and the pressure and urgency. So the convergence of those two things means that, on the tech side, we now have intelligent systems that are even more capable of evaluating complex inputs. I would say for the better part of the last decade, our customers have asked for predictive signals in some categories of hiring that we have simply chosen not to release because we weren't confident enough in the science and signal the technology could provide in those specific signal areas. But as technology catches up, that's changed now. There's a whole new set of talent signals that have been unlocked that we're now able to measure with the science and rigor that Criteria expects, in our product set. So that convergence is is really unique, and that tech side has afforded us greater opportunity. And, again, to echo what Julian said, on the business side, companies are facing more and more applicants. They're forcing, they're facing more and more pressure to hire, teams, in effective ways, and those higher expectations, really lean into experience and efficiency. And and if you fast forward a year or two, it's really clear that we will need to look at talent differently. Data shows that to build the most successful teams of the future, we must place less emphasis on particular specific hard skills or maybe years of experience that we're conditioned to hire on from the many, many years of hiring we've all had and focus more on traits like aptitude, learning agility, cultural contribution. And, ultimately, some of those credentials, like years of experience or college degrees, are more static. And I think what we've been observing and what we believe is that potential is the better, more dynamic credential that we we, like, have, you know, created systems to measure. And that new lens is how we can use responsible AI systems to allow us to to apply that at scale. And lastly, I just say, like I like to say that we're we've moved from a world that asks candidates, what have you done before, to one that asks, what can you become next? And I think, as we evolve our hiring processes in this, stage of evolution, we're we're gonna be asking that question more often, and we're gonna be looking for scientific ways to measure those signals that give us insight into that dynamic of a candidate. Yeah. I love the way you phrase that. The what can you become next is so critical as we move into the future, and we see new skills developing. So hundred percent agree with you there. And one thing we wanna know from all of you, in our audience is, we'd love for you to add it in the chat. But for those of you who are already using AI in your hiring process, let us know in the chat some of the ways you're using it. We're really curious. We'll give you all a moment to formulate some responses. But I know internally at Criteria, we've got a number of great uses for AI that we use across all different departments. So we're just interested in how others are using it. We'll give you just a moment. But in the meantime, we can jump to, the next topic, which we wanna talk about ethics. So question for both of you. There's a lot of talk about ethical AI. But what does that really mean in the context of hiring? I'll jump in. I think Yeah. Go ahead. Please. Thank you. Ethical use is gonna require well defined guardrails. So active human oversight and transparent communication with candidates about how and when AI is used to make hiring decisions. Of course, ethical use will also be embedded. It must be embedded in the integrity of the model. And then then I'm gonna hand it to you, Chris, because Chris knows all about models. I think that's great. I think ethical AI is a term that is thrown around a lot in these early, you know, this early transition of technology. I think everyone's gonna have a page on their site that says responsible AI or ethical AI. It's just gonna be part of the marketing engine. But what it really means under the surface is teams and us as leaders in talent acquisition and just, you know, ethical company, you know, operation. It means asking if our intelligence systems are fair. Are they transparent? Are they compliant? Are they helping us make better human decisions, not only just faster ones? I think that's really critical. They have to be better quality human decisions that we are making from all of these inputs from these intelligent systems. And I believe that ethical hiring technology really needs to elevate candidate strengths. It needs to minimize bias. It needs to ensure that every individual along the process gets an equal opportunity to showcase their potential. And, ultimately, ethics also implies and means accountability. So who audits the system? Who checks the system for bias? How do we ensure transparency when those checks happen? These are now really essential parts of any implementation road map, especially when you're building systems that are in a process like talent acquisition, which is a protected area. Our decisions need to be based in science. They need to be predictive. They need to be fair. That's a a standard we all need to hold ourselves to. So, what I like to say is similar to how, conventional custom software or software is made, we have what's called a software development life cycle. And, with creating these AI systems, there is a clear ethical AI development life cycle that starts all the way at the beginning of when you're thinking about what problem you're gonna solve with AI. And there's a process that goes all the way through a very complex cross functional stage of of r and d. And, ultimately, no matter what solution you're considering, these these are all the questions you should be asking a vendor that you're looking to work with and and ultimately decide, they taking the phrase ethical or responsible AI seriously? So in the context of hiring, that's what we really look for. And, of course, those are all embedded into our product development life cycle. Great. And question for Jillian. What hesitations do you hear or experience from HR leaders around using an AI, and how do you think they can be addressed? I will bucket them ultimately into three categories. One is ethics, which we've spoken about, but I'll get into it a little more, about accuracy and then ultimately about HR department job security. And let me address that. So first and foremost, obviously, the ethics is is especially around bias and fairness is imperative that that's kept in the forefront. HR leaders often I'm witnessing they're asking, I don't understand how AI is making these decisions. How can I ensure that it's fair? And while it's perhaps the right question to ask, AI systems are only as good as the data and assumptions that they're built on. And if we are not intentional, we risk automating bias at scale. Not every organization has an internal industrial, an organizational psychology team like we do, and I get that. So to address this, I recommend from our HR leaders to demand, as Chris just mentioned, transparency from our vendors and leaning into the tough questions about how the models are trained, ensuring that the teams then that are leveraging those models understand the limitations, and ultimately, understanding that AI should be designed to support human decision making and not replace it. When it comes to accuracy, especially in high stakes decisions like hiring and employee performance. Again, the key is making sure that we remain human focused, human staying at the center of that, and AI is informing but not dictating. And then that interesting fear about AI replacing jobs, replacing human resources, is there's some truth to it, but it's also what we're witnessing is it's actually freeing teams from repetitive administrative work and then allowing the teams, especially ours, to focus on strategic human centered initiatives, things like culture, leadership, and and development. And the shift perhaps is away from, like, oh, is AI gonna take my job to how can I use AI to elevate and uniquely position the human work that only I'm able to do? Great. I love that perspective. And question for Chris. How does how does Criteria specifically ensure that the AI we build is responsible and reduces bias and is compliant? I know you've already touched on this a little bit, but anything you'd like to add there? Yeah. Obviously, very, very important. So at Criteria, we build, as Julian mentioned, with psychometric rigor. There's a lot of science that goes into our, the our model creation. And there's human oversight completely from the start. Our models are validated to ensure that the predictive accuracy and fairness, occurs across diverse populations. I think what's really important, particularly in the world of generative AI, is it may work in a subset of a population or it may look like it works in, a small set of data, but then extrapolated across a larger, more statistically significant dataset, that is not the case. So it's very important, and we audit for disparate impact. We also give customers transparency into how decisions are made along the way. A method that we like to use in our AI strategy more generically is including humans in the loop, and that's particularly important because of the high stakes environments we all operate in, like hiring. We wanna make sure that we're leveraging the best capabilities of our human resources alongside the augmentation and efficiency that comes with, advancements of agentic AI. And, again, for us, compliance isn't an afterthought. I think if you are a product company like Criteria, you must build into every stage of your design considerations for global regulations like the EU AI Act. You know, we pay a lot of attention to, like, the New York City local law one forty four. We, you know, of course, comply with ISO twenty seven thousand one. And even the newer standard we're looking to certify against is the ISO four thousand one, which is about AI creation and model compliance, which, of course, is is the edge of of r and d for us right now. And, again, we're looking to proactively comply with those so it doesn't later slow us or our customers down, and there's a a trust dynamic established from day one. And the last thing I'll say is what I love about our product and technology strategy at Criteria is we really aren't scared to throw something out. As painful as it is, if we've worked even if we've worked really, really hard on it and teams have put hundreds and hundreds of hours of r and d into it, one thing that's core to our culture is that honesty and science prevails. We we make sure that even if we're attached to a method or we really want a talent signal to work, we are not scared to, you know, scrap the whole project because it doesn't meet our rigorous standards, because the last thing we would want is something out in the wild that's not fairly or, you know, accurately predicting talent success. And this is ultimately where companies who may have less experience in the hiring space, building technology like this, or maybe companies that don't have the depth and breadth of scientific signal that Criteria has in our portfolio, this is where those companies will stumble. So as Jillian mentioned a moment ago, building AI on top of poor quality signals isn't responsible, and it will fail. Great stuff. Yeah. Thank you for that context, Chris. And, before I move to the next topic, I did wanna just circle back to some of these chat comments that we saw, from that original question of how y'all are using AI in hiring. It looks like some themes that have emerged are definitely using it for job descriptions. I think that's a great use of, some of the it's a pretty easy use with the ChatGPT and all the great tools out there. I'm also seeing, a lot of people using it for, coming up with interview questions, which I think is awesome as well. Chris, Julian, anything you wanna comment on some of those chat responses? I think it's great. Yeah. Yeah. We're leveraging both as as well. We do a lot of, ensuring that our job descriptions don't have bias. And so asking, questions, really digging into how to effectively ask the prompt so that we are getting the results that bring the lens of reducing risk and bias to job descriptions, interview questions, you name it. So we're right there with you. Yeah. And I I really applaud the the usage of those those use cases, with Gen AI. I think for us, when it comes to our product strategy, we see that a lot in the market. We hear testimony from our customers that says, I'm using ChatGPT to summarize an interview or, you know, evaluate and create questions for an interview. And I think while that's great and those are the early steps of how we should leverage tools and get experience with it, you know, our mission at Criteria is really to bring rigor and science and customize those large language models and AI tools for the hiring use case so that it's defendable, protected. So, what I love to see is kind of the ambition of our customers and even those who aren't working with criteria using tools like that to get those outcomes. And I think that's where we start in our exploration process. We see those things, and then we make them more scientific within our suite of capabilities. Really cool. Great. So let's pivot to the next topic, which is about humans. So question for both of you. How do we strike the right balance between automation and human judgment in hiring? Great. Jillian touched on this in a couple different ways, and I really admired, what she said about it. I think, ultimately, these intelligence systems should guide human judgment and not replace them. I think that the best systems will create structure that elevates our human judgment rather than override it. I'll give you an example. Structured interviews, for instance, if you combine data driven recommendations, and you're combining recommended, you know, direct competency based questions to be asked during an interview and you're applying that consistently, you're giving hiring managers superpowers. You really are. And, ultimately, people will still make the final call, but the final call will now be more informed. It'll be less biased, and it'll be a lot more confident. So, ultimately, we continue to design to design our our interfaces and our products to emphasize the combination of, these intelligent systems and the signal and data driven recommendations they can provide, but ultimately empower, what's you know, the one of the best qualities about us as human resources is our human judgment. So our products will always emphasize that. And my response is echoing Chris's a bit, which is the automation should just inform but not make. You've said that. It AI is so delightful in that for human resources, it's it's giving us faster and more data rich insights, which is so exciting to be at this precipice. But we also have to be cognizant that employees and our candidates aren't just data points. So meaningful hiring and meaningful use of HR of AI, pardon me, in HR is still going to hinge on conversations and inclusive practices that our machines can't replace. Great. And, Jillian, another question for you. How have your own teams experienced or responded to AI powered hiring tools? It's a good one. Our team was a little curious and a little cautious. So at first, there were some understandable skepticism, especially from talent acquisition and our hiring managers who take pride in their instincts for that human connection. And a lot of the conversations we were part of was a little bit of fear around AI taking this process of hiring and or performance management and learning management and making it cold or transactional or even overriding judgment. But once we've really got our understanding of how we are leveraging AI, especially internally with our tools, our attitude shifted pretty quickly. So I I know the AI functionality of, like, screening resumes for qualifications or servicing candidates that may have been overlooked, in the early stage of sourcing have been around for a little bit. But AI tools to help surface talent that maybe we didn't have a line of sight on or focus our attention on candidates and, their skill sets of what they could become has been a game changer for the way we're going about, talent acquisition currently within our own organization. Our team also has been really intentional and relying on AI as just data points that give us a broader understanding of the individual that we're assessing or having conversations with about a job or performance or career pathing and career growth. And so we're keeping human judgment, I've said this before, at the center. And our adoption of leveraging h of AI, pardon me, has been part of bringing our team along, making sure that there's clear understanding, and then ultimately measuring its impact. And once we're seeing, like, efficiency gains, better hiring, we it's not even a threat. The adoption's been extremely enthusiastic and, quick for our team. Great. I love that. And I think seeing that impact, that immediate impact, it really seals the deal. Chris, question for you. One of the biggest challenges in hiring today is obviously high volume hiring and the sheer volume of applicants applying to any given role. What role can AI play in identifying and highlighting the best candidates? I I feel like every role today, seems like high volume hiring. Yeah. Whether you're in the hiring in the United States or, you know, I hire a lot internationally as well in regions like India, and I've been doing it for for a long time. And I feel like, really, just the amount, I think the data supports that the amount of, applicants per available role is continuing to increase. I'll also say that AI plays a pretty direct role in that. There are, unfortunately, you know, applications out there where you can fill out your information and just tell the AI to go and apply for jobs for you on your behalf kinda while you're sleeping. So, you know, everyone at their fingertips has this AI powered automation that can expose them to more roles and automatically apply. I think we've all seen the phenomenon that if you take a job description in your resume, you can ask ChatGPT or any AI model to give you the perfect resume. And, unfortunately, what happens even there is that the model will hallucinate and put competencies or capabilities or experience on there that aren't even real, but makes you look like a rock star. So if I have an application pool of five hundred candidates, maybe four hundred of them look perfect because they've gone through this kind of AI augmentation of their resume. And, really, that has made in the volume problem of high volume hiring, it has made resumes nearly unusable. And our new methods and capabilities, thanks to the AI advancement and the the plethora of signals we can collect through structured interviewing or, you know, something like the, criteria language proficiency tests, can surface underlying competencies that can't be faked by artificial intelligence or tools that you can purchase for nineteen dollars. And we don't, of course, focus on resume keywords or a signal that would come from what is now, nearly an unusable signal in in the world of hiring. So, I think, you know, this webinar is about AI, of course, in hiring, but a quick reminder that there are many valid traditional signals that need to accompany an AI signal for the best, most well rounded compounding talent signal is what we call it. So right when you meet the candidate, you get a small signal from, top of funnel assessment, then you get a stronger signal from a a video interview as part of a structured interviewing effort. And then, you know, you can continue down that life cycle, and you're getting this compounding talent signal that gives you even more data to make the best decisions for your team. So, ultimately, it's about highlighting candidates who have nontraditional paths, candidates who show strong cognitive skills, learning agility, and, of course, role relevant strengths that aren't easily seen on paper or maybe are over magnified, thanks to, you know, our AI doctored resumes, that people are applying with. So, ultimately, it's not about shortcuts. It's really about precision, and we're trying to find the best possible match, not just the most polished profile. Great. So interesting. And we'll get into a little more about how criteria helps with this, in a few minutes. But, the next topic I wanted to pivot to was a little bit about the future and the future of AI and hiring. So, question for Chris. Where do you see AI heading next in the talent space, and what technologies are on the horizon? I definitely have a kind of one year, three year, five year perspective. I think for the value of today's webinar, I I'd say we're entering a new era where these intelligent AI tools are now becoming proactive collaborators in the process of hiring. They're not just reactive automators. Again, we're transitioning from just automating the workflow. Like, how do I move candidates from part a of my pipeline to part b? Like, it's not just about simple workflow and automation now. It's about doing the job better, getting better talent signals, making the best hiring decisions. And these AI tools are going to continue to become proactive collaborators in that process, not only just tools that automate something. So those systems are gonna help hiring managers plan. It's gonna help them decide, and it's gonna help them grow talent with far greater precision. They'll also help us contextualize needs in our team by analyzing things like Amex. They'll help us see gaps in capabilities and surface the right, you know, area for development so we can build the best teams. Again, these Copilot kind of style systems are gonna guide hiring managers. They're gonna give things like timely nudges. They're gonna kind of identify when bias is likely to surface. Maybe, the interview structure or the interview loops for a particular company lacks structure, or maybe candidate feedback is deviating from rubric based scoring and it makes that, you know, outcome not as predictive, we're gonna be able to jump on those with these copilot style systems. And and I would say, ultimately, for Criteria, we focus and we're, you know, really involved in the evolution in hiring, but we have a position that it also goes beyond hiring. You know, we're building toward also an era of post hire intelligence where, we can bring, you know, prehire assessments, real time interview behavior, and onboarding interactions. And all of this data we collect, and we're creating this dynamic talent blueprint. And, really, that talent blueprint provides a living real time map of an organization's potential. It can showcase workforce skills. It can show growth trajectories. It can highlight development needs, and it ultimately replaces kind of what we're used to as, like, a static organizational chart with this adaptive insight driven perspective. So that's where I think we're headed with this technology. Great. That's super exciting. I can't wait to see what the future holds for us. Next question for Jillian. This is an interesting one that probably a lot of people will relate to on the call. But what role do you see HR playing in the responsible rollout of future AI tools? In my humble opinion, we play a pivotal role in responsible rollout, not just as a user, but as a strategic partner in driving the value of leveraging AI while also managing risk. So as we are the function the sole function in the organization that is closest to candidates, every employee, and in our case, across the globe, compliance, and then culture. So we're uniquely positioned to evaluate AI tools, their impact on people, processes, and performance. And I think our role is to ensure that any AI solution we bring is in alignment with our business objectives, obviously, but also legally compliant and ethically sound. And so in partnering with both legal, with our enterprise IT teams, we must assess, like, data privacy, bias risk, and then ultimately explainability so that we are doing all of this upfront before any use or rollout. We also are the ones to help lead change management. We need to equip our managers and employees with context, training, and clarity around the use of the tools. At the end of the day, AI, I think Chris said this, it's a it's a force multiplier, but only if it's implemented with clear governance, it's aligned with the business needs, and HR's job is to make sure that implementation enhances decision making. It helps improve efficiency and processes, and it drives measurable outcomes without compromising what are the core values of many organization, trust, transparency, accountability. Great. Question for you both. If you could give one piece of advice to teams just starting to explore AI and hiring, what would that be? Jillian, kick us off. Okay. Be clear on what problem you're solving. So is it like time to fill or reduction of bias? Candidate experience. Let your business needs, not the buzz around AI, guide your adoption strategy. Choose your vendors who can explain how their models work and what data they're trained on and how to mitigate, how bias has been mitigated in that model. And then adoption really it falls apart oftentimes when HR is the last to know or we're blindsided by it. So involve your team in the testing, provide the training, the cocreation of the guardrails so that your team members are confident and empowered and are taken along in your journey. I echo that completely. Thank you, Jillian. Sure. Great. Yeah. Beautiful answer. Now I wanna turn to a quick poll. So let me launch this poll for you all. But we wanna know from you all on the call, how optimistic do you feel about how AI can impact hiring? So we will give you all a second to fill that out, and then we'll share the results and talk about them together. I see some responses flowing in nice and quick. We'll give you another few seconds, and we'll close the poll in three, two, one. So here are the results. So we've got, it looks like a solid mix of very optimistic, somewhat optimistic, and a few people not very optimistic. Any any commentary from either of you? I think it's This Yeah. Yeah. It's pretty like, if I were to if I, you know, ran this poll twelve months ago, I think the distribution would be very different and that different in the way of there's quite a lot of folks that maybe were less familiar with the capabilities, the dynamic, the how, and I would have seen a lot more not very optimistic. The fact that we see zero not at all optimistic is definitely a transition as well. So this is aligned with what I'm seeing as sentiment from customers and prospects, and I'm really happy to to see that we've gotten over that that hurdle, and and we're now looking for the best way to adopt it and moved past kind of, initial questioning as to whether or not this will impact hiring. It's here, and now we're all doing our best to apply it. I really like that that change in sentiment. Great. And I'll echo what Chris said. Yeah. It it that distribution may have been the inverse had we pulled twelve months ago, in my humble opinion. Yeah. Great. I'll stop sharing that. And then the next question, we're almost to the q and a part of the session. But before we get there, I do wanna talk briefly about the application of AI. And as an example, Criteria recently launched a new AI product called interview intelligence, which is an AI powered interview scoring solution that can score video interviews as active, as accurately as an expert human grader. So, Chris, I was wondering if you could walk us briefly through how interview intelligence was developed and what problem we were trying to solve. Awesome. So I think we could all relate that, for the most part, interviewing can be one of the least structured and most biased prone parts of the hiring process. The reality is managers often go in underprepared. They focus on irrelevant signals just based on our human nature. A lot of them take unstructured notes. And, frankly, like, in my experience, particularly to engineering, I think the world assumes that the highest performing, most competent engineer is automatically a great hiring manager. And I think for those of us who have done recruitment or hiring in technology, that is not always the case. So I think there's just some structure that makes that, you know, that problem, is what we were setting out to solve. So, interview intelligence was developed to change that. We are looking to bring structure, science, and scale to the interview process. What's been amazing is our organizational psychology team, essentially, as expert as you can get when it comes to interviewers. I mean, our team went through structured supplemental trainings. They went through a calibration towards each other, and, you know, they're they're kind of as expert interviewers as as we could get. They then proceeded to score thousands of real interviews to create a proprietary dataset for criteria. And this really excellent high quality dataset became the training foundation for what was ultimately a fine tuned large language model. And, again, a large language model is the same kind of technology behind the tools like ChatGPT that we've all, you know, come to love. And I think the biggest difference is that this fine tuned large language model that is unique to Criteria is built specifically for the purposes of hiring, and that's really, really important. We're really proud to be the first company to bring a fine tuned large language model to market for the purpose of reproducing expert human interviewing and predictiveness at scale. So this purpose built model allows us to automatically evaluate asynchronous interviews with this high level of rigor, consistency, and fairness that we'd expect at Criteria. Just imagine getting your own IO psychologist on your team that, you know, can work twenty four seven for you and score thousands and thousands of candidates in your pipeline, and you have, you know, real strong confidence in in that consistent and fair signal. So it scores candidate video responses based on structured rubrics that our software helps you to author and create. So just like a great interviewer word or just like a expert IO psychologist would do if you were working with them to set up your interview process, our platform does that with a few clicks of of the buttons. So, really, this approach gives us all the efficiency you'd need, all the predictive accuracy you'd need, and that expert level of consistency to bring that in a repeatable and scalable way to realize your organizational goals. We've also got intelligent transcription and summary capabilities. And, again, it's a whole life cycle. So when you're starting to even think about the questions you wanna come up with for your structured interview, you know, we have a a subset of smaller, highly capable models that guide you through that process. And it'll actually tell you, oh, that's not a great question. Try this question. Or, yes, that question is, you know, relevant to competency for the role you're hiring, and and we think that's a great question for you to, you know, evaluate objectively against when the candidate responds. So really, really exciting, really, really groundbreaking. It's really the edge of innovation right now, and, we're just so lucky to be leading the way. And, I know our team is putting a lot of hard work into this, so all of us can benefit in the world of hiring. Great. And question for you both. I know, Chris, you touched on this a little bit already, but what makes interview intelligence different from other AI based hiring tools out there? I'll take this one. As Chris mentioned, it's a wild moment in time to realize that resumes may become increasingly irrelevant, which is for someone who's been in talent acquisition for years, game changing. Having said that, we're also there's just a New York Times article out a few weeks ago articulating that LinkedIn's applicant flow is up forty five percent with eleven thousand applicants a day, and we see it with our open roles. We've gone from five hundred candidates to a thousand in two to three days, with job postings across the board of all jobs. And, therefore, without another signal, we would get mired in all of the data coming our way. It's really hard to sort through that. So our platform from assessments, and I'll get to interview intelligence in a moment, to video interviewing, to structured interviewing, ultimately is enabling us from the HR talent acquisition side to take that human and then add additional data points that unfold. How this person thinks, how and who they will become other than the opportunities afforded to them in the past with that resume. One data point we noted, and it was true internally, is that the sheer volume then of candidates that are coming through our pipeline was resulting in hundreds of video interviews. And video interviews prior to our transcriptions and prior to the automated scoring could bury one in data. It takes five to six minutes to go through and score the video interviews to get individually. We also noticed we couldn't always get through all of them, and what our product does thoughtfully based on the work of our industrial psychology team building that model, it enables us to say, oh, these top fifteen, twenty have a signal that perhaps is something we should look into. Let's focus on these candidates, and then continue to dig into the other applicants. But we'll it shows us where to focus first, and it gives a bit of, discipline within all the chaos and noise that seems to be happening within many organizations because of the volume of candidates. Great. Great. I I see a couple of people, were interested in seeing a demo of interview intelligence. We love to hear that. We don't have a demo set up for you all today in this webinar, but we'd be super happy to reach back out to you and circle back on that. But I think we are actually ready for audience q and a. So you can feel free to add, q and a questions to the chat. You can add questions to the q and a box, whatever wherever works best for you. I think there was a question a little earlier in the chat about what other AI based tools Criteria offers. So I don't know. Chris, if you'd like to share, a little more detail about some of the other AI tools that Criteria has. Yeah. Happily. So our philosophy, ultimately, at Criteria is how do we get the most predictive talent signal out of any part of the process in hiring. So I think the answer is we've been looking to apply AI across all of our products in our suite where appropriate. I think it's, again, a bit of hype to just not think about the outcome you want and just start with AI because it's, you know, exciting and fun. You know, we even have to limit ourselves on that occasionally. Right? Like everybody does. So just an example in our assessments portfolio is, you know, for the better part of ten years, our customers have been asking for a language proficiency, assessment. And, you know, because of the the structure of or I should say the lack of structure in written response text, it's been a traditionally difficult problem to solve with, standard machine learning or or AI techniques. But one thing that GenAI and large language models are exceptional at is making sense of a lot of unstructured text. So, our CLPT, our criteria language proficiency test, is kind of like a smaller large language model, if you will, that is tuned for scoring those responses in a predictive way. So that's one example of, you know, us not having a a way to measure language efficiency signals, and now we have one. You can apply that to how we're automatically scoring video interviews now. You're gonna you can apply that to how in the future we'll be providing live interview signals, whether it be to help the hiring manager do a better job in the live interview or, even providing some type of automated signal potentially from those interviews. So, really, for us, it's about collecting the best, most predictive signals, and we're willing to do that across our entire product suite, across the entire life cycle of of hiring, and, of course, doing that in an always responsible way. Great. We've got another question here. One person is asking, one concern I have had is AI's unreliability. I've tried using AI to score resumes, but its results are not always accurate. How do we mitigate this? I love that. I'll maybe talk to to the tech side, and then, Julian, anything you'd like to add, I'd love to hear it. So there there are really scientific ways to get an AI model to perform more predictively. Like, it's a whole deep discipline of research. And I think this is why I would respectfully caution HR practitioners or just everyone in general from, you know, taking, like, a transcript and putting into the chat GPT and saying, what do you think about this candidate? Right? That's not a very scientific approach. There aren't guardrails built into it. It's actually gonna give you more of a statistically, you know, happy answer than it is going to give you something, robust and scientific as far as a hiring signal. So that's kind of like a don't do that. You know, look for tools like what we're making where we can build in those guardrails in science. Also, like, if if your comment about resumes is, that you're, you know, essentially analyzing keywords and you're not getting a quality talent signal from it, I think that would be a, an echo of the sentiment I shared earlier where we think that, you know, resumes have been increasingly less effective of a signal. Even back when, you know, keyword stuffing, if you've heard that phrase, started happening, where as soon as applicant tracking systems began analyzing keywords, people were using, like, white font at the bottom of their resume in really tiny text to, like, add a bunch of keywords to boost their likelihood of being picked up by the applicant tracking system. We're seeing a similar phenomenon to that now in, in AI and resume tuning for a job description. So maybe your your comment there is a reflection of of that reality that that we've definitely observed in the market. Great. And we have a question here in the chat. Is there a plan for an AI tool that can do the first round of video interviews on its own? And this person clarified further. What I mean is having an AI interviewer complete the first round of interviews and narrow down the candidates before involving a human interviewer. Yeah. That's really cool. We have I gotta admit, our product team is amazing. We have so many different slices and variations of of this idea, you know, rolling around on our road map. And, ultimately, we're very client led. We'd love to hear, know, what our prospects and customers really want. We wanna make sure we're building tools that are solving real hiring problems. One of the way we think about it, and this isn't a functionality that's available right now, but we we think will be coming soon, is being able to configure, the involvement of AI scoring quite flexibly. So let's say maybe you want in your third stage of interviews, you would like to have the AI score kind of as a panelist alongside humans. That's great. We're working on that now. Maybe you would like, you know, the first round of specific questions to be all AI scored. And when you advance them to the next round, you would like humans to, you know, be panelists and and evaluate that. That's that's definitely an option. And then in the future, maybe we have, like, two or three potentially different, you know, variation of of language models that are tuned a slightly different way, almost act as multiple participants, like you would have two or three humans on a panel, and then maybe you'd augment that with additional human talent in the the process. So it's a really great question. We're very customer led. We'd love to hear what you guys think. You know, please drop it in the chat if you have a strong opinion. Great. And I think at the very least, interview intelligence can do that first cut of highlighting. You know, as that stands now, our interview intelligence along with our video interviewing tool can do that first cut of candidates submitting asynchronous video recordings, and the interview intelligence tool will just automatically go through and prioritize your candidates. That's right. Great. We've got another interesting question about interview intelligence. Chris, maybe you can comment on this. Does Criteria interview intelligence use keywords? Good question. No. Not explicitly. So one of the marvels of a fine tuned GenAI model is that we love how it can take broader context and kind of the compounding science of a candidate's, like, sentence. Right? So under the hood, we're we're transcribing what the candidate is saying. We're not using any type of video input or something that may lead to extraneous bias. We're taking the transcripts, and we're able to look at the full structure of what the candidate is saying in the full context. The equivalent of, like, verbal keyword stuffing in a in a video interview would not be a successful approach to to gain the scoring in any way. So we do not rely on keyword analysis. We do rely on general language sentiment and and context. And all of that is evaluated against, a BARS guide, which is a behaviorally anchored rating scale. So that's kind of the rubric that underpins the science that the AI model is using to score that interview in an objective and consistent way. Great. Great question. Yeah. Great question. Chris, would it be fair to say that the the age of keywords was sort of the last generation of of scoring, and now we're in the more, advanced section of natural language processing? I think that's totally true, and I think that's broadly applicable even beyond the world of hiring. I think that's true about search engine optimization and all of the other disciplines that involve text and language. So I think that's absolutely true. And part of this advent of technology is what makes Criteria comfortable to get in the realm of things like scoring automated interviews, or or automatically scoring interviews because the technology that has been around the last, you know, five to eight years preceding this era would have heavily, you know, indexed on things like keywords or less scientific signals. And and that's why we're really ready to lead the way here because we can take the decades of experience and data and science criteria has from our assessments and beyond and apply that, to the most predictive talent signals that are gonna help us all build the best teams. Great. I have another question here. As a nontechnical person, how do you determine if an aide AI tool is ethical or trustworthy? Oh, Jillian, do you wanna talk about how you might do that? Yes. An AI mechanism is only as good as the data it was built on. So you, when working with a vendor, must ask and understand how was this agent built. For example, and Chris alluded to this, but the discipline and rigor that our team, industrial psychology team, had to go through to individually and separate from each other rate over five thousand interviews, leveraging the BARS guide took time. It took discipline. As you heard Chris say, they even had to reset themselves and recalibrate. That is the rigor you want in an AI model that is used. Said differently, I have been a bit more involved in interviewing recently, and I've had numerous experiences where I've spoken with individuals from HR tech organizations who have articulated, oh, yeah. We use AI to vet resumes. And what I have found fascinating and perhaps occurring more than we'd like it to happen. Is that deeper question, great. How was that model built? What how did you create that scoring mechanism? What happened underneath that? What is that mechanism? It falls short, and that that would worry me. I said earlier, it you must implore as as an HR practitioner the vetting of your vendor to understand how was this trained. Otherwise, god forbid, you end up with a model that's echoing bias or hiring signals that you can't defend down the road. So it must come with transparency. Great. I know there are a few more questions in the chat. Unfortunately, we are at the top of the hour. We'll we'll still go through all the chat and the questions, and we'll try to reach out to, anyone who might have outstanding questions. But out of respect for time and the incredible discussion today, we're gonna we're gonna close out for the day. But, Jillian, Chris, any last thoughts you wanna share with our audience before we close out? Good luck in your next chapter in this new era of hiring. Thank you for listening today. And thank you for attending. Do the demo. It's gonna blow your mind. It's very, very cool. Great. Thank you all for attending, and have a great rest of your day.