Hi, I'm [music] Janette Sutton. >> And I'm Eddie Berto. >> And we welcome you to another episode of the Alerting Authority. And as always, we encourage [music] you to subscribe, follow, listen, and most importantly, participate in these podcasts. We want your questions, concerns, ideas, problems, pain points, and your success stories so we as alerting authorities can better do our jobs and make our community safer. And this episode is sponsored again by the alerting authority.

Um, it's the program and training that we go out and provide. And we do that because, and the quotations, we'll figure it out when something happens is not an emergency communication strategy. Um, so what we do is we help organizations strengthen their alerting programs through practical training, governance, policy reviews, message design, exercises, real world guidance that prepares people to make good decisions under pressure.

When every second counts, confidence should come from competence, not cross fingers. And I'm throwing this in because Janette, we I tend to do this anyway, folks. You know, but sometimes we talk to people who are like, "No, my messages are fine. We don't want a message review or we don't need help with design. Oh my goodness. I don't Any thoughts? >> Oh, I have so many thoughts, but people can read about that in the war room.

[snorts] >> We we still need people need help. And it's okay to have a message review done. And it's not a a gotcha moment. It's really an a chance to um learn and build on foundations um based in research. So that's my invitation to you. Yeah. And I agree wholeheartedly with getting messages reviewed and I am grateful that we've had the opportunity to work with several organizations already to help them to improve their messaging.

Um, and we'll continue to share more about that as we continue to do this across the country. Um, but for now, I want to introduce our speaker for the day. um or our our guest our the person who has joined the podcast. [laughter] We have joining us today Dr. Jessica Jensen. She is a senior policy researcher at Rand and a national expert at the crossroads of disaster policy practice and research. For over two decades, she's been helping emergency management as an emerging profession and distributed function rethink how we mitigate, prepare for, respond to, and recover from crisis, especially as disasters become more frequent, complex, and far-reaching.

Before joining Rand in 2023, Jessica spent 18 years shaping the academic field of emergency management at North Dakota State University. She was the department head and endowed full professor. She holds a PhD and a master's of science in emergency management which is a rarity in the field. And today Jessica is joining us because she is the first author on a report that was just issued called the aid report, AID, which is an acronym.

It stands for AI for disasters and emergencies. And this study was released um about two weeks ago, so um early August. And it's gotten a lot of conversation, a lot of attention among practitioners and people who are AI curious and AI watchers really trying to understand like what has Rand learned about AI and about AI for disasters and emergencies. And um there are some things that are included that we would expect and there are some things that aren't included that become recommendations for future research and practice.

Um but we're going to talk with Jessica today and learn all about it. So we're happy to have her today. So Dr. Jessica Jensen, welcome to our show. >> Thank you so much. And just Jessica, please. Whenever someone says Dr. Jensen, I'm like where where is she? [laughter] >> Well, that's fine. I'm outnumbered again. It's two doctors to not not a doctor. >> Soon soon, Eddie, we know. >> I'm I'm hoping hoping in the next, you know, within two years. >> Yes.

Yes. And someday we'll have a podcast just covering what you did in your doctorate of leadership studies, >> right? >> Yeah. Folks, we'll see. It's not a promise. [laughter] Well, Janette's men mentored many students before I have as well. We can keep you honest and check in on you if it's helpful. >> No. Hey, honestly, anything. And for anybody who is going through it or is thinking about going through it, it is not easy, but I do think it is worth it. >> And for any of the my professors who are listening who have mentioned that when I talk about it, thank you for your support.

I'm I will turn everything in on time and just we'll we'll keep moving forward little by bit. >> That's what it takes. >> You'll get there. >> That's right. Well, I want to I want to start things off here as again as I as I as you know, we have academics and we have practitioners and those that are mixing both that are listening. And so to start it off, would you mind kind of just starting at the basics and what is the AIDS study and why was that study done and and kind of what are the primary goals? >> Yeah, a little over a year ago, the Marle Foundation began the artificial intelligence for disasters and emergencies initiative. uh they sponsored an effort to try and learn as much as possible about what emergency management's needs are in terms of artificial intelligence technology.

Where is it at? Um what does it aspire to for the future where it comes to a AI enabled technologies? They asked Aspen Digital, a partner to the initiative, to do that research by hosting multiple different convenings with different um different people within the emergency management community, which the initiative defines as traditional emergency managers working in emergency management offices in state and local government, but also those in nonprofit settings and faith-based organizations who serve emergency management and other related fields.

So, they did a lot of convenings to learn those things. In parallel, they asked Rand, also a partner to the initiative, to come alongside and to really identify the full breadth of the landscape of available AI technologies to inform emergency management. Again, with that really broad definition of emergency management at work. What the Marco Foundation really wanted to understand is what is the hope for the future for these technologies to really empower communities to save lives, save property in the environment, and facilitate a whole and just recovery more quickly and effectively than we've ever seen before.

And so our concurrent efforts really gave them the information they needed and uh to inform the initiative's next directions. Um, of course, then we also produced a report about our work. Aspen did as well. And so I'm here to talk a little bit about that product landscape, uh, particularly as relates technique to alert warning technologies, which is a subject matter that you all hold near and dear to your heart. >> That's right.

We do. So, I know there are other things that I'm sure we'll bring in because alerts and warnings are related to so many other parts and and while there may not be a singular silo of alerts and warnings in every location that that's all you do, it is tied into many other responsibilities and duties and people wear many hats. Um, and it shouldn't surprise anybody we're talking more about artificial intelligence because it really is impacting everything.

And and I think we'll get into that obviously in in what we're talking about today um which we've done in other um episodes and we're going to do more. So no, I I think that's that's really good. I'm I'm in >> Thanks. I'm glad to be here and I'm glad excited to talk about it. >> So um when I reviewed the report, which is it's it's lengthy. It's got a lot of really good deep information in it. Um the one of the first things that I noticed is that it's focused on products and not companies.

And so this is kind of a methodological question for you about how you went about doing what you did for this report. How did you find the products? What did you measure? Um why did you measure those things? Give us more of the the the methods part. >> Yeah, absolutely. Uh so we began actually not with products, not with vendors, not with even AI. We began with the work of emergency management in preparedness, in response and in recovery.

Our sponsor scoped mitigation out. So we looked at what are the use case categories. What are the major domains of work in these areas that AI technologies might support and the search that we did actually went from that foundation. So if there is housing recovery in recovery then we were looking for products to support housing recovery. So the search went from the use case category not uh backwards. Uh so that's how we began is with the use case categories 45 in total across preparedness, response and recovery.

Then we shifted to a multi-pronged approach to try and find the universe, the absolute universe of products that could possibly be brought to bear to help disaster outcomes. And that meant that we were looking for products that were specifically designed for an emergency management purpose and preparedness response recovery, but also that our sponsor was super interested, wanted no stone left unturned. really wanted to understand what about everything else like there might be other things that if they're just configured for emergency management they might become really helpful.

So we employed multiple different strategies to do that. We did talk with vendors. We looked at uh vendors and expos and conferences uh that were doing advertising in emergency management spaces. We talked with emergency managers. uh we did multiple searches of um websites that track this stuff like Amazon marketplace and then we also did independent web searches uh using multiple different strategies. So through that entire uh universe we found about 1,800 uh possible products and then applied a number of scoping uh scoping criteria.

For example, our sponsor was mostly interested in understanding landscape of techn or products that was produced from US headquartered companies. So those internationally headquartered company products got scoped out. Um so we applied that and dduplicated uh because sometimes products are supporting more than one use case even response and recovery and so we dduplicated and then we after having this list then we began the process of trying to gather information about them. you know what would have been ideal, right?

Would have been ideal, what I think the emergency management community most needs and deserves, especially given their limited bandwidth and how, you know, just how underresourced they are as a community. I think what they could have used most as an outcome of this process is tested products that are verified uh that we met with every single vendor vendor and did the demo time that they don't have time to put into, right?

Like that we took that burden on. But what would have been possible with 100 200 products is not possible when you have 1,179 with 717 different vendors. So we had to shift what we originally thought we might be doing to something that was realistic within the time frame and resources we had. And so we tested iteratively over many many many many many many many hours and rounds. Uh what is the kind of information that is available publicly and not just from vendors but from like uh local contracts that are often published from meeting notes of city councils deciding what products they're going to approve.

Uh it's in Amazon marketplace. There's data about these products in many places not just vendor websites. So what are these sources? What is the information out there? what is meaningful that we can gather consistently across the different products that we have now on a list. And that ended up being things like do they publish evidence of cyber security certification? Um have we been able to find problems legal track record like a problem a track record of legal problems or cyber incidents? um based on the kind of data that the products work with, might there be privacy concerns that would be meaningful to emergency management and they'd want to know about it?

Um what is it that the AI is actually doing in the product? Uh what are the use case categories that it could support? Um what kind of data does it need? And is that like a one-time thing uh that the user has to provide? Do they have to provide it on an ongoing basis? Is it one kind of data? Is it many kinds of data? Or is the vendor taking it all on for them? What are the subscription models like? How can they access it?

How much does it cost? We worked really hard to try and answer that one. We ultimately could not. But that's, you know, a sample of the the things that we were able to consistently gather information about or in the case of cost try. So every product that you that you scoped down on about 1,100 right around there, you were able to then look for all look look for each of those different elements that you just described and characterize those products based upon that set of of review um criteria that you were using.

Um so we can then look at individual products and see how they do. Um um but >> classify not do. [laughter] So classify and then and then you've taken them and um in the report you have actually classified them so that when we when we look at we're going to talk about alerts and warnings in a few minutes but um in the particular category of alerts and warnings there are um you've you've rated the diff the products kind of as a category about how they're doing in that particular area.

Is that a good way of describing what's in the report? >> I'd say we're we're looking at them as a group in terms of their features. What are the features of the products as opposed to how do they do or how they perform which is not something that we measured. Highly desirable but not not possible at scale across all all the products we had. So um yeah, then we we gathered information uh triangulated as best we could across multiple sources.

Uh we did a number of different data gathering strategies to optimize and to verify and then humans in the loop um looking across a random sample with multiple AI experts and making sure that we had data that we could trust. I just it boggles my mind how you actually did this because and I'm thinking how many decades did this take because as someone who's gone through different um like if if you're putting out a proposal and you're meeting with vendors and and as you said you're sitting through presentations or or going combing through information um doing it for six it you know and that you're trying to look at I felt taxed after six I can't imagine and going through all of these.

Um, I mean, it just in my mind when you said the full numbers and I'm thinking, okay, you know, was it just a cursory search? Are you okay, almost like a Google search? But no, it sounds like you guys really, you know, dove in and and tried to get all the details. Um, >> well, it also is a search, right, of internet resources that yielded the information that led to the conclusions in the report. what we were able to do because I work at Rand which is a fabulous place like if I had been back in my higher ed institution which for the audience is where I came from before Rand I would not necessarily have access to computer experts uh and AI experts who were able to code uh develop thousands and thousands of lines of code to uh connect to an API uh like an interface with the internet and LLMs to then execute these enormous searches at scale through an iterative process and then to build really complex code on the other side to do a secondary search and then to see if the information from the two searches matched and then to check that again and to monitor and grab all of the data including every single link for every single resource where they found the information and then for humans to get in the loop and to check and check and recheck as we iteratively gather the data.

So I mean what enabled us to do it actually was AI because AI can aggressively quickly search and search a wide range of resources that you and I like you said Eddie like it's overwhelming to try and get information about one product or a few products and you have to really click around those websites right but machines right like the AI is able to do very quickly what it would take us quite a while to do but of course it became really really important that we trusting the information that we had given it appropriate context that we had given it appropriate instructions that we had broken it into small enough tasks which is very important when you're working with those products and then of course to have humans in and on the loop all along the way.

So it was it was pretty complicated but if we hadn't had access to AI we would not have been able to do it at scope scale and speed. That is my favorite way to use artificial intelligence is when it is there to assist people and people are there as a part of it and and are there to lead it and and everything else. Now, I'm a big fan of that. I'm it it sounds like you had just mountains of information, things that that you studied.

What specifically did you guys not study or not do in this study? >> Yeah, I mean, as I've already mentioned, like we did not test individual products to see how they perform. So that is a something that is incumbent on any listener to do and to to make sure that they are pleased with the performance of the products and that they're doing what um vendors or or even like Amazon marketplace would say would say that they do. uh we don't recommend products in that report uh largely due to the fact that we did not test them but also just because that is not the purpose of our research and it was not the focus.

So we're not providing anybody a buy list. We're not saying go out and buy X Y or Z product and we did not actually go and look at what every single person is using. We did end up actually finding a lot of evidence about these general products that I had mentioned, products that actually weren't purpose designed for emergency management, but we found evidence that quite a quite a significant proportion had started that way that had been used.

So, we ended up finding evidence of use, but that is not the purpose. We did not go out and study what are people actually using, what are they actually valuing. to some extent Aspen Digital's efforts of convenings got to some of that um but that was not within our domain or our purview and then of course we didn't report cost I could talk to you all day about cost I think a lot of your listeners if not you yourselves have probably tried to find cost of products out on the market um we had the smartest people around uh that studied cost uh estimation procurement.

We had people that could do coding. We had AI experts. We had people that have worked with vendor over time all on the team trying to figure out how to do this and if we could get the information. We tried so many times and let me tell you the vendors just have it really opaque and we ultimately couldn't have confidence in that data and so we did not share it and so we don't report costs. That's something we didn't do.

Um, and then of course I've already mentioned this too, but I just want to really make sure we're really transparent. Read our methods. If anybody's into the report and into the stuff, read our methods. Our stuff relies heavily on publicly available information. We really did try and triangulate, but we clearly say in our methods and in footnotes that that does then mean that products that don't have a heavy presence on the internet, that could be those in pilot case for instance or or others would be products that we did not discover and report on.

So that's a good list of the things that we didn't do. I think one of the things that I noticed when I was looking through the report is that what's not included in what's actually written down in addition to cost is you don't list the the actual products that are in that are included in each category. So an organization may not know if their product is or is not included because there's no there's nothing's written down about that. >> Yeah, that's very true.

Um there is discussion and dialogue about this uh a lot of it and again we know that what people would like is lists of what performs well and doesn't perform well or even lists like okay here's the 131 I I know I want a product let's start looking down the list and poking around websites and see what we want to demo um we understood that but this market is moving exceptionally quickly and our work ended in March of 2026 I already know of products that have come on the and I already know of products that have been enhanced that now do things that they didn't in March.

That is the primary reason why those lists are not made available. We are not here to recommend products or to bias our community towards one product or another product, especially because we did not test them. So, the risk we would run if we published those lists is you inherently do bias it. you make a claim in the list that they don't have X, Y, or Z because they're on this list with this uh claim about their AI enablement when they've moved past it since.

And so it was really out of preserving the integrity of our work and preserving the integrity of the market and competition um that we chose not to save the entire list. Now, individually, by the way, listeners, Jay Jensen ram.org, or I am happy to talk to anyone anytime. If you'd like to know how we characterize your product or you'd like to see the row of your data, let me know. Let's have a conversation. I'm happy to talk to you about it.

Um, and if the record needs to be corrected, that's something we do all the time in ACT. Yeah. So, happy to do that if we mischaracterized anything. >> Well, that's an open invitation. any of you AI companies that are out there that want to know was I included and how did you how did you uh score my different aspects of my product? >> Yeah. >> Yeah. Uh so let's take a a deeper dive into the area at the alerts and warnings.

Um so you have a whole section because it's one of the 45 areas that was the the impetus for looking at the the product scope. Um, and it's about public communication, alerts, and protective guidance. Um, can you tell us what's included? Because it's not just one category. There's multiple dimensions in that category. Um, and give us an overview of of that section. >> Yeah, you bet. So, we found approximately 100 well, we found 131 products um, in that area that we mapped to the area.

That doesn't mean they only supported that category, right, of use case. It could also for example have supported situational awareness as sometimes alert and warning products do. Uh so it definitely 131 um that puts this uh use case category set of products about in the middle of the pack. Situational awareness for example was by far the most represented in the data. Uh and this one and that had just for your context 516 products.

Uh and then incident coordination 340. So you know this is this is less than than most in terms of the market well populated but not like where it's concentrated. Um we found a bunch of clusters in terms of what the products do uh that would matter for emergency management. There are what you a lot of people who are listening might expect like mass notification platforms um that do you know push uh messages geoargeted messages across like text voice email u mobile push sirens um and and various ways to get the message out even social media.

So, mass notification platforms is one. Um, AI message drafting products, which Janette, I know is something that you are testing and looking at. Uh, but we found those that compose messages, claim to optimize them, and claim to translate them appropriately. >> I'm putting the air quotes on there for everybody here because she's saying it, but I'm >> claim. Yes. Okay. >> Yes. Claim. Right. We did not test products. Um that's that AI message drafting.

Uh then we found like public facing hazard information. I don't know if you're like familiar with Google SOS alerts or watchd duty. These are products that put emergency data in front of the public directly so that it's not coming via an official source in a jurisdiction if you will. Uh whereas that mass notification platform often is a jurisdictional tool um where government is putting pushing out the messages. Um, we found independent of message drafting purely translational and multilingual delivery tools.

And this is a space where those kind of general products kind of come into this space where they offer possibility uh because of what they do, but maybe traditionally they haven't been packaged as part of this category of tools before, but they could be applied perhaps uh usefully. And then finally, emergency broadcast. So, oh well actually not second to last um emergency broadcast. So these are products that automatically capture government alerts and then rebroadcast them over radio and TV and using text to speech uh so that they're they're disseminating messages.

And then finally facility level alerting. So think of these as like mass notification platforms minus right like with a small with a small mass notification. So, and things like uh sensors or gun detection that are being used at sensitive facilities and then sending alerts to employees. Um that would be another category, but that was by far in the minority um that kind of product in this category. So, across all of this, you you might be interested.

I don't know if if users would be interest or viewers would be interested, but the AI isn't what people might expect. There's a lot of conversation about AI like synonyms with LLM which is has an important role to play and as you're testing it again Janette like the quality of those messages and and other features of them and we've really got to look at that but a lot of it is actually dedicated to detecting conditions that are on the ground uh that should trigger the alert drafting the message uh and translating it. the actual execution layer as the market manifests right now for these products um is really rules-based and human driven.

It's doing like um they are not insignificant tasks that the AI is supporting but they are not replacing humans in the loop. Sometimes they're even in the background enabling tools like a mass notification uh platform that they already have already had maybe even for a long time. So just adding AI um to it to enhance its capabilities, maybe make it faster, more accurate or to do other things um that we could maybe get into later.

But uh that's like kind of a synopsis of that market. in terms of the characteristics of those products as relates to privacy or security or um data inputs, it really tracks they that these products really track the rest of the market for the most part. So a lot of the products well uh they require data input on a one-time basis at use and or on an ongoing basis. So data that we humans have to provide. Uh most of them require technical integration meaning that you've got to have someone with IT support or IT background um that is capable of integrating these products into your environment.

Most of them require continuous internet connection because they have they are all or in part SASbased. So software as a service meaning they're relying on the internet and the cloud to function. And then if that's not functioning, then the product isn't functioning. Um, and we can, you know, I'd happily answer more questions that you might have, but in general, if you look at the RAND report, this subset of products tracks with the wider market for the most part. >> Yeah.

And and you were using you did provide a couple of examples as you were talking through the different categories. So just for for listeners, those were just examples. So you might be able to orient to like um the uh watch duty is an example of a kind of product that would be in that particular category of the public communication. But again, you know, we don't have the complete list of 130. So this is but you can contact Jensen at rand.org.com uh >> rand.org or if you want to know about the product that you are that you uh you represent. >> Okay. 100 You said 130.

Is that what that we're kind of >> 131? Um a that that is I I'm again trying to process this. Just goes to show the value of AI because I'm trying to wrap my head around it and yet there are technologies out there that would have already generated the report for me on this. um as you as you talked about the different parts as you went deeper into this um and you actually started talking about some of the findings, but I wanted to give you an opportunity.

Were there any specific findings that really stuck out to you um for the public communications alerts, protective guidance, things that are relevant to the listeners? And there's a part two of the question. So you can do part one and then it's and I'm taking this one very you know internal here because what should we know that as we face the future with the not only development but adoption of all these new tools out there um because if I'm not um someone who does this full-time and I'm not you know in AI all the time as a programmer as really I'm just as a end user are trying to do it.

So, what are things I should know? So, that that's part one and two. >> Yeah. I mean, I'll just treat them together. What stood out is kind of what I think people might want to know or what I think is important to know. Number one, as I was talking about the different things that the AI is enabling. You were sharing mass notification as a thing in and of itself, but like the message drafting as a separate thing. That's not universally true.

Some mass notification platforms have looked into that. But you're hearing that there are things that this tool does that this tool doesn't that this tool does that that tool doesn't. This mirrors a trend we saw more broadly in our data but it's true to these products you might be most interested in listeners and that is that these are relatively narrow solutions that would require stacking. That means that if you want fully, you want all the functional possibility that AI can offer you and you want it in one product, well, the market really isn't there right now.

And it doesn't mean it's not getting there because the technology is there. The technology to do various different things that might interest you and concern you that you'd want ideally in a in a platform. That's not just blanket in every product. They instead they do more narrow things by and large. That's not universally true, but most of the products on the market. So, if you wanted all of it, you'd have to buy all of it.

Not all of it. You'd have to buy the different functions and then you'd have to link them together. Again, technologically possible, but that has real implications for your offices in terms of your resources, both your IT staff's time, uh, up upfront costs to set things up and to integrate them and then to maintain them on an ongoing basis. So if I were wanting that whole full full functionality, that's something number one I'd recommend people really ask about when they're, you know, doing vendor meetings.

Really make sure it's the holistic solution that they want. But also, I'd say that's not where it's at now, but it's likely to move in that direction as vendors are interacting with this community because I do believe that that is what the community most would want. So that's something I think is important. be aware of that stacking um that the possibility is there but it will require the stacking for right now and if it's important to you ask ask how holistic the solution is.

Um, the next thing I think that's really important is to be aware that a lot of the change that's occurring with AI is in products that exist already. And that means that we've had a lot of discussion and and appropriately about the caution that we should approach AI with the carefulness that we need to um take in terms of integrating it into our environments, the necessity of humans being in the loop and on the loop.

These are this is all very very important. And also a lot of what's happening is behind the scenes in in products that you may not be aware of at all and may be relatively innocuous depending on how you define what risk is for your organization. It's just helping the tool go better and faster, maybe be more accurate. So just to be aware of that when you inquire and engage with uh vendors uh in the marketplace you'll ask those questions like what is the AI actually doing here and is it on a back end or is it something that may have direct implications for me my organization and and how we perceive risk.

Um, next translation seems a real near ne near near-term win. Um, within this space, we all know it's important. We know it's been very very costly in the past. Um, and the solutions over time have varied in their ability to to really translate effectively and for the full scope of populations and the languages that they speak um in the jurisdictions you might serve. It's been an ongoing challenge. Um, I cannot speak to the performance of any of these specific products, but there does seem to be real hope in these and why because they do on the whole tend to require far less data and data at use, not on an ongoing basis.

So that eases that kind of burden on your organization. Additionally, while you will need to check it, it is capable of generating first drafts across many, many, many different languages, but again might be present in your jurisdiction. They also tend to require very little integration into your environment. Some of them you can just access online as a software as a service. Why do I call that product category out? Well, I've been studying emergency management a really long time and I know that resources vary significantly and I know it can be a real challenge.

I also know that we've had historical problems of disproportionate impact to varying populations in our communities and we know it and it's been really hard for us to deal with this problem. And so this category of product might be a place to start if you're in a lowcapacity jurisdiction, but you care about these issues and have been wondering how to how to meaningfully move progress forward. Uh so that's why I'm that stood out to me and it's something that some some listeners might be interested in.

Um the last one I you know is this good? Is this bad? Let's talk about it. uh you know very little of these are actually doing protective guidance personalization and is that good is that bad? Uh you tell me again but there we do know that people vary in in exactly what their needs are for protective actions and uh they live in different explicit locations in their communities. Um, you know, I know Janette's research, so I know that we're not universally drafting perfect messages now.

Uh, so it's not doing that. Could it really help us with that? I don't know. Is it great if if it does? Maybe if it's done really well, but again, we didn't test the products. So, they're not doing that right now. And I want you all to know that that's that's not what they're doing. Humans are really important. You matter. [laughter] you still really matter in terms of drafting those messages, knowing your communities really, really well and understanding where there are some differences that might be meaningful and, you know, adjusting your communications accordingly. >> Well, I just was on the phone um with an emergency manager passing through a pretty serious emergency and then again um with we we just had a hurricane that hit uh the big island of Hawaii, >> right?

Just just recently. >> Um to get fit again. I think >> I know we're hoping that one veers, right? That's that's the hope that >> it's going to move. Um, but you you talked about different communities that have limited resources that are trying to look for tools out there to help them process all this information. Um, but because again, their resources are small. What's interesting is, and we've talked about this on other podcasts, even just using the term AI turns some people off.

And and I think one of the things that people can benefit from your study is that holistic view of it is in so many things and it is going so many places. Um and and I appreciate you identifying some of the different aspects of it because AI cannot just be talked about as AI. It really needs to be broken down into the different parts because as the person I was talking to just yesterday who said, you know, no, they they don't want to use AI, but I'm like, yeah, but they'll use their iPhone, they'll use Siri, they'll use all these things.

And it's like, >> yeah. I'm like, help me understand. Well, they said, well, help me understand. And I said, well, again, big umbrella, very specific tasks within artificial intelligence. and we've just become very accustomed to some parts of it. Um, while others are definitely that next step and are going to take a maybe a leap of faith or maybe just some understanding or they start by reading your report. >> I think that's a that's a great start if you can stomach it.

Otherwise, listen to this podcast and like get some familiarity. Um, we're trying to reach audiences where they're at understanding the technical nature of the report and how long it was. But that would you know that was required frankly like that was a necessary foundation for conversations like these not everybody's going to have the bandwidth but I do want to highlight that um and build on something you mentioned which is called the AI effect.

It's our tendency to uh relabel and normalize a a AI technology once it's become uh reliable like once it's already doing things then we tend to just like call it software it just becomes a product um AI has been around for a long time and it's been reflected as Eddie you may made some some um examples there uh it's been around doing a lot of things that we have appreciated and valued and then relabeled accordingly to diminish its novelty, right?

Like because it's become normal. We have to have a real conversation in emergency management uh about where there actually is AI that we need to be careful and cons and concerned and perhaps concerned about and where there is AI that is eventually just part of products. it's just a product, you know, because it's just simply doing something that isn't the risk that maybe we thought it was until we learn more about it.

So Eddie, to your point, like in having the conversations through our colleagues, I just encourage everybody to do a little bit of that boring research, a little bit of that boring work to learn about actually AI, you know, and the range of techniques that are out there and actually what it does. uh because I think you might learn that that that playing field that of technology is far broader than you might have thought because the conversation is really focused on one part of it and and justifiably uh but there is the rest of it that might interest you and really help your jurisdiction and organization too. >> Yeah.

And my mind is spinning as we're talking about all of these different things. >> Yeah. There's just there's so much and I and the report is long and it is it is dense. There's a lot to read. Um and I really encourage anyone who's listening to this podcast to go and get the report and look through it. Um and find those sections that you're most interested in. And you know the alerts and warning section I think is you know what we're covering here but there's a lot more to learn about because there are the 45 different use cases that they started with.

So emergency managers there's definitely more to reflect on in the report. Um but it does it does require a few minutes to sit down and get through it. >> [laughter] >> It does. And and nonprofits too, Janette, too, right? Like we highlighted some of the products that have helped them um or utility companies or the other players in emergency management. >> Yeah. Yeah, that's good as a good reminder, too. Um so there was one there you had several recommendations in the report that there's one that really stands out to me. >> Yeah. >> Um and it's directly related to what you did not include.

Um >> absolutely. And that recommendation is about launching a national level EMAI readiness and adoption accelerator to function as an independent consumer report style clearing house and provide product evaluations, standardized buyer guides, model procurement and contracting language, peer learning cohorts and practical deployment templates for lowcapacity jurisdictions. And I read that entire recommendation for our audience because I wanted them to hear how big this recommendation is.

If this this huge accelerator could actually be created and what it could contribute to this to this group to any organization that's trying to make a decision about purchasing, integrating, implementing that technology product into their work. Um, so, um, I wanted to I wanted to just dive in and talk about, uh, just like one of a few of those individually, the different things that were recommended and what it would look like.

Um, so what would an accelerator look like for product evaluations? >> Yeah. Well, uh, first let me make clear like this is an old recommendation in part. It traces its one age to 2007 when theies recommended such a like a consumer report style clearing house for these techn well not these technologies. It wasn't oriented to AI specifically back then but this need has been known for a while and requested and like it's been recommended that it be pursued and it just hasn't up until now.

So I want to call out it wasn't like an original idea in in full. I was like, "No, that that's a really good idea still and the field really really needs it still." Uh, so just want to be transparent on that front. I also want to say it's not uh new in the sense that there's some precedent. Department of Homeland Security has the Saber program. I'm going to look at notes. It's system assessment and validation for emergency responders.

That's what Saver stands for. But they they do a lot of what we're recommending in the accelerator concept. I mean they do impartial hands-on practitioner relevant product testing. They answer two questions. What's available and how does it perform? We covered the what's available not the how does it perform. So there's definitely precedent for this kind of a model to exist. We didn't say who should do it. Um, and I don't don't think that that matters as much as fulfilling the function.

But those in the position to have the resources to do it to fund the work might look like they tech, might look like hyperscalers, might look like philanthropic organizations. Would they be doing the product testing? I do not think that that would um be ideal because of a conflict of interest that some might have because big tech is producing products in this space. So who uh well you'd be looking for an organization that's a nonprofit that has no commercial interests and has sufficient technical capacity to do it.

But this field really really needs it. And so between that kind of saver model and what we can see in around the globe actually there are these kind of efforts that are trying to produce the rest of the the recommendation uh templates and product reviews and starting contracts and RS requests for information of vendors. Um city of San Jose is something y'all might want to look at. there's an AI coalition there and a website that I'm not speaking to the quality of the individual things there.

I am saying that there is precedent for exactly those kind of resources to be provided. I just don't know how many people know to go to that website and again I'm not commenting on the quality specifically. So I I think this is absolutely doable and I I just you all know this community they have deserved and needed this for so so long and the you know the pace and scale of disasters it's not decreasing and the resources available to them not increasing.

So the problem is getting amplified and exacerbated over time and the the field needs these resources but they need to be able to trust them. they need to be able to count on them. And so I think this accelerator concept is one that we as a community like scholars, vendors, everybody. I think it's one we need to push forward and find a way to fund and to sustain because the work is too important and the situation um is urgent. >> Yeah.

If I can just make a very brief plug for one thing that I'm aware of. um this there's an idea within AI um review of something called benchmarking which is essentially evaluating a product based upon a set of criteria that are set by subject matter experts and then they um the product is validated against that criteria to see how well it scores and that is something that's happening within the alert and warning space um and it's sponsored by a company that has no conflict of interest in this area because they focus on preparedness and planning, but they're building benchmarks for alerts and warnings.

And that company is EM1 and people can look it up, but you won't see anything about alerts and warnings there yet. Um, but it's that benchmarking process that is going to demonstrate how well other alert alert and warning products or products that are out there that could be used for alerts and warnings, how well they stack up against the standards that are set by the the researchers who have said this is what a good message should look like.

Um, and I would love to see that kind of benchmarking happen in other product evaluations. Um, so that people can really trust that the quality is is good, that there's no hallucinations. It it tracks with the the research record, but it's also clearly aligned with what that organization does or is presently doing in a response or recovery activity. Um, it's a complicated process. It takes a ton of human hours, which s maybe not so surprisingly happens in a snap with AI doing the validation of all of it.

Um, but it's just an incredibly important thing to to be doing for for any of these products. >> Absolutely. I mean, and you're tackling one of the domains. There's 44 left people. Like, there's room for people to contribute here. If we can't do this at scale at like a national level in a coordinated way then scholars you're out there like and I know that you're you're interested some of you in these other areas. This is something at minimum if you can't engage in product testing yourselves really taking the issue of benchmarking seriously thinking about how all of our quality and effectiveness research translates into assessing the effectiveness of these products or these different spaces.

That alone would be a significant contribution uh if to no one else than the emergency managers who would now have language right and ability to discuss and ask questions purposeful questions of the vendors about the performance of the products. Although clearly we'd rather have have people testing them and relieving them of that burden and those conversations as much as we can. >> But yeah, it's super important to see more of that.

Yeah, >> I'm looking right now uh because you mentioned earlier that you know this is for you know um it can be applied for people with a lot of resources, those with limited resources. I'm trying to think of the exact way it was said in the recommendation. Um I'm going have to go back because I thought it was actually really funny. Janette, how was it for um oh for low capacity jurisdictions? I thought that was a really nice way to to say I I my mind goes a few different places, but having worked for places that that have abundant resources and those that really are stretched with with one um I was just going over some numbers here that that are just it's amazing to me that we have close to tens of thousands of cities and communities with less than 10,000 people in them, right? um for for cities that are over over 10,000 according to the internet.

So it is not verified but 3,200 and change um cities over 100,000 343ish and then it says 12 in the United States with over a million. Um there there are so many people out there with different uh abilities and capabilities, resources, and it may not be as necessary for some, but uh I can really think of tens of thousands of communities that are going to have to rely on technology like this in order to be the best that they can and respond the best that they can.

Um, and I'm going to go back to Hawaii. Um, because they are limited in their resources. Um, and I know I'm going to get a comment or two by saying that, but this is just reality where they they are isolated. And I've been told it's the most isolated place on Earth geographically. And so they have to be able to do things on their own for longer than what we would do here on the mainland. and looking at the the tools that you're identifying or the categories um as areas for them to develop I I think are are key for them to just help themselves, you know, and help their community.

Um so, and I I don't want to get into Janette's final question because I'm looking forward to that about like all those next steps here, but is there a way to break down I mean, because it is a really, as Janette called it, a very dense, it's there's a lot of information. Is there a aid study for dummies coming out or anything to help break it down even in a simpler way? >> Yeah. So in the um if you were to visit the aid initiative website which I encourage everyone to do you will see that there is a reports page and it has the aid report at the top prominently featured and it includes the best of the findings or the most important of the findings from the Aspen report from the Rand report and then includes the direction for where aid plans to go next.

And so that is a a short much much shorter document with highlights. You will also find there one pagers for each of the reports. So again the most important details for each of the reports. I got to tell you like this was I've written a lot of things in my career. This was an extraordinary challenge. Three functional areas 45 use cases 1179 products. uh needing to discuss not just the products and their features but whether they could be implemented or are likely to be adopted in the near future.

That's a lot of terrain to cover in a uh in a in a document. Uh it was a it was a challenge to to get it there. But again, what we're offering the field in this report is the authoritative resource upon which other things can be based. So please consult those other resources. They're for you. And I'd also add that I am really willing to talk to anybody any time. I mean this is been an important aspect of my career to date.

If you are in practice and you have questions, I want to talk to you. And so again, anytime reach out to me. I'd be happy to talk to you. Just talk, converse about what we learned as relates to your interests. Uh and your interests might not be the same as another person's. And so I can cater our conversations, tailor them to to what you're most interested in because I I see the necessity of the report as written and also understand that people have limited time and bandwidth and different interests.

Um so yeah, Eddie, I just want to make sure your listeners know that that we're here to help. Well, because that's where again the big recommendation that Janette talked about that you guys are the ones that cited is I think of everybody out there who is again in the trenches how valuable that would be cuz they could go to a single source to say okay let me evaluate this or let me see what the evaluation is instead of it being rellyant on every single one of those individual municipalities or or things that I mentioned to do it independently.

And that's that's where I think we get into trouble as practitioners is because I may not do my due diligence and I may hear, you know, oh, somebody recommended it, so I'm going to choose this and it may not be factually correct, right? I I could have been sold something um versus it being good. And I am like that's why when Janette was talking about that recommendation that I saw earlier, I'm just like that would be that would be amazing because currently it it comes down to the saleserson and then it comes down to the individual entity to to verify all the claims that are made.

And um I think there's a bad taste in a lot of people's mouths because of past experiences because there just isn't that reliable source. >> Yeah. >> I got to say um there are other models elsewhere in other industries, right? Consumer reports is an actual thing. The entities pay to have their products reviewed. That can be a way to get this done is you want to be in this market. you want to be marketed to this community, you pass this, you know, bar, you pay and you have your product reviewed.

That's a model. Um, standards is another way that's done in other industries. There are standards for products and so there's an independent testing entity that blesses it and if it has that stamp, that means that that product's okay. Um, and so that is funded through insurance in in some industries. So there are there are a lot of ways to get this done. if there are people who want to talk about getting it done. I'm your girl.

I'm here. Let's have a conversation. It's like you said, it's brilliant. >> So, in as we wrap up, uh I want m to make sure that we talk a little bit about this summit that's coming up where I think you're going to be discussing the report maybe in greater depth or it's going to lead to more conversations. Um so, the summit, it's the aid summit occurring on September 9th in Texas. what what's it about? What do you hope is going to happen?

How is this going to affect emergency managers? Like what's the next step? >> Yeah. So, the reports really are going to play a very very small role uh out outward facing role at the summit. Um they're they're the evidence base upon which we're having the summit which is we know that there's these disconnects between the vendor community, between the research community, between the emergency management community. We know that there's things that could be done.

Let's and we know that there's a community of emergency management that is interested. They're cur they're AI curious at least maybe skeptical in some places but need to learn more. I recognize they want to learn more. So bringing these groups together to have conversations about what matters to emergency management, what considers what what what are they concerned about, what are they thinking about, being able to dialogue and and hear from these other entities that are impacting their space and looking for that kind of synergy across them. uh so that we can really move this field forward in a careful purposeful manner but in a collaborative manner that maybe we haven't seen as much of yet as we would like to see.

So that's the summit and out of the summit we hope people come um having been heard um inspired to maybe do things a little bit differently as they can within their respective domains to make either the best use of existing technologies or to produce technologies that can be adopted broadly. And then the next day there's actually going to be a follow-on discussion with with some entities to look at a what a longer term alliance might look like.

So it's not just like a oneanddone day where we have a conversation and then it's like well what happened to that where'd that go? So they're really going to bring together some additional people for a really focused dialogue about what an enduring entity might look like. Um and uh that conversation will be really exciting too. So I think the summit's an important moment for the field um to kind of as a benchmark if you will or a milestone uh on this journey forward and I really look forward to what's coming out of it and I hope that we see a lot of practitioners there uh so that your voice is heard and represented in all the conversations >> and will it be conversations or I I saw a list of really wonderful speakers but are how is how is it structured?

Will people actually get an opportunity >> to talk with each other and do that kind of networking outside of outside of breaks? >> Yeah. So, from what I understand to date and the agenda is still coming together as we approach the event, the model is more a panel model with discussions as opposed to um presentations. Hence why the reports are not the focus of the day, right? We're not getting up there and reporting on a thing that you can download and read yourself.

So instead, it's focused more on on dialogues through that way with experts on the panel representing different these different domains and conversation Q&A back and forth with the audience. So it should be more of an active day um than than a sitting on your hands and listening day. >> Yeah, that sounds like a great way to have a summit. And I love the idea of the second day where there's something that's really oriented towards action.

Um, which is exactly what your recommendation was about is having action to create something that can really support this community um in their understanding and adoption of AI. So um this has been a really great conversation. Do you have any closing thoughts? just thank you and thank you for the warm room and the alerting authority and all the things you're doing Janette and Eddie for this conversation and all the enduring presence that you're having your commitment um doing all this stuff can be extra and you're giving it the time and allowing me in this case to bring some discussion of the evidence of our report to listeners that otherwise may not have had the time the luxury of time or the even the interest in waiting through uh the density of the report.

So, thank you so much and really appreciate your work. Site it site it often Janette in your case. >> Well, thank you. Thank you. It's always a pleasure to get the opportunity to to work in this space and I I I think that the this the AI report is just so needed for helping to move things forward. So, we really appreciate the work that that you've done and I look forward to hearing more about what happens after the summit.

So, >> yeah, we'll look forward to Did you have one? >> Yeah. >> Just that guys, I know the space can be scary. Like absolutely. Um and often though for me when I'm when I'm nervous about a field or about something, it's because I don't know enough about it. And and the more I learn about it, then the more comfortable I tend to be because I understand what the parameters need to be or or the guard rails or anything else.

Um, so I would just encourage you to open it up, take a look, learn about this stuff because when it comes to AI, we're not running away fast enough that you you can't. Um, it is going to be a part of our lives more and more and a part of what we do in our different jobs and assignments and how we do them. Um, and so just the invitation again, just as as was shared here, is learn about it. um know that there are um studies and things out there to help you.

So, you are not alone. If you are in that small little community and you're the dog catcher and the emergency manager, um you can do it still. So, just thank you very much for being on and helping to educate me more on this. [music] That was a great conversation and there's just so much in that report that really requires a deeper dive to get the details of it. But I am appreciative that there are those executive summaries that Jessica pointed us to for anyone who's looking for a written summary um where they don't have to read 150 pages of deep detailed text. um and may come out at the other end wondering, well, what else do I need to know?

Um and it's clearly identified in the executive summary um those recommendations that we talked about with her um how how important this uh consumer report type of an organization would be for the whole area of emergency management and practitioners. No, I And again, this isn't the start and it's definitely not the end. And so, it's just that idea of guys, this is a continued um topic that we're going to we're going to hit because it is very relevant to what we're doing.

And and for those that again are just starting in this field, welcome. For those that are the the trailblazers, you're probably saying, "Okay, hurry up. We've got even more stuff coming." And for everybody in between, that that's why we're here. Um, so I think it's great. Um, >> yeah. So I I do want to to point out that the the conversation we were having about benchmarking and alerts and warnings and how important that is.

Um, yes, it is really important if you're going to use AI for writing your alerts and warnings, knowing that the system you're using meets the standards that have been set by academic researchers and the uh what we've learned about the should be included in messages, but it should not override your personal professional knowledge about alerts and warnings and all of the things that we've been teaching. in this podcast and in the writings at the war room and in the trainings that the alerting authority does across the country.

U so I do want to emphasize the importance of the human in the loop and that your knowledge that you bring to the use of AI is going to become even more important as you are thinking about how to communicate effectively. And if you have questions about how to communicate effectively, this is where reaching out to us is going to be really beneficial to you. We do provide the trainings to organizations across the country.

We conduct message audits to look at the messages that you've sent historically. We look at your templates to make sure that they do conform to the standards that have been set. We provide policy assistance. And if you love doing deep intensive work with subject matter experts who are sitting with you on the computer for 10 hours over the course of a week, we have a warning boot camp and the next one is scheduled for the week following Thanksgiving um which is coming up in just a few months.

So we hope that you guys sign up um reach out to us. Again, we love hearing from everybody who listens. I mean Janette and I talk about it um when we're not on the podcast. We'll talk about people who've written in and questions or comments they [music] have. Um, we're here for you. This is the alerting authority. We recognize every second [music] has a story and we are happy to help share yours. Thank you for listening. >> [music]