Beyond the Bot: Introduction to AI in Mental Health Report Writing

This episode focuses on AI report writing in psychological assessment, with a special emphasis on school psychology. Dan Florell and Adam Lockwood look at why AI is being adopted so quickly, where it genuinely helps, and where human judgment still matters. They also dig into the tension between compliance-heavy technical reports and the need for parent-friendly, usable summaries.

In this episode, Dan Florell, a professor at Eastern Kentucky University and founder of Psyched to Practice, is joined by Adam Lockwood of Kent State University, who consults on AI implementation and AI report writing. Together, they explore report quality, defensibility, readability, and the practical realities of using AI in educational and clinical settings.

Brightminds Ad

PAR Ad

This episode is brought to you by PAR.

Explore the AI Report Writer here:

https://www.parinc.com/product/ai-report-writer?utm_campaign=38111624-Psyched%20to%20Practice%202026&utm_source=P2P%20Podcast&utm_medium=Related%20Podcasts

Learn more about the RIAS-2 NU here:

https://www.parinc.com/product/groups/rias-rist-assessments?utm_campaign=38111624-Psyched%20to%20Practice%202026&utm_source=P2P%20Podcast&utm_medium=Related%20Podcasts

To hear more and stay up to date with Paul Wagner, MS, LPC and Ray Christner, Psy.D., NCSP, ABPP visit our website at: 

http://www.psychedtopractice.com 

“Be well, and stay psyched"

Transcript

SPEAKER_00 1:01

The following is a site to practice production.

SPEAKER_01 1:14

Well, welcome to Beyond the Bot AI and mental health. We're going to tackle a unique AI kind of topic, especially in mental health coverage for school psychologists, and that is report writing. But of course, that goes across the gamut of a lot of the mental health field, as many of us are required to do psychological assessment and coming up with the right type of combination and whether we can just press a button and forget it, or if there's a little more nuance to that. As you might guess, we are going to go with the nuance a little bit more on this particular podcast. So I'm Dan Florel, professor at Eastern Kentucky University, in addition to having a private practice and an online continuing ed company, Cite the Practice, which this is a podcast part of. And I'll let Adam introduce himself.

SPEAKER_00 2:04

Thanks, Dan. Adam Lockwood. I'm at Kent State University. And uh I guess I should do my conflict of interest since we're talking about report writing. I do consulting around the implementation of AI, particularly for AI report writing. So for example, Parr is um a client that I work with quite often. So I want to be transparent if I'm out there ringing the bell for Parr, you know, keep that in mind that I do have a relationship with them.

SPEAKER_01 2:41

Well, and I think uh we agreed today when we were getting ready for this episode that our focus is really going to be more global on a lot of the issues that I think are real pertinent. We're not going to bring up any particular products at this point. Maybe in later episodes we might start doing some reviews and once again being very transparent about our roles since uh we do have companies that reach out to us for consultation or even a little bit more than that. And we'll make sure to let you know if that's ever the case. But I think we have several episodes worth uh of just talking about generalities because this is something that I almost guarantee no graduate training has gone over with their graduate students in regards to how to treat AI in the report writing process, other than maybe Kent State has a session or two. I'm just gonna go on a limb, Adam. Do you talk in your grad class about AI report writing?

SPEAKER_00 3:39

I talk about AI all the time. I don't know that they listen, but just my quick uh spiel about that is I don't want my trainees, including interns, using AI unless they're writing it themselves first. We're gonna get the de-skilling of lots of skills, but yeah, if you can't do it on your own, you definitely shouldn't be uh outsourcing it to AI.

SPEAKER_01 4:05

Right. Yeah, that's good advice. So I think the first thing that we're going to really discuss is why we're even talking about this sort of thing. And I think it may be fairly obvious in the fact that as of five years ago, nobody's really talking about having an AI report writer. If you're lucky, you have maybe a template that you kind of ginned up with some programming language to kind of get you a really efficient template that can switch gender and you know add little sections here and there. Uh, but AI was a real quantitative leap forward. And Adam, you want to talk a little bit about how AI has been a big leap over just the traditional template formula approach that a lot of people learned to do for their reports.

SPEAKER_00 4:55

Yeah, well, I think the first thing is is before we even talk about that, is why has this been adopted so much? And the reason is that most school psychologists spent on average about a day a week on just report writing. So when this came out, it just filled a need. And there were there were things, you know, you had your click reports, and I had students that were creating their own uh Excel spreadsheets that would populate Word documents, doing some cool stuff. But for me, the first time that I I tinkered around with ChatGPT, I thought, I wonder if this can write a report. And it could. Even back then, it could do a pretty good job.

SPEAKER_01 5:35

So I got off track there, but uh no, it just well, we're talking about you were impressed with AI, right, and how that it is now able, even in its nascent stage, right, able to go ahead and create a pretty accurate looking report. And really, why would we want to focus on this for an episode or several episodes is because of the increasing value uh for school psychologists who do a ton of assessments, but this could be for neuropsychs, it could be for clinical psychs that do some testing. And so, you know, looking at time savings, but also I think for school psyches in particular, just uh the job expectations because of the chronic shortage in our field, not having enough time. I always with my students when I was in our training program, the good news is you will be able to get a job and you'll probably be able to get a job pretty close to wherever you want to live. Bad news is you're gonna get that job and they don't have enough people to do what they're asking you to do. So uh lots of good mobility and opportunities, but also the downside is when you have a situation like that, it implies there are enough enough people in the field to meet the demand. And so uh it usually translates to your job narrowing a lot more into just assessment, which most school psychologists didn't go into the field just to assess their entire career, but also to vary it with consultation, intervention. And so this may be a very appealing aspect to adopt because of the time savings that uh you're not going to be spending and then can spend on some of those other activities that are part of a well-rounded service of care, I would say.

SPEAKER_00 7:23

And I would add that when used wisely, you can improve the the report writing services that you provide. So, like some of the research that I've done suggests that AI does a better job of writing reports than clinicians do. Sure. There always is.

SPEAKER_01 7:45

So when you make a global statement like that, because I remember at least some of your earlier work, uh I know that you've had a lot else come out, but you know, there was things like uh report recommendations that AI tended to do that quite well, but with say summaries of the content and like your conclusion, the AI struggled to kind of uh sift out the report data that's important to have in a report, but it's not the main focus. And humans much better at knowing what is the key stuff that you really need to have in a conclusion. Uh, and that kind of was where the advantage for the human clinician was at the time. And once again, I think that research, if you recall, was just kind of a general purpose, generic AI at the time. And that was oh, what, three, four years ago generation. So, I mean, if you played with AI now versus four years ago, you would see a huge difference in output, I would imagine.

SPEAKER_00 8:46

Yeah, and I'm working with with Parr right now, and we're comparing the different sections of reports, humans versus AI, empirically. And I I think we're probably gonna find that across the board that people prefer AI. But you know, I would argue that, and I guess I'm becoming more Pollyannish about this as time goes on.

SPEAKER_01 9:08

You're gonna knock me over with a feather, Adam, because you know, that's what you always claimed I was more of, and that you were a little more of a cynic, I would say, in some ways.

SPEAKER_00 9:17

Aaron Powell Well, it's funny how now that can happen sometimes. Uh but I would argue that recommendations are the most important part of a report, anyways. And so if it's report if it's improving reports, it's improving the the most important part of the the that's the whole purpose. I mean, the AERA, their whole thing is like assessment should lead to intervention, should lead to treatment. And so that's really the point of this. I mean, eligibility, of course, is necessary, but when you're assessing, it should be to improve the lives of clients you're working with or the education of the students you work with. And the truth is a lot of psychologists, a lot of school psychologists, me included, have a bank of recommendations that they give to almost every student, time and a half, preferential proceeding. And, you know, those those can be helpful, but AI can take the specific strengths, interests, and areas of needed support and tailor recommendations around that and the location of where the student or the client lives to make recommendations that in my mind are often superior to something that I would have come up with on my own. I just don't have the time to do that research. Yeah.

SPEAKER_01 10:42

No, I I agree with you. I I think one of the reasons when I first saw that result, uh, I thought, well, that's because it's AI's like that little puppy that's not learned anything and it's just full of energy and you know the overachiever. Whereas as a school psychologist, you've been in so many meetings where they're just basically, does the child qualify? And then you defeatedly three years come back to the child's file, look at the report, and the report's not even creased. In other words, nobody has looked at it. And so after a few years of that, it's only human, I think, to be just slightly discouraged of spending three or four hours coming up with marvelous tailored interventions for students when you sadly feel that nobody's looking at this thing. And then it becomes reinforcing, right? So that your recommendations get less and less specific, more generic. And so then if anybody did read it, then they're less incentivized to try to read a new one because they're too generic to be helpful. And so maybe throwing AI into this process in that regard may break some of that cycle. Uh, and maybe also that uh school psychs and other psychologists have to do a little more training on the end users of these reports to actually, you know, don't just go to the summary and then stop. Please look at those recommendations. They are extremely tailored to hopefully enhance the success of that student, adult, child, whoever you've been assessing.

SPEAKER_00 12:16

Yeah. And Dan, I think if we're going to be honest, we have to accept some of the responsibility for our reports not being read. So if you look at the research on the major stakeholders, teachers, parents, there if they think these reports are understandable and have any sort of utility, the answers you're finding aren't too great. There was a recent one that just came out, and I'm I'm trying to remember who it was. I I decided it, so I'm like, I can't believe I can't remember this. But yeah, so so part of that is on us. And that's another thing that I think AI definitely helps with is help me take my jargon-filled report that I have to have for compliance, that I have to have to meet the letter of the law and generate something that is jargon-free, maybe at the sixth grade reading level, so that parents can understand, maybe make it a little shorter.

SPEAKER_01 13:13

I love that idea. And like I well, really one of the reasons that we kind of got to talking more regularly is that we had talked about the flexibility, the promise, like real early on that AI could provide. But then I also go back to a very intense experience in the Houston public schools where I did my internship long ago. Uh, and I remember they were under audit, like many places are under audit, you know, from time to time. And for those of you not in the know, the Houston Public School District is at the time, it was like the seventh largest in the country. And they would come and do these half-day, all-day trainings for this audit that was upcoming and said, you know, like, if this item with this particular verbiage is not in the report of any child we have in special education, then it's it's a failed item. There was not even a 99% required. It was 100%. And so, you know, yes, it's great to think about we can modify it, we can make it more friendly, but then what if we make it friendly and it loses the verbiage that absolutely has to be there? And then so it makes me paranoid. And then it's like, okay, we have two separate reports. We've got your legally defensible, all the jargon that needs to be in there, and then we have the sixth to eighth grade level explaining it for parents. But then we need to give both reports to parents, I assume, because you can't just give them the parent-friendly report, because that's going to, by making things more accessible and understandable, you are going to lose some of the nuance that the technical language conveys that can't be conveyed in just common language, or at least not easily. I mean, you don't want to make a 15 or 10-page technical report into a 45-page but very easy-to-read report because nobody's going to read that either, I would think.

SPEAKER_00 15:39

I always argue for three things being provided: the 30-page psychobabble, jargon-filled reports that are required for compliance, a shorter parent-friendly report for parents that's very actionable, and then bullet points for a pediatrician.

SPEAKER_01 15:56

Would you do anything for teachers?

SPEAKER_00 16:00

Or the educational one. I did look and it was, I thought it was, I was like, I wasn't sure. Matt Burns, who did a study on recommendations in school psychology evaluation reports for academic deficits. And basically people are like, I don't know, uh, this isn't too helpful to me.

SPEAKER_01 16:19

Right. Well, I mean, I can believe that. Um, you know, one could argue that the jargon field is one of the reasons that, you know, we're a profession, like any profession has jargon. And then it's, of course, to your point though, you want to make sure that it's actually usable. I mean, in the end, is the real purpose. Uh, well, that and to get paid for the service. If you're not in the schools, if you're like private practice, you've got to get paid for the service. And it's the reason why we do some things that also are not very friendly for the end person or the target of the assessment. So yeah, but I I agree.

SPEAKER_00 16:54

Well, I mean, sometimes specific jargon is necessary because it's the language that most accurately describes what you're seeing. But and so, you know, you need to have that included. But if it's only understood by someone who reads it every three years, there's got to be a it seems like it would there should be another another something provided for everybody else who are actually more important in that student or or client's life than the psychologist.

SPEAKER_01 17:24

Yeah, my uh colleague Ray Christner, you know, we've talked about assessment and evaluations, and he's certainly in the camp that you're talking about, and to the point, you know, where he has gone to more thematic-based reports and not the you know, instrument description, chunk instrument finding, instrument description, instrument finding. And maybe at the conclusion, you get a little bit of integration of that data, which is a very common report style, and I will admit to using that myself because school districts and stuff, they're paying me, and that's what they expect to see. They don't want some avant-garde person that's showing them something they're not expecting. But I mean, there's been uh a thematic report writing kind of style. There's been books that have been written about it, and of course it makes sense. And it even kind of goes to the core as a psychologist, uh, you know, that you want multiple sources of data coming in. And so if you have adaptive behavior indicators in, say, your rating scales and your adaptive behavior forms, why wouldn't you combine those two in the same section of your report? Why would they be in two separate areas by instrument? Because it's going to help you conceptualize better, and you're more likely to be accurate because you have multiple instruments that are indicating uh from this. So uh I'm in agreement. Uh and but you know, I guess AI could do it both ways, the flexibility of the AI report writers or not. Uh as I've been and I haven't used the AI report.

SPEAKER_00 19:01

What I don't know, Dan, and I would be really curious to hear. Maybe I should ask Christopher Thomas. He's uh down in Florida, but are there any legal ramifications for having a a not quite technical report that you're handing out? Does that somehow open you up to you know liability? And I don't I don't know the answer to that, but it is something that someone should should write about.

SPEAKER_01 19:34

I think I think you've already been saying how exhausted you are with all the pubs that you have coming out. So, you know, just put that one aside and uh, you know, when you have a little bit of free time, you can work on the legal ramifications of AI, different AI report styles, or how consistent or what are the core elements that must be within a report, regardless of the style that you write in.

SPEAKER_00 19:58

Yeah, good, good idea. I'm working on a couple defensibility, liability things with some people who have their JDs. I think that's gonna be of interest. It's definitely of interest to me.

SPEAKER_01 20:12

Like, well, yeah, I mean, I think it would go towards the recommendations that we would make for our listeners or other psychologists, right? I I don't think anybody knows at this point, which is a bit scary. Uh, but I really hope that there is. Maybe there's just a core that has to be the same. But I really like the idea of being able to have multiple reports from the same data on the same student to provide to the different populations. Ideally, that would make them more utilized because I know I often write with half an idea towards, like when I do in my private practice, I'm writing for the parents, but they also want to present it to the school for possible services, but they also want to give it to a pediatrician or child psychiatrist for possible medication. And so when I'm writing, what do I need to do? I mean, I can't, I don't want to make it uh more simplified so that a parent would pick up on it totally, because then a pediatrician may say, well, this is too basic. It's not really, it doesn't even look like it's professionally written. You know, like this doesn't seem to be, or this doesn't have the codes that I need to have the assurance that I'm going to prescribe this medication.

SPEAKER_00 21:27

Give them some bullet points, because having worked with pediatricians in hospital settings, they don't have much time, you know, and and they're not gonna, they're definitely not gonna read a 20-page vow. They're gonna thank you for it, they're gonna scan it, or they're gonna give it to their administrative assistant to scan it. But if you give them a couple of bullet points, I I think they would glance that over.

SPEAKER_01 21:48

Well, Adam, you know, I like how you're optimistically thinking that the pediatricians are reading all their content and not having AI summarize it for them into bullet points. So would it be better to have the highly detailed legal document that they can scan into their EMR, which has the AI scribe as part of it, can break it down and then give you the core of what it's needed, like diagnose, here's the diagnosis code, here's the optimized treatments for these given prescriptions, here are some of the identified weaknesses from the report. Now, would it be even though you're doing it through AI, would it be more efficient not to do it through AI, assuming that the medical professional has AI on their end to interpret your report?

SPEAKER_00 22:37

Well, that's pretty meta, man. I think that providing both is good because then the bullet point is like, you know, you show up to a doctor and you're giving them something for the first time, they can look it over and have a little bit of information. And then next visit, they never remember, right? They they go back and have the full brand of AI.

SPEAKER_01 23:00

Give them a summary, but well, you know, full disclosure here, I'm married to a pediatrician, so I get to hear the whole side of things uh when she gets stuff, right? And so I've been hearing all about like open evidence and uh AI scribes, you know, that are embedded in their electronic medical record systems or being able to offer intervention suggestions based on, you know, open evidence's AI recommendations. And so it's uh I know it's intriguing. You know, and like you said, it gets a little meta when you start considering we are not the only profession that is incorporating AI in the healthcare system. And there are some that are way beyond us in exploring some of the implications. So, you know, how do we work together, even with those other professionals, healthcare professionals, to get the optimized outcome for the students, children, adults that we work with?

SPEAKER_00 23:59

Yeah. Yeah. So, Dan, so for people who don't know, open evidence is uh kind of a it's like a database where you ask questions and it supposedly searches through like the New England Journal of Medicine, JAMA Journal of American Medical Association. You know what's interesting? There was a study where they compared just straight up frontier models, ChatGPT, to open evidence. It didn't, if I remember correctly, it didn't do any better, or it might not have done as well on answering questions.

SPEAKER_01 24:33

So Yeah, I don't I think the assurance that the kind of like a notebook LM would provide only from the strict data source, or what um I believe is called let's see, the small language more of a small language model where you've vetted the sources in theory, anyway, should lower the hallucination rate, which I think is, as you might imagine in the medical field going to be a huge concern. And that may be why that's picked up. And then of course it's also been targeted to medical professionals. So if you took, say, Chat GBT as it is now, we disguised it, we put a sticker on it, and it is now called Psych GBT, and we say, hey, this is top-notch psychological knowledge, gave it to psychologists. Do you think that there would be some self-fulfilling prophecy in the view of, well, this is for psychologists, so of course this is the most accurate.

SPEAKER_00 25:29

Yeah, I mean that's the thought is that fine-tuned models would outperform. But I'm looking at this article in Nature, and uh what they found was that general purpose large language models outperformed specialized clinical AI tools on medical benchmarks. And I'll share this. Maybe we can put this in the show notes. Pretty interesting.

SPEAKER_01 25:50

Um So if you say that, Adam, then why the heck are we talking about report writers? We've got them all, and they're only 20 bucks a month.

SPEAKER_00 25:59

Well, the problem with that is are they HIPAA compliant? Are they FERPA compliant?

SPEAKER_01 26:06

Yeah.

SPEAKER_00 26:07

And we don't we don't know that they do better, but that is research that should happen.

SPEAKER_01 26:11

Yeah, I agree. You know, when we first started this particular episode, we were worried about, you know, would we be able to have enough to get through and stuff? And I don't know if we've gotten through our first point yet.

SPEAKER_00 26:25

Well, it's because it's because we're being tangential.

SPEAKER_01 26:30

We are, we are, and hopefully, hopefully the listeners are enjoying the tangential part of it. You know, there's a lot of formalized trainings that you and I have both done that are available online, and then there's also a ton out there by other people. I think it's these tangential things. I think this is the stuff, the discussions that I know in my presentations, I just am not able to get at because it is under a quick time limit, and that we're not able to go down some of these rabbit holes about discussing the state of the field because a lot of people are still getting up to speed or even being even introduced to the area. And so I think that, you know, for this podcast, I think the focus on uh working through kind of some of these issues as we're seeing them, and I can't promise that we're going to give our listeners answers to all these things. I think a lot of this is still up in the air, but one has to ask the questions before you can answer anything. And so seeing where there are holes in the way that we practice and how people are trying to address them and whether those forms are being successful or not, I think is at least I feel like it's worth the time to put in with that. Uh, even if it is we have, even if we get an audience of 10 core people and the rest are like, these guys are just saying everything that we don't understand.

SPEAKER_00 27:50

And I I will I do want to say that this study was kind of like answering medical questions like you would for, you know, a a test. And so that's a little different than performing actual tasks. It's just knowledge, access to information, essentially. It's not, you know, writing ability, it's not ability to code, things like that, because there are kind of more fine-tuned models like cursor, for example, that that does very well due to the post-training that occurs for coding. And actually does it uh in a much more efficient way as far as compute and stuff. So I didn't mean to say like, you know, just go ahead and use a large language model for everything. But you should whenever you're I think you're using a tool, you should say, does this do a better job than a large language model that I can get for free? You know.

SPEAKER_01 28:52

So basically a general purpose LLM is your baseline. Like you have to do minimally as well as one if you're going to claim some specific specialty for a new AI that's out there. With, of course, the caveat, and we keep coming back to it, but it is so portent in healthcare with being FERPA and HIPAA compliant. And I think that's the one that trips up most of these general purpose AI models. So Adam and I, by the way, are not endorsing anyone using and writing their reports using just a general purpose LLM, especially a free model. You definitely want to have some assurance that the correct procedures have been put in place, that that data stays where it is and it remains your property and is not disclosed to anybody that is not supposed to have access to it.

SPEAKER_00 29:45

Yeah, and that's even if it is redacted.

SPEAKER_01 29:48

Right. Because AI is a bit smart, right? It can figure out who you're talking about.

unknown 29:55

Yeah.

SPEAKER_00 29:55

And there are models that are just general, they're just Chat GPT wrappers that are HIPAA compliant. And that has value too. I'm not, I'm not, you know, disparaging those companies. There's value in that. Definitely. But like you said, the baseline should be, well, first of all, maybe we're getting ahead of ourselves, but I always want to know what does a company provide? You know, like what is what exactly is their product? And if it is just HIPAA compliance, okay, that there's value there for sure. And then, yeah, that's the baseline. Is is this product better than you know, Gemini with a with a BAA in place that's HIPAA compliant?

SPEAKER_01 30:39

Well, I mean, there's my next business idea, Adam. I mean, I just figure out how to get one of these HIPAA compliant and I do absolutely nothing else. And I charge $2 over what it costs me. And voila, you know, I've got a decent product. It's got the compliance, and it's continually updated for whatever newest model. So, you know, I don't have the time, frankly, to do that. But if any of our enterprising listeners want to do that, something like that, there's a real quick and easy idea uh on how to enter the market. And then you just have to have your marketing expertise to convince everybody else that it's an excellent report writer.

SPEAKER_00 31:14

Well, I mean, those HIPAA-compliant bastion exists, and it's it's a it's just a wrapper that for Claude or Chat GPT. So someone's already there.

SPEAKER_01 31:26

I know. It was, and that was one of the first ones that was out of the gate, too. It's been around for a few years, and you know, we've heard people who have used it. So, um, and you know, I've not really heard bad things about it. I've also not heard people screaming on the mountaintop saying this is the next, you know, this is the best thing ever. So but just one of many products out there, and and we're not endorsing any particular ones, just to try to make people a little aware because each report writer, even though they call them report writers, how they get to the report writing is varied, and that can influence the results. But I don't think we'll have enough time in our episode to get there, but we will be continuing our discussion about AI report writers and any other things that pop in our head as we gradually work through some of the things that you want to be thinking about with report writing. So join us for our next episode in Beyond the Bot, AI and Mental Health. And we will be back with you in a couple weeks, uh, probably talking more about report writing. See you then.