Episode #256 – Why Jury Research Needs Validity and Reliability – Part 1 of 2

Bill Kanasky, Jr., Ph.D. describes the scientific concepts of validity and reliability in research and why they are individually and collectively so important when conducting jury research. The question that validity helps answer is: are you measuring what you think you are measuring? Bill gives examples of how you can limit or improve your validity through witnesses and presentations in jury research. It’s critically important to secure a clean read in your research and Bill explains how to achieve that. 

Reliability in research refers to the consistency and repeatability of a measurement, so that if the same process is repeated under the same conditions, it should yield similar results. Having reliability in your data means you can count on the results and increases confidence in the findings to better guide decision-making on your case.

Full Episode Transcript

[00:15] Bill Welcome to another edition of the Litigation Psychology Podcast brought to you by Courtroom Sciences. Dr. Bill Kanasky coming at you. I’ve got an injury update, folks. I have, I think I have a heel spur. Oh, I mean, good god. I mean, this thing is absolutely It is It is killing me. Absolutely killing me. So, I can barely walk. Uh, and I’ve got the standing desk, so I’m I’m kind of standing in pain here. Got to take a little pressure off the foot. Give it a rest. God, getting old sucks. Getting old sucks. But because of this audience and how much I love you, we’re going to uh we’re going to plow forward here. Had ice packs all night, ice packs all morning. You have to give it a rest, folks. You know, I I I tell you, I’m a big health nut. You got to stay active, but man, you’re going to have nicks. You’re going to get nicked up. You’re going to get nicked up, and then you got to back off a little bit. So, you don’t end up in the hospital. Yeah, we want to keep you out of the hospital.

All right. What are we going to talk about today? This is really uh important. We’re going to talk about jury research and some scientific concepts that no one freaking understands. Actually, let me use my soundboard for that. Let’s let’s let’s do it this way. Okay, let me repeat that. Right. We’re going to go over some scientific concepts that nobody [bleep] understands. God, that there’s a delay there. I’m not liking that. I am not liking that at all.

[02:03] Bill Okay. So, two concepts that you need to think about when it comes to jury research. And listen, attorneys, whether you’re a defense attorney, plaintiff attorney, uh, insurance claims people, most 99% of you have have no training or background in what I’m about to tell you. I mean, unless you took a research methodology class, right? Um, Dr. Steve Wood and I have advanced training in this. This is where a lot of your jury research can go wrong. Again, whether you’re plaintiff or defendant, your your jury research can go wrong if you’re not doing it the right way and you can misinterpret the results.

So, the two things that we’re going to talk about are validity and reliability. Validity, validity and reliability are the keys to everything in any research. Medical research, you name it, engineering research, jury research. Okay, let’s start with validity. Okay, so what what is the definition of validity? Okay. Validity. Validity is are you measuring what you think you’re measuring? Okay. Are you measuring what you think you are measuring? Do you have valid results? Okay. So, this is directly tied to what are you putting in your presentations? What are you putting in? So, for example, if you’re doing a uh mock trial to prepare for a trial, say you got a trial in a couple months and you do the mock trial and you don’t put in any video deposition clips of witnesses. You just kind of skip that part. You want it to be, you know, down and dirty or whatever. And you you don’t put any of that in. And you just do your your your clopening, which by the way, I’m gonna I’m gonna blow that out of the water later today. In fact, let me write that down. Clopenings suck. Not only do they suck, they [bleep] suck, right? Got to love this soundboard.

[04:32] Bill Okay, so let let’s back up. So, validity, right? So, say you’re doing your mock trial, you don’t put any of the wit—Okay, so we know, we know that the quality of the witness testimony is a massive predictor of outcome when it comes to particularly liability issues, damages to an extent, but liability issues. Absolutely. And you don’t put that in your mock trial. You just have your attorneys up there kind of putting on a summary of the case. What I would argue that you have a validity problem. You’re not measuring what you think you’re measuring because you’ve left the key component out. All right? You’ve left a key component out. That’s why when you’re in trial prep mode and you’re doing research for trial prep, okay, that really needs to be in there. That really to—to maximize your validity. So, you’re measuring what you think you’re measuring.

Now, here’s your problem. Not all depositions are videotaped. So, what do you do now? Well, you have a couple options. Number one, you summarize the testimony. Okay? Kind of like you do at trial. What would you do at trial if a witness wasn’t available? You’d read in the transcript, which is like, you know, shoot yourself in the face if you’re a juror. That’s terrible. But that’s what happens at trial. And that’s what you’d want to do during your research to increase your validity. Summarize it, put it in PowerPoint, maybe take some excerpts, right? Blow some of the answer Q&A, right? Blow some of that up. And you’re going to have to read it in. Okay? Limits your validity. Okay, but it doesn’t torpedo it. Doesn’t torpedo it.

[06:28] Bill Now, that being said, the best way to do this is to make a new video. Okay? So, for example, you’re the defense and you have three key witnesses. They’ve been deposed. However, none of them, none of the deps were videotaped. Okay? You bring them to your office. You videotape some direct and cross and there you go. There. And then you you play that to your mock jury because what that’s going to do is going to increase your validity. All right. Witness testimony is really, really important to jury decision-making. And yes, you have to have a good story. Talked about that. But the actors, okay, the people in the story are the witnesses. So, getting a read on their likability, their trustworthiness, their competence, their knowledge, and their overall credibility is key because you may have a good story, but your witness sucks. They suck. You’re going to have problems.

Now, here’s the key why validity is so important. If you just tested the story without the witnesses, your story may be great. You’re going to get okay, you’re going to get a false positive. That’s your problem. False positive because all the mock jurors are going to be like, “Yeah, we love your story.” Whereas, if you would have done it with the witness testimony, they’re going to come back and say, “Hey, your story’s okay, but boy, your witness sucks.” And and hence your problem. Okay? So, you got to do whatever you can to build that validity up. And a big part of that is putting in the witness testimony. Very, very important. Okay. Are you measuring what you think you’re measuring?

[08:32] Bill All right. Now, that being said, you don’t have to—remember jury—if you have a two-week trial, jurors don’t understand every single thing you tell them. They don’t they don’t remember everything. I mean, they try to take notes and stuff like that. So, do you have to put on your entire case in your mock jury to get validity? No. No.

So, let me tell you a story. So, I parachuted in on a case. They had recently done a mock trial, okay? And it was a long day, long presentations. They had they did have witness clips of videos. They were like an hour long a piece. Okay. However, validity validity. One of the presentations which happened to be the plaintiff presentation was fantastic. And then this attorney recruited one of their colleagues to to to be the defense attorney in the mock jury. Obviously not going to tell you who it was. Just let me tell you what this presentation was terrible. I mean it was it was bad. It was really I mean it was really really bad. Yeah, that’s that’s exactly what I saw. So, the plaintiff put—Right. So, the the plaintiff attorney who’s a defense attorney, right? This person puts on a solid presentation, summary presentation of the case and it lasted about 45 minutes. The person who was recruited for the defense, they really didn’t know the case. You could tell their total presentation was like 12 minutes.

[10:35] Bill Validity. You can’t get validity if you have one of the presentations being shitty. Okay. Validity. Are you measuring what you think you’re measuring? So, this project, which I’m guessing cost about $80,000, didn’t give them valid results. Because they’re not measuring with it because the defense presentation stunk. Now, I’ve seen it the other way where one of the defense attorneys puts on the plaintiff case and it stinks and the defense case is great and then you get all this positive feedback. It’s like okay so one another one of the ways to improve validity is by abandoning this is going to be very controversial by the way but I do believe in it abandoning the mock trial format altogether it’s unnecessary the only reason why it’s done is because it’s more traditionalist okay I mean how many teams do you see how many football teams still run the wishbone offense? What? Army, Navy, and Air Force. The the armed forces teams outside of that. Okay. Oklahoma’s not run the wishbone anymore. Nebraska’s not running the wishbone anymore. Michigan’s not running the wishbone anymore. Why things have evolved?

The focus group model eliminates this concern. Because this is what you want to do in any mock jury. Now, everybody follow me. In every mock jury. The last thing you want in the mock jury is persuasion. Okay, let me say that again. The last thing you want in a mock jury is persuasion because you can never balance it. It skews the data. Screws up validity. Okay. And we don’t we don’t want that. I object, your honor. Oh, see, I knew somebody was going to be upset about this.

[12:48] Bill So, you can have in your mock jury research to increase validity, have one person present the case covering both sides. That eliminates persuasion. Okay? Imagine you’re doing your mock jury. Imagine you’re a professor in law school counsel and you’re in front of your law school class and you’re going to educate the class on a case and what the themes and theories and angles are on each side. That’s how you want to do your mock jury. Take the persuasion part out of it because we want to increase your vote. I want a clean that’s what I call a clean read. A clean read that’s not contaminated by persuasion, right? Before they take a blood test, right, they they they prick your finger. What do they do right before they prick your finger? They clean it off with an alcohol swab. Why? They want a clean read. They don’t want it contaminated. So, this is very very important to validity.

Now, every case has various challenges. Every case has various challenges. Sometimes you have the witnesses on video. Sometimes you’re—right. You got to make adjustments. I can help you with that. But here’s the thing. You want to maximize your validity to the highest level possible and practical. One of those ways is switching over to the focus group model which takes persuasion out of it. Okay. Now, can you have multiple presenters in a focus group and maintain high validity? Yes. But each presenter needs to cover both sides of the issue. Once you have, I’m just telling you right now and and sometimes we do this because you know nobody listens. Okay? Nobody listens and we get stuck in that model.

[15:02] Bill And then for each issue, you have one attorney doing one thing, one attorney doing the other. And the problem is what if one of the attorneys is better looking than the other one, what if one of the attorneys is more uh you know, um talkative uh uh comes across better, screws up the it screws up the data, the validity. Okay. And then you’re arguing, you know, one one attorney’s passionate, the other one’s more educational. Screws up the data. Have one person covering both sides of the issue. That ensures ensures your clean read without contamination from persuasion. Boy, nobody’s going to like this podcast because I’m going against the grain here. Going against the grain here because a lot of traditional jury research really lacked the scientific rigor. Like when I came into this, it kind of lacked the scientific rigor at times. Then you get results that are invalid. You don’t know it. And you start making big time, hey, we’re going to take this one to trial. Then you get you get smoked. We don’t want you getting smoked. We want validity so you’re making the right decisions.

[16:32] Bill Okay. Now, another challenge with validity happens all the time. I got a call yesterday. We need to test this case. We don’t know if the judge is going to let X, Y, and Z in or not. So, we don’t know how to do our like do we put it in? Do we do it? Okay. Test. Retest. You do two smaller projects, one with it in, one with it out, then you’ll get validity in both of those circumstances.

Okay? Now, here’s the mistake. This is what this is what you all do. This is I gotta I gotta go to soundboard for this one. This is what you all do. You want to test it the one way and at the end of that project you want to say, “Well, what if I told you X, Y, and Z or A, B, and Z or X, Y, and Z. What if I told you this? How would that have changed your opinion?” No, I have to go right to the soundboard on this one. That doesn’t—That doesn’t work. That’s not—You can’t do that. It’s not science. Okay. You can’t put the toothpaste back in the tube and then have them reassess that situation. You got to you got to test retest the two the two scenarios.

[17:50] Bill Okay, here’s the other good news because you may have a lot of witnesses. Okay, you may have a lot of we’ve tested this. You can show a 10-minute clip of a witness in one project and then in a different project show an hour clip of that witness and the results will largely come back the same from jurors. don’t need an hour to start making judgments about. You have in fact if yours is 5 to 10 minutes per witness clip, you’ll get all you need to know from from jurors. Okay? Now, there may be some exceptions where you up that to 20 minutes, say it’s your corporate rep or something like that depending on what kind of case you have, but you don’t need to be going playing an hour or 90 minutes of a witness to get a really good read from jurors. Okay? So, we’ve showed that you can get validity. You can increase validity with shorter clips.

Okay, so back to this other mock trial, this uh case we parachuted on. So, we did a follow-up focus group to their mock trial because they they butchered their mock trial. Okay, we cut the project from eight hours to four hours. We took all the hour videos. Okay, the hour videos we made 10 minutes each. So now I have the mock trial data on what they thought of the witnesses. And now I have my focus group data and what they thought of the witnesses. Mock trial was hour clips, focus group 10-minute clips. Guess what the results were relating to the ratings of the witnesses? They were identical. They were identical. Which tells you to gain validity, you don’t have to show these long clips. You don’t have to do mock jury’s research all freaking day and spend $100,000. Okay? Validity. Are you measuring what you think you’re measuring?

[19:47] Bill All right, that’s enough on validity, right? You happy with that? Are you happy with the validity talk there? You got—Oh, wow. Thank you. Thank you so much. Let’s talk about the other uh issue here. The other issue, the other issue is reliability. This is key. I think much has been lost here and much can be gained. Okay? And I’m going to say it right now. Everybody’s going to get mad at me. Hats off to the plaintiffs’ bar. Because the plaintiffs’ bar has figured out that validity is not enough. You need validity and reliability.

So, how do we define reliability? Reliability means is that the results you’re getting are reliable. They’re consistent over two time points. Back to test retest. Okay. If you test something once, you there is an there is statistical odds that you have an outlier. You test it twice, you get similar results. Now, now you really know. Cuz here’s the thing. You don’t want to know. You want to really know.

[21:11] Bill Okay? So, I don’t have time to do this. I can’t do it’s too expensive. No, no, no, no. You could do shorter projects. You can adopt the focus script model, and you can test these things a week apart. Make some tweaks to the first one and then retest and see how reliable. Okay? Because on the second time around, it may crash and burn. Aha. Now you know you have a problem that otherwise you would have never known about because maybe you had that outlier. Maybe you had that outlier in the first project. So again, the whole I don’t want to have to do this twice. It’s going to double the cost. No, you can do more streamlined focus groups which are actually under well under the cost of your traditional standard in-person all day project. You do it virtually. It’s exactly what the plaintiffs’ bar, this is exactly what the plaintiffs’ bar does and they do it well. They do serial focus group testing both early in the case and then in trial prep because they want to repeat the test to make sure of two things.

Number one, they’re getting consistent results. So that they really really know what’s working, what’s not working. And then number two, if they tweak something, they want to know is this making things better, is this making things worse.

[22:39] Bill Plaintiff attorneys figured this out a while ago. That’s what they do and they’re doing a good job of it. There’s nothing. Why in the world would the defense not want to do the same thing? So, every time I’m talking to clients now, I’m really pushing hard on rather than doing this all day or day and a half or two days prior, there’s a time and a place for those. But in most of these cases, you don’t need that. You adopt the focus group model, you’re going to get higher validity. You’re going to take a deeper dive. You have more juror interaction. Okay? You’re going to truncate it because it doesn’t need to be all day in most of these cases. And then what you’re going to do is you’re going to tweak and repeat. Tweak and repeat and see how reliable your results are. Okay? Sometimes you need to do it a third time. Guys, we’re talking four hours a pop with mock jurors on Zoom. This works. We’re doing a ton of this now. It works. It works. Now, most of these cases, if you do it twice, I think you’re good. Some of these cases, you kind of step on a landmine in project number two. It’s like, oh boy. So, then you know you do a third one. That right there should be very sufficient.

[24:04] Bill Okay. But it’s testing over a time period and getting multiple data points rather than just relying on one. See, the old way is to do your jury research, and you make all these massive decisions based on one time point. Well, you may or may not have validity, but you certainly don’t have reliability. By definition, you cannot attain scientific reliability without repeat testing. You can’t do it.

So, if you want to make the best decisions, whether you’re a plaintiff, defense, whoever, focus on maximizing your validity, but then don’t forget about the I think people forget about the reliability aspect of this. Okay? I think that’s really important. Doesn’t get talked about enough. But you know, you have cases here. You have cases here in which I mean this is a no-brainer particularly when you’re not sure if something’s going to be in or out, right? Or you have a motion in limine. This is coming in coming out. Well, you do two projects. You keep everything the same and that’s your manipulation factor. Okay? It’s one of your independent variables. and you test it and you figure out, okay, when this when this evidence comes in either, oh god, this blows up my case or it’s a thought, it’s a nothing burger. What if it’s a nothing? Well, you see, you don’t know. Okay, you don’t know. And again, you don’t want to just know, you want to really know.

[26:08] Bill All right. So that’s the pod, that’s the podcast for today. Want to cover very two these are very very important topics. Don’t get enough attention. We are constantly getting very creative with our jury research here at Courtroom Sciences to explain this to everybody to say we want to maximize your validity. But let’s not let’s not forget about uh reliability.

Okay. Now I’m going to go on my rant for today. And I may have maybe I’ve ranted this before in some way, shape, or form about these dishes in the house. My kids—listen I love like I love doing dishes. You want to know why I love doing by hand. I like washing dishes by hand. And you all think I’m crazy? Why? Because I can’t use my phone or my laptop. It for this forces me to do something without a electronic device in my hand. It’s great. Calms the mind. But here’s the issue. When one of these kids, the worst thing that can happen in this house, they put they they put the dishes in the sink, they don’t wash them off. That’s bad enough. The worst thing in the world you can do that with is egg yolk. And my kids love fried eggs. And it’s throw, just throw the dish in the sink.

[27:28] Bill And then I come in two hours later go, “Hey, I’m going to wash me some dishes, take a break from email, take a break from LinkedIn trolls.” And I looked down and it’s like it’s it’s like super it’s like uh super glue. It’s like uh gorilla. You see the that’s the new stuff they got now. The gorilla glue, the gorilla tape. It’s like somebody has dumped Gorilla Glue all over a plate. And I’m getting the rough side of the the the sponge, right? The scrubber. Trying to scrub. No, it’s not coming off. I have I have to like get a hammer and chisel to clean my own dishes. I’ve had two family conferences about this. Two. Jeez. Boy, I feel a little bit better. See, I got my rant out. I got my rant out.

So, tell your kids, hey, it could be your husband. It could be your wife. If you’re just gonna, you know, if you’re going to just throw dish, I’d prefer you just wash the damn dishes yourself, right? Or rinse them off, put them in a dishwasher. But if you have egg yolk on a plate, you can’t just throw that in there. It turns to concrete. Gosh. So, there you have it, folks. Validity, reliability, and egg yolks. No other no other podcast in the world. Joe Rogan’s not talking about any of this stuff. No, you get that here right on the Litigation Psychology Podcast brought to you by Courtroom Sciences. We’ll see you next time.