Episode #288 – The Science of Jury Research: Why Mock Trials Aren’t Enough

In this episode, Bill Kanasky, Jr., Ph.D. explains why most defense teams misuse jury research by relying solely on a single mock trial and skipping the exploratory phase required by the scientific method for validity and reliability. Bill breaks down how early focus groups are critical in revealing juror confusion, hidden vulnerabilities in your case, and dangerous misconceptions that mock trials are unable to uncover. He also emphasizes that early exploratory research can shape discovery, expert strategy, themes, and voir dire long before mediation or trial.

Bill warns that when defense teams skip this exploratory step, they enter mediation and trial preparation with major blind spots and lacking data while the plaintiff’s side often has extensive exploratory data and ammunition, which is particularly impactful with mediators. Bill closes by urging defense counsel to adopt a disciplined, phased research process that begins early with exploratory focus groups to reduce risk and improve litigation outcomes.

Full Episode Transcript

[00:15] Bill Welcome to another edition of the Litigation Psychology podcast brought to you by Courtroom Sciences. Dr. Bill Kanasky coming at you. How goes it, folks? God, we’re approaching the end of the year like at rapid speed. Gee, flies by. God, it flies by. But in my experience, the fall is the worst. Oh, crazy, crazy, crazy, crazy. Everybody trying to fit things in before the holidays. People losing their minds. Yikes. But we got to stay positive, right? We got to stay positive as we push towards the holidays. Uh let’s talk a little jury research.

And um you know, we’ve we’ve talked a lot about the importance of the scientific method, right? And let’s kind of cover this again big picture because again many of you are new podcast listeners and you haven’t listened to previous episode so I I can’t just jump right in. I am hammering down the coffee today. Geez. Okay. So, if you if you’re doing research in any type of research, but you’re talking about jury research on your case, the process goes as follows. This is the scientific method, which everybody skips and ignores. And we’re trying to make a difference here to get you back on track so you do things the right way. Okay. Number one, it’s a multiple step process.

[02:06] Bill I can’t tell you how many times we just get phone calls like, “Hey, we got a trial in 60 days. We want to do one project. We want it to be a mock trial and that’s what we’re going to do to prepare for. trial” That’s not the way to do this, folks. And I know that’s the way you’ve been doing it forever. I’ve been screaming at everybody. That’s not the right way to do it. But you’re going to the fi—you’re skipping the previous steps in the previous steps which are the focus group model. Okay, those are the explor—so you have two phases of research. You have exploratory, you have confirmatory.

Exploratory is to explore. It’s not to persuade. It’s not to argue. It’s not to test your strategies. It’s to explore. You’re going to run the facts of the case, evidence, deposition clips, whatever. You’re going to run these by mock jurors and explore how they perceive your case. You do that neutrally. You don’t want persuasion in your exploratory. No, no, no, no, no. And you’re going to explore. Okay? You will be astounded with what you find in the focus group process because it’s highly interactive. You’re going to get a ton of questions. You’re going to figure out juror comprehension issues. You’re going to figure out where they’re connecting dots where they shouldn’t be. And you’re going to save yourself a lot of headaches because you’re going to go, “Wow, okay. And ideally, you want to do at least two of those.”

[03:54] Bill Okay, that’s the exploratory phase. You’re going to get through all the noise. You’re going to see where all the misconceptions are, your juror comprehension, and you’re going to have town hall meeting, deep dive discussions with jurors about the case. What do you think about this? What do you think about that? Okay? And you’re going to work through it. That’s the whole point to that because you’re going to find your hidden problems, vulnerabilities, misconceptions in that process. Ball don’t lie. Focus groups don’t lie. They never lie. Ever. Ever. Ever. Ever. Ever. Okay. The problem is everybody skips this step. You just jump right to the mock trial and then all that stuff we just talked about that you don’t know about is going to get dragged into your mock trial. You’re going to have one test at one time period and your data is going to suck because you skip the first two steps. You skip the exploratory part of this.

Things like theme development. Where do you think that comes from? Doesn’t come from a mock trial. I’ll tell you that. Mock trial. You have two attorneys going at it, arguing all day, testing strategies. Well, where’s that coming from? Think about that. Okay, that should come from exploratory testing, right? Where you’ve refined it, you’ve tweaked it, you know what works, you know what doesn’t work, you know what’s confusing to jurors, you know what’s not confusing, you know what ticks them off, you know, and then then you go to your mock trial and then that’s confirmatory testing. That’s the scientific method. That’s the way it should go.

[05:57] Bill But everybody calls, “I need a mock trial.” They skip the exploratory part. Just skip it. And then what ends up happening is you do the mock trial, and you end up with more questions than answers. Okay, that’s to me—well, number one, you’re going to have a bunch of validity and reliability problems. It’s a big waste of time and money. Big waste of time and money to get data that’s iffy at best. Right. So, the much better method is to do this the right way is number one: start early. Number two: explore. Take a deep dive in these issues in more that interactive focus group town hall meeting format. What you’re going to learn is astounding.

Now to make this point let’s go right to the plaintiff’s bar. I’m going to share my screen. I’m gonna try not to screw this up. And here’s a video from a plaintiff’s—uh, see, I screwed this up already. I screwed this up already. Hold on. Let’s try this again. This is from a plaintiff’s podcast where they’re talking about this exact issue. All right, let me let me let me let me try this. Boy, I really suck at this. This is—this is too much going on. Okay, let’s—so listen. This is—uh these are two plaintiff attorneys talking about focus groups and early and and and and why you want to do this the exploratory way first.

[07:41] Video “I mean, there are cases where young moms are just death.”

“It’s always the least, you know, you might think in a kid case, right? Gosh, you know, the young moms are going to be right on my side and and then they’re all against you and you’re like, what? Or like with sexual harassment, women are terrible on a jury if you have a female victim. Um, you know, just little things like that. But you only know those things if you do a bunch of focus groups and develop that pattern.”

“That’s right. I mean, and and and until you do that, you don’t—you just don’t know. Um, and and worse than that, you know, if you’re—if you have the tendencies like I do, um, which I fight against constantly, you probably think you know. But you don’t know, which is even more dangerous. Um and so um you know that that’s that’s that’s key. Um and so you know every focus group you do on a case is good for identifying the people who are not going to respond favorably to your case. And why not start early? Um why not start on that uh from the very beginning.”

[08:47] Bill There you have it. All right. Let me stop this share. All right. So, there—there you have it. Sorry, I gotta close that. Okay. So, let’s—let’s think about what he just said. Change my screen around. There we go. All right. So, he says you think you know, but you don’t know. That’s the most dangerous part. That’s exactly what he said. The other thing he said I thought was very curious is, “you know I have these tendencies. I fight these tendencies.” That’s the confirmation bias we talk about. So, the plaintiffs’ bar has figured this out. They look at their case and their brain goes right to confirmation bias and they are aware of that—very, very dangerous. It’s dangerous for them. That’s why they focus group early because the focus group’s going to give you—it’s going to tell you things that you would have never come up with because they’re hidden.

[10:05] Bill Okay, defense. Same thing. There’s no reason why you can’t do this stuff early. It’s—it’s efficient. It’s effective. It’s affordable. Okay? And I think that’s the most dangerous thing. You think you know, but you really don’t know. Imagine the multi-million dollar mistakes that can be made in that formula. Confirmation bias is not just dangerous strategically, it’s—it’s dangerous economically, right? So, in these exploratory focus groups, which totally can be done virtually, not a problem with that.

Okay, here’s something I want to focus on today, which again doesn’t get a lot of attention because defense—until you do this as the defense attorney, it’s like really hard to get your head around because I talk to new defense attorneys. They’re like, “I want to do a focus group.” And I tell them what we’re doing and they like really struggle. They’re like, “Well, I’m not—going to wait, don’t I put on a clopening?” No. God, no. That’s dumb. Clopenings don’t work. Juror cognitive overload. Okay, we want to explore. “Well, don’t I need to—we have two attorneys arguing?” No, no, no, no. We need—you need to like take law school professor mode and we’re doing a case study. Okay, you’re a narrator. You’re a professor. You’re a teacher. You’re neutral. And you’re gonna explain what happened in this case to the mock jurors.

[11:38] Bill Okay? Show exhibits, show video, show pictures, go over the who’s who, what the claims are, deep dive, a lot of handholding, and be neutral. Now, what are you going to get from mock jurors? I think this is the most important part—does not get enough attention. And here it is. The jurors’ questions and curiosities are just as important as their comments. See, everybody goes into this like, “I want to hear what jurors think about the case.” Yes, of course, absolutely. But where there is a lot of hidden value is in the questions and curiosities that they have—stuff that makes your head spin. You go, “Wow, didn’t think of that. I—I didn’t think anybody would come up with that.”

Okay, but that’s important because that’s what jurors do when they’re in the jury box. The same thing happens. But now you’re doing it now well before the trial and you’re figuring out what’s going on in these jurors’ brains. And it’s not just how are they reacting to things. Again, really, really important, but where are their curiosities? Where are their questions? Where’s their confusion? Okay, that’s where the gold is at. Plaintiff bar, they figured this out. That’s where they start. They do focus groups on specific issues. They break up the case and the issues. They rarely test the entire case. Don’t need to, particularly early on. You just look at liability. You haven’t touched damages yet. Not really necessary. Or you touch it at a high level. Okay? But take a deep dive. Get that conversation going by asking jurors, “What are your questions? What are your curiosities?”

[13:44] Bill The stuff that you’re going to hear is going to be very, very telling. It’s going to be very, very telling. Okay. Now, the other benefit of doing it this way, right, that we—we already talked about, but I’m going to reinforce this is you’re gonna—when you figure out kind of where these juror comprehension issues are, these misconceptions, these concerns, okay? It’s going to—it’s going to impact how you see the case and your theme development, in your strategy. Now, think about the attorneys that just jump right to the mock trial and they skip this step. You’re dragging a bunch of junk and error and misconception into your mock trial and you don’t even know it.

And by the way, mock trials aren’t really designed to find this stuff because you’re testing strategies against each other. So, if you get to the mock trial, you want to be testing a strategy that’s been vetted, that’s clean, that’s gone through this exploratory process, so you’re testing the right things. You’re not dragging all this error in with you. But nobody does that because why? Nobody listens. It’s not your fault. This is kind of the way the industry was kind of formed. And again, you know, typical insurance defense, you know, posture is, “Oh boy, case didn’t settle. Now we need to start doing our homework. Let’s do a mock trial.” Like that’s just what everybody does because they don’t know any better.

[15:33] Bill So, what we’re trying to do here at Courtroom Sciences and the Litigation Psychology Podcast is to tell you there’s a much better way to do this, which is going to save you a lot of headaches. Okay. Explore early. Now, how early? Yeah, we’re—we’re doing these fairly often pre-suit. Now, that makes your head spin, right? “How am I going to focus group a case that—there is no case?” Oh, it’s coming. And you know it’s coming. But you may have that dash cam video. May have that police report. Yeah, you got that. We have police bodycam footage, post-accident pictures, surveillance video, whatever. You got plenty. Trust me, you got plenty. And start to get a read on this stuff early and see how jurors are receiving the information, how are they processing information, where there’s comprehension issues, where are there concerns, what questions do they have?

Really, really important stuff that can—that you can do before the case is even filed. That’s very aggressive, but we have clients that they’re aggressive defendants. Remember, knowledge is power, but early knowledge is the best power. Okay, now here’s the key. It’s the repeated and serial testing through this exploratory process. So, as more information comes in through discovery, you can keep retesting. These are not long—these are not like—don’t think like these are all day or two day. No, these are two to four hour pops in front of mock jurors saying, “Okay.” Right?

[17:23] Bill So, these pre-suit ones typically two hours. That’s all you need. Get that kind of foundational read as more information starts to come in, depositions, right? You can retest and then you go four hours and start introducing more information and then seeing how—again, see—explore how jurors are reacting to that. See where their questions are, to their curiosities. Through that process, you’re going to get through all the noise. You’re going to see where all the mistakes are, where all the comprehension and misconception pro—problems are. Okay? And now you’re going to have some weaponry because now in those subsequent tests, then you can start introducing some damages stuff because you’ve got some more information. Okay? Now, say you do this two to three times. You should have a really good idea of where you’re at on liability, say apportionment of fault, right? You should have a decent idea on damages, right? From an intensity perspective on where you are and say now you’re going into mediation.

I gave a speech a couple weeks ago and after I spoke on jury research, the speaker after me was a mediator. He got up and the first thing out of his mouth was, “I agree with everything Bill just said. I’m a mediator. And when I see that you’ve done your homework on a case, you’ve done some focus groups and you—that helps your position and it helps me talk to the other side.” It’s the first thing he said. Second thing he said was if the plaintiff has done their homework and you haven’t, it makes a massive difference in the mediation process. That’s from—that’s from the mediator.

[19:14] Bill Okay? So, you take this material in and you—you—you talk to the mediator going, “Hey, we did our homework. We have a good idea of, you know, where—where we’re at. We—we’re not—we’re not dummies here. There’s no confirmation bias here. We know what our weaknesses are. We know where our strengths are too. We really got a good idea on damages too.” Okay, massive difference in the leverage that you would have at mediation. Okay, and that’s the goal. You want to get these things resolved, right? So, you go through that exploratory process. Okay, let’s say mediation fails and now you’re more in a pre-trial mode. Okay, so now you’re ahead of the game.

Now, let’s look at the old way of doing this, right? So, plaintiff does all this exploratory stuff, get to mediation, it fails. They go into pre-trial mode, and they’ve done all this homework. So, like the scoreboard’s 28 to nothing and you’re playing catchup because that’s typically when the defense goes, “Oh boy, okay, now we need to get our [ __ ] together.” It’s too late. Okay. If you do the exploratory stuff and say they do too, which is most likely the case, mediation fails. Now the score is tied going into pre-trial and now you’re ready for your mock trial.

[20:43] Bill Okay, you ran everything through the exploratory process. So now what you put on in your mock trial, your validity, your reliability, and your accuracy, it’s going to be 10 times higher versus if you skip all the exploratory process, you jump right to that mock, you’re dragging in all of this error, uncertainty, hidden vulnerabilities that you don’t even know about and you still may not know about after the mock. Okay, there’s a better way to do this.

Now, on top of all that, here’s the real key, right? Let’s say you get through this process. You do the exploratory stuff. Mediation fails. You do your mock and now you’ve done it the right way. So, you have valid, reliable results. Now what you have is you have a database of juror data that we can look at and statistically predict what types of people are going to be pro-plaintiff, what types of people are going to be high damages, what types of people are going to be pro-defense, what types of people are going to be low damages. So, if you have to go to trial, you’re fully armed with a data-driven approach for jury selection and when you’re coming up with your themes, opening statement, you’re—you—you know, you know.

[22:15] Bill Now let’s go back to the video. You think you know, but you don’t know. That’s where these problems are occurring. So, if you follow the scientific method the way it’s supposed to go with exploratory first followed by confirmatory, the benefits here, if you do end up in a courtroom, you’re covered. Now, remember, we’ve talked about certainty. There are things you cannot control. Can’t control a judge’s rulings, right? Can’t control the jury venire that shows up that particular day. Can’t control that. Expert testimony that could blow up in your face. You don’t have a whole lot of control over that. Plaintiff witnesses. Yeah. Yeah. They may shine. Yeah. You don’t have control over that. Right. Bunch of things you don’t have control over.

So, what are we really doing here? This is a—a battle of probabilities, right? It’s a battle of probabilities. Like, you can go to Vegas, and you could be great at blackjack and you can—even if you—if you knew how to count cards, if you counted cards in Vegas, you’re—just—you have a probability edge, not a certainty edge. The best count—the best card counter in blackjack still loses hands because they can’t control everything. But what they have is an edge. And what you want as defense counsel is an edge.

[23:58] Bill Now, if you go to trial and the plaintiff’s done all this homework, they have an edge. If you haven’t matched that effort, they’ve done their homework, you’ve done your homework. There’s no imbalance there. Then you kind of both have equal risk, right? But if they’ve done all the homework, they’ve followed this method I just described, you haven’t—yes, you both have risk, but now you have greater risk than the plaintiff. You have greater risk than the plaintiff and the tables tilted in the wrong direction. And I would argue that’s not necessary, but unfortunately that—that’s what I see—um, that’s what I see a lot of. That’s what I see a lot of.

So again, this podcast is just—is to reinforce and we’ve talked about this before, but I’m going to keep talking about it because it’s the only way the message is going to get out is there’s a much better way to do this. But it requires a change in philosophy, a change in mindset. I gave a speech on this and a veteran—a friend of mine, a veteran defense attorney of over 30 years raised his hand confused like, “How am I supposed to test my case when I don’t even have the plaintiff’s expert report?” And my response was, “You don’t need that. You’ve got all this other stuff that you need to test. That can come. We’ll get that eventually and then we’ll test that.”

[25:25] Bill But the—the tendency is I’ve got to have everything before I test. Absolutely fundamentally 100% false. Absolutely not. Absolutely not. And then if you get those exploratory answers early during discovery or right before discovery, that can direct your discovery, right? That may impact how you work with your expert or what—what expert you may even hire. That’s going to affect how you prep your witnesses. So, this, you know, early intervention here with this exploratory method is a win-win-win-win across the board for you as defense counsel, for your client to get that information, get it early, better decisions being made.

I can’t tell you how many times we’ve gone through the exploratory process and a client that was dead set on, “we are trying this I’m not giving them a nickel,” and then we get through the exploratory process and they end up settling the case going, “damn, damn, I misread that.” Confirmation bias. You think you know but you don’t really know. That’s where very, very expensive mistakes tend to happen. Have a great week, folks. Litigation Psychology Podcast brought to you by Courtroom Sciences. I’m Bill Kanasky. See you next time.