Episode #257 – Why Jury Research Needs Validity and Reliability – Part 2 of 2

Bill Kanasky, Jr., Ph.D. continues discussing the importance of validity and reliability in jury research and specifically talks about the use of the clopening in jury research. The clopening is a combined opening and closing statement – basically a summary presentation of the case. The issue with the clopening is that it impacts your validity and reliability because jurors don’t hear clopenings in a real trial so any feedback collected is skewed. Also, in order to get the most accurate data in jury research, you have to measure immediately after the presented stimulus/information. For example, if you want feedback on your opening, you must measure immediately after the delivery of the opening. If you want feedback on a witness, the measurement must come immediately after the mock jurors hear from that witness. Waiting to gather feedback until all witnesses have been shown will not provide an accurate measurement. The most scientifically sound methodology for conducting jury research is to take measurements immediately after completing delivery of each piece of content that you want feedback on. Any other process for data collection will compromise your validity. 

Full Episode Transcript

[00:15] Bill Welcome to another edition of the Litigation Psychology Podcast brought to you by Courtroom Sciences. Dr. Bill Kanasky. I mentioned something on the previous podcast and I forgot, I forgot to break it down, so we’re just going to do part two to the validity and reliability, uh, podcast. Yeah, I got to, I, I jumped right into my egg yolk rant and forgot about, uh, talking about a really important topic which is, um, I’m just going to call this the, uh, the death of the clopening in jury research. The clopening. What is a clopening? Huh? It’s a combined opening and closing. Closing and opening—clopening. It’s a summary presentation of your case; you kind of put everything in there. Okay, so the plaintiff does their clopening at the beginning of your mock or focus group, throw everything in there, lasts 45 minutes, an hour, hour and a half, whatever, two hours. All right. And then afterwards you collect data from jurors, and then the defense puts on their clopening, right? This is how jury research has been done for decades. Okay. And I have a problem with it because I think scientifically, okay, I think we have a, I think we have a validity problem. Remember back to validity: are you measuring what you think you’re measuring, right? And you don’t want to contaminate your validity by using, using poor measurement strategy.

[02:04] Bill Okay, so here’s the problem with the clopening which everybody still does. Everybody still wants to do it. It’s not, I mean, I talk 99% of my clients, I talk them out of this. I’m gonna tell you why right now. Okay, well number one, real jurors don’t hear clopenings. What do they hear? They hear openings, then they hear witness, break, witness, break, witness, break, witness, break, witness, break, then they hear closings. That’s how information is presented in every courtroom in this nation. There is no such thing as a clopening in any courtroom. Yet that’s what so many people want to do because they find it very efficient. Yet you’re going to suffer, you’re going to suffer on validity because you’re not measuring what you think you’re measuring, and you have significant contamination. Very, very important. We don’t want contamination. Now let me kind of explain to you how this works. So, and the clopening is typically a longer presentation. You have your PowerPoint slides, you, you have the ability to argue unlike you doing a traditional opening statement, and you’ll insert your witness clips in your clopening, right? And you put all that together. The jury sits there and they, they have to absorb an incredible amount of information that they would never have to absorb in a real courtroom. In a real courtroom, they sit there for 30 minutes, listen to your opening, then there’s a little break. Then they sit there for another 30 minutes and listen to the other side’s opening. Then there’s a break, right? There’s a natural divide in these stimuli in a real courtroom which allows information to sink in and judgments to be made.

[04:06] Bill Then you go do your Jury Research, right? And you just want to clump all this stuff together. Okay, the cognitive load, okay, is incredible for the juror. It’s entirely too much information at once. You want jurors to sit there and listen to the plaintiff’s case or the defense case—the whole thing—and you’re just going to give it to them all at once with no breaks and no data collection. Okay, efficient? Perhaps. Valid? Not necessarily. Okay, so let’s talk about this, uh, analogy. Let’s say if you’ve ever been to Wine Country, okay? You go to Wine Country, you’re going to do a wine tasting, and they put six little, I put a flight, five or six little glasses of different red wines. Okay, would you sample all six in a row? Not have any water, just all six in a row and then determine which one you liked best? Imagine that, right? So, I put a flight of red wine in front of you and I tell you, “Hey, drink the first one, then drink the second one, then drink the third one,” right? And then at the end of the six, then I go, “How would you compare wine number one versus wine number three?” Is that going to work? No, no. And that’s what a clopening is. You’re giving it to all, you’re giving all this information, you’re—the cognitive load’s through the roof, and then you’re asking them for data. Imagine that. And then you’re going to, oh by the, okay, so now, now compare, uh, wine, uh, number two and number six from the flight that I just gave you. It’s impossible. You’re not going to get good data. And that’s what happens with the clopening. But, and the reason everybody does it this way is because that’s the only thing they know. It doesn’t mean it’s correct. So, for to increase your validity and importantly, the sensitivity of your data measurement—see, when you use that, this wine concept, right, you’re losing sensitivity because you’re giving them all six wines and asking for all the, the data collection at the end. It’s not, it’s not, it’s going to be garbage data. And the last thing you want in litigation is garbage data.

[07:00] Bill You’re investing time, money, energy into gathering data. You should probably think about doing it the right way. Okay, so in your mock jury exercises, okay, space things out. So, if you’re doing a mock trial, set it up the traditional way. Do a traditional opening and then immediately collect data on that opening. That’d be like saying, “Okay, here’s wine number one, try it.” You good? All right, now fill out this little questionnaire on the qualities of wine number one. Then you’re going to take a drink of water, cleanse your pallet. Wine number two, take a sip of that, swish it around, okay? All right, let’s fill out this questionnaire on wine number two. And then you work all the way through six. Now you’ll get, now you’ll get good data, you’ll get valid data doing it that way versus doing all six at once and then testing at the end. And in fact, you’re going to get crazy different data. Your data is going to be much better testing after every glass of wine. And that’s why you want to do your jury research the exact same way. Okay, do a plaintiff opening, test the story, decrease the cognitive load, test the story, collect data immediately. Defense traditional opening, test the story, collect data immediately afterwards. Now, same thing with witnesses. If you put all the witnesses in the clopening, it’s back to the wine tasting, right? Now you’re, okay, say, say you showed seven witness clips in your clopening and, and now after the clopening you’re going to have jurors go back and rate the w—get out of here. No, no, no, no, no. That’s, that’s, that is not going to happen. Well, it’s going to happen, but it’s not going to happen correctly. You need, you want good data. So, what do you do? Do it, do it like they do in a courtroom. If you’re going to do a mock trial, do it like they do in a courtroom. Opening, plaintiff, take your measurement. Defense, opening, take your measurement. Get rid of all this clopening stuff, right? You’ll have much better measure, much, much better accuracy and sensitivity to your measurement.

[09:32] Bill Now with witnesses, put on the witnesses. Witness one, measurement. Witness two, measurement. Witness three, measurement. Witness four, measurement. You will get fantastic data versus the clopening where they sit there for an hour and a half, sometimes even two hours, I’ve seen these, and then they sit through all that. So cognitive load, you’ve totally overwhelmed them cognitively. Then you’re going to say, “Okay, uh, give me these ratings on witness, witness number two.” Yeah, by the way, that happened an hour and ten minutes ago. No, not, that’s not how the juror brain works. And at trial, that’s not how it works. What happens? It’s witness one, witness two, witness three, and there’s breaks in between, and jurors take notes, and that, they have a, they have a, a time period to determine how they feel about the witness at trial. So, you mimic that in your mock trial. Even your focus group, which I’ll get into in a second, right? Then once you do that, you do truncated closings at the end. Same thing. Plaintiff closing, measurement. Defense closing, measurement. If there’s a rebuttal, rebuttal, measurement. That’s how you’ll get the best data because if you clump it all together in these clopenings, once the jury starts, you start collecting data from the jury, everything’s contaminated and mixed. It’s like the wine tasting, right? Once they’ve sipped all six wines and they haven’t had a chance to let it sit and cleanse their pallet, and then you’re going to ask them about the difference between wine number three and wine number five, it’s impossible. Oh, they’ll give you data, it’s just going to be garbage, invalid data. I think that’s something, I think that’s something we want to, uh, avoid. It’s a much better system to get your measurement immediately after the stimulus. It’s a very good system. Little Goodfellas reference there. It’s a very good system.

[12:00] Bill Let’s do that one more time just to, I mean, Goodfellas is fantastic. Henry Hill. It’s a very good system. Uncle Paulie had a system of slicing the garlic. You got to have a good system. The clopening is really, it’s an efficient system, but you need an effective—efficiency doesn’t really get you anywhere in scientific measurement; it does the opposite. You need an effective system, right? So, what do you do if you’re doing a focus group instead, right? Well, focus group, again, you should have one presenter, not two attorneys going at each other, cause that’s going to contaminate everything with persuasion, okay? And you take your measurements immediately after the stimulus. So, on focus groups, you’re not really doing the opening, closing, witnesses and stuff. You’re going issue by issue. Do issue one, collect data immediately. Issue two, collect data. Then here’s the key: if you want to have witnesses being tested in your focus group, all you have to do is isolate them. So, you have a key, okay? So, you’re going to do your, do your issue one presentation, okay? Collect data on that. Then tell the jury, “Okay, now there’s two key witnesses to, to issue one. I’m going to play a ten-minute clip of number one.” You play it, collect data immediately. And then witness number two, collect data immediately. That’s how you do it in the focus group format. That’s how you’re going to get your very, very, very best data. You want accuracy, you want validity, and so the timing of your, uh, measurements is essential. You can’t ask a juror for a measurement to something they heard an hour ago and that was mixed in with a whole bunch of other stuff. So, kind of use that wine tasting analogy and when you’re setting up your jury research, think about those questions.

[14:15] Bill So that’s, so anytime I sit down with clients, that’s the, I say, “Okay, what do you want to measure?” And then, and here’s of course what you guys tell me every, “Well, I’ve, I’ve done, I’ve done 15 mock trials and I know you do a clopening and blah blah.” It’s like, okay, well maybe you did 15 of them that are wrong. Let’s do it this way. And then when they get the data, they’re thrilled. Our clients are thrilled, going, “Oh wow, I didn’t know we could do it this way.” But if you think about it, that’s how jurors are making decisions during trial, okay? Things are coming in a logical order, they’re separated, there is a chance to digest this information. So do your jury research the same way, whether it’s mock trial or whether it’s, or whether it’s, um, focus group, you know, research. Research is difficult and what ends up happening is you have non-scientists, you know, doing science, and then you can get, yeah, I, the worst thing you can get is garbage, you know, is garbage data. Particularly on the witnesses because the witnesses are so important, right? So, you want really accurate data on that. The only way you get the accurate data is to take the measurement at the time.

[15:45] Bill Okay, now here’s kind of another way to do this, particularly in the focus group. We do a lot of, lot of focus groups. I think they are ten times better than mock trials in most situations for a, for a number of reasons. But one of them is, say, say you’re presenting on your issue, and you want to present the witness within the issue, not at the end of the issue, right? Okay, that’s fine. So, you’re going to give your, say, little presentation and then you want to go, “Okay, now let me show you, I want to show you a ten-minute clip of this witness.” And you play the witness, but you still got half your presentation left. So, what do you do after you play the witness? Take a little timeout and go, “Okay, we want to collect data on this particular witness.” It takes three minutes. And then once you collect the data, you pick up your presentation and proceed. Okay, so think about this podcast as when you’re taking your measurements has a major, major impact on validity of the accuracy of the measurement. You want to take it immediately after the stimulus. And the problem is it’s really difficult, it’s impossible to do that in the clopening, right? The only reason the clopening came to exist is again, uh, it’s an efficiency-based model. Um, we rarely really do that now unless some client just screams, “Oh no, this is what we’re doing!” and I tell them like, “We’re not going to get the very best data.” Okay. And so, as we talk and we educate our clients, most of them once we kind of lay this out about how the sensitivity of the measurement is impacted, then they go, “Oh wow, oh wow.” And, and by the way, what does a clopening get you? You know who does, again, who, who’s ever, you know, email me if you’ve ever done a clopening in a real trial, because I’d like to hear about it, because I don’t think it exists. It doesn’t. It’s a, it’s a, it’s a, it’s a shortcut.

[17:59] Bill And shortcuts in research lead to problems. How about medical research? You think they’re doing shortcuts there? I, I certainly hope not. You don’t want them doing shortcuts there, do you? Right. So, as you’re designing, as you’re designing your research, think about, “Okay, I want accuracy, I want sensitivity of my measurement. Let’s space this stuff out. Let’s isolate the stimuli and let’s take measurement immediately after the stimuli.” I see this done with experts too. They’ll have like a, even like they’ll, like they’ll do the clopening like, “Okay, we want a separate section with experts,” and they go and they play four videotapes of experts back to back to back to back. And then afterwards, and that takes two hours, uh, 30 minutes a clip, and then afterwards you’re going to collect data? No. No, no, no, no. It’s not going to work. It’s not going to work. The truth of the matter is is that there’s a lot of shortcuts taken in jury research and clients do not know the consequences of these shortcuts. I mean, that is the truth of it. Okay, it, it is, I mean, it is what it is. “You can’t handle the truth!” [Laughter] Well, you better handle it because listen, these are easy fixes. These are easy fixes. Now here’s the push back: “Well, I’ve done 17 mock trials, I’ve done 17 focus groups, I’ve been trying cases for 30 years, like, this is the only way I’ve done it. I’m not comfortable.” Well, well you, well you need to get over that. You need to get over that, okay? It’s a better way to do things.

[20:01] Bill I explain this to, let me tell you a quick story and then I’ll let you guys go. So, I’m out to, so I’m out to dinner with a, with a new client and I had been prepping witnesses for them all day. It’s my first interaction, so I have the defense attorney and the corporate client, the head of litigation. We’re out to dinner at a steakhouse and they’re very, very pleased with our methodology which, as you know, is highly sophisticated, highly scientific when it comes to witness training. They were thrilled, like, “Oh wow.” So, at dinner they’re like, “Well, tell us about the jury research,” and I kind of reversed the question. I’m like, “Well, I know on this particular case you’ve already done your jury research.” I’m like, “Tell, tell me how you typically do it and I’ll kind of tell you if we do it the same or we do it different. Just kind of tell me.” And they walked me through the whole, “Oh, we’ve been using Company X for like the last five years and this is kind of our standard research setup.” I looked right, this is right at the head of litigation, I looked right at him and said, “Well, that’s no good.” And he kind of looked at me in shock. He’s like, “Well, what do you mean it’s no good?” I go, “Well, that methodology, you’re going to hurt your validity, and your data is not going to be, your measurement’s not going to be sensitive enough because look at where you’re taking the measurements. You’re doing your clopenings then taking the…” so I just, I went through the whole thing. I went through the wine analogy with them. I went through the whole, this whole podcast is essentially what I told them. Now everybody’s got like filet mignon in front of them. Everybody dropped their forks and just listened for 15 minutes while I talked about this, about scientific measurement and the risks of cognitive load on jurors. And everybody’s sitting there going like, “Holy… well that, that makes a lot of sense.” I go, “Well yeah, because that’s, that’s how you measure, right? This is science.” I mean, the name of the company is called Courtroom Sciences, right? Clopenings, you’re really kind of killing the science and you’re, you’re put… here’s what you’re doing: you’re putting efficiency over science when you do the clopening model.

[22:13] Bill And there’ll be a million people out there that are going to argue with me and they’re more than welcome to either come on the podcast or call me, and they’re going to lose that argument, okay? Now I’m not saying doing jury research using… I’m not saying it’s useless. I’m not, I’m not saying it’s useless. That’s not what I’m saying. What I’m saying is your data collection and the accuracy and sensitivity of it is compromised by definition because of cognitive load and timing. Guys, all this could be fixed. All this can be fixed. But you have to change your methodology and collect your data at better time points. And this is again, mock trials or focus groups. So, you know, take that for what it’s worth, but if you work with me, uh, that’s the, that’s the way I’m going to try to persuade you to do it because you know, I, I have to put my name on all this stuff. So do my colleagues, and we want the best data possible for you so you’re making good decisions, so you’re making good decisions. And you’ll be amazed on the difference in the data between doing, between doing the clopening and doing it the, uh, more isolation way where you’re taking your measurement immediately. It’s very, very different data because you’ve decreased the cognitive load. When you decrease the cognitive load, you’re not overwhelming jurors; you get better data. Again, so next time you think about this or next time this comes up, think about the wine tasting, right? Think about the wine tasting and think about how you’re doing your jury research. Think about that measurement, and if you’re at the wine tasting, where, how are you going to get the most accurate measurement on wine tasting? It’s going to be one wine at a time, cleanse the pallet, right? So, it’s going to be wine, measurement, cleanse pallet, wine, measurement, cleanse pallet, wine, measurement, cleanse pallet. And that’s exactly how you want to do your jury research. This is Dr. Bill Kanasky, Litigation Psychology Podcast brought to you by Courtroom Sciences. We will see you next time.