Expect the Worst and Report the Unexpected (Judy's Questions 6–7)

Chapter 2 Episode 6

6. What do you expect to find? What did you find? Results.

7. What does this mean? Conclusions.

Unlike Q1 through Q5, which cover the hard work of defining the context, gap, research question, and methodology, Q6 through Q10 shift toward interpreting and positioning the work. This episode covers Q6 and Q7, where you anticipate your results and turn them into findings, and the next episode covers Q8 through Q10, where you position the work for your stakeholders, your community, and your career.

Q6: What Do You Expect to Find?

You do not wait until a study is running to start thinking about what to expect from it. Instead, you should be able to generate reasonable hypotheses about what the results will look like before you run the study, and those expectations should follow logically from your approach (Q4) and the gap you identified (Q3).

I am not talking about vague hopes like “I think the system will be useful,” but rather realistic and specific perceptions and feedback from diverse participants, formed by envisioning what the interaction would look like if the system works as intended (and sometimes when it does not), and how participants would perceive it.

I always ask my students to role-play as participants while designing a study, so that they can better anticipate the different kinds of feedback that may occur during the live session and respond to it more confidently. Role-playing is a very powerful skill, but only if you do it properly.

Most importantly, you should never expect participants to behave the way designers (you, specifically) hope they will. The interview protocol should not be built on the assumption that participants will always welcome your technology. Remember that participants may have no prior knowledge of the study or the system, and they may already carry divergent opinions (sometimes even the opposite of what you expect) before they participate. In other words, always prepare for the worst.

The healthcare setting is a good example because, surprisingly, a significant number of people are inherently antagonistic toward novel technologies, or have zero trust in them. Many professional stakeholders worry that designers (again, you, specifically) are trying to develop systems to replace them and cost them their jobs — even if you actually are not. They will tell you that they do not trust anything an AI produces, and that they typically will not use it. If you were not prepared to receive such negative feedback while designing the study, you may find yourself lost for words when you actually face it.

There is more to say about interview skills, but that is beyond the scope of this episode, and I will cover it in a later one. For now, assume you have prepared well and the study went smoothly. What’s next?

Q7: What Does This Mean?

Eventually, you will need to turn the analytical results from your study into the findings of your paper. I have heard students say, “Writing findings is the easiest part of the paper because you just need to summarize what you have heard or collected.” If you hold this belief, you are very likely to fall into several common failures.

The first is reporting something that lacks novelty — either common sense or something other people have already reported. This can be caused by two primary factors: you did not do the literature review comprehensively, or you did not dive deep enough beneath the participants’ words. If the key observation is a straightforward paraphrase of several participants’ quotes, then it is almost guaranteed to be too superficial.

A rule of thumb for findings is that a good one makes readers, who are mostly experts themselves, feel that the result is unexpected yet entirely reasonable. For example, if you are designing AI technologies to support home-based care, reporting that “patients often lack the technical literacy to use novel technologies” is common knowledge and thus should not be reported in your work. You should instead dive deep into the factors and rationales underlying the observations.

A second common failure is overclaiming. You should not draw conclusions that go beyond what the data conveys and what your methods were designed to test. For instance, if your study quantitatively measured task completion time and subjective trust, you cannot conclude that the system “improves clinical decision-making” without evidence about decision quality.

There is another misperception junior students often hold, particularly about qualitative studies. Students often believe it is best to receive only positive feedback during the interviews, because it demonstrates that the system functions perfectly and that participants all like it. I would take the other side, however, because I believe that receiving mixed, even conflicting, feedback is much more interesting and worth investigating.

First, if all your feedback is just “your system is good or useful,” a minor issue is that you have nothing else to report in the findings, but a major issue is that it may indicate that the study lacks comprehensiveness and that the results lack in-depth analysis. What people like and what they do not, what could be improved and what is redundant, what works as expected and what goes the other way — there are tons of directions in which you can dive deep and derive interesting findings. If your results surprise you in a way you cannot explain by tracing back through the logical chain, or if one participant’s feedback conflicts with another’s, that surprise is a signal that your understanding of the problem was incomplete, and, very likely, that the whole community’s understanding is incomplete as well, which leaves you an opportunity to contribute.

You now have results and a defensible account of what they mean. The remaining three questions turn outward, asking who your findings matter to, where the work belongs, and how this project fits the arc of your career. The next episode covers Q8 through Q10.