Research Rigor, Reproducibility, and FAIR Data — Time to start playing games?

Chess, crosswords, Pokémon Go; most people indulge in games and scientists are no exception. Thanks to social media, I’m more familiar with some scientist’s Wordle streak than their publication record. Are games an unexplored tool for improving the quality of science?

That question occurred to me a few weeks ago while I was attending a meeting of the NIH-funded Community 4 Rigor (C4R), a project with tagline “less wrong everyday.” C4R is one of several of learning resources working to make science more reliable by countering the reproducibility crisis. C4R and its peer efforts improve science by teaching what many refer to as the “hidden curriculum,” all the “things no one teaches you” which are needed to succeed. Similar efforts and even mandates are transforming other areas of scientific practice, such making data more findable, accessible, interoperable, and reusable (FAIR).

Most learning happens outside the classroom

Why do scientists, after years of study, still lack key skills upon graduating?

Becoming a researcher is a progressive journey. Time spent researching, solving problems, and testing hypotheses is fundamental to this process. In life sciences, graduate school, while intense, involves little formal classroom time. Instead, it’s filled with moments where informal knowledge acquisition is crucial.

Informal lessons you should learn as a life scientist are short, sweet, and countless: “comment your code,” “talk to a statistician before you design your experiment,” “check if your container is safe for the autoclave.” These lessons often come from mistakes or from observing experienced colleagues. This informal, self-directed learning is vital, yet often secondary to formal education. However, it’s this blend of formal and informal learning that embodies the essence of a discipline’s academic culture. Our parents teach us how to put on clothes, our cultures teach us what to wear.

As valuable as it is, the graduate experience and its embedded untaught curriculum comes with an unavoidable flaw. There will always be some necessary knowledge or skills that aren’t covered in formal classes. It is unreasonable to expect that everything needed prepare someone for their career can come in a textbook. Some things you have to learn on your own, right? However, extensive problems with the reliability or reproducibility of research suggest that leaving some knowledge and skills for people to seek out on their own is costly or even dangerous. Additionally, we must also account for knowledge and skills that were left out of training because they did not exist or weren’t considered crucial at the time. Finally, I think it’s also worth considering that some graduate programs — those with more resources — are more likely to have both more enriched formal and informal learning opportunities. I’d suspect that even when the formal curriculum is revised in an attempt to deliver equity, the informal and untaught curriculum gets left behind. A low-resource institution will have an easier time implementing an Ivy League school’s learning objectives than reproducing its intangible academic environments and networking opportunities — important sources of informal learning.

Maximizing the potential for high quality research requires us to stop leaving the acquisition of informal knowledge and skills to chance.

Playing Games with Science

What unifies many attempts to improve scientists’ professional preparation is that the focus is on replicating the formal learning environment. As C4R and many other projects demonstrate, there is value to sitting researchers behind desks, providing needed training, and mirroring the more formal structures of graduate education; hands-on learning isin’t necessarily excluded here. To address training gaps, post-graduate professional development is a solution many rely on. I’ve written about the importance and challenges of this approach here.

Can we intentionally improve informal education for researchers? It occurs to me that games could be an answer.

In the few minutes needed to solve a Wordle or a crossword, we should be able to introduce, reinforce, or correct important knowledge and skill gaps. Development of pedagogically-sound games for researchers might be a tool to address a knowledge or skill gap.

Want to be a ScienTest?

Imagine our game, ScienTest, a smartphone app designed to prove to yourself — and most importantly your peers — that you have what it takes to be a rigorous, reproducible, and respected scientist…

On a daily basis you are presented with a short quiz or puzzle. Rather than inquire about trivial facts, your challenge is carefully chosen to test your judgement in an area of scientific practice. Challenges might require you to estimate what power requirements you should choose in a study design, if you need to rewrite some code into a function, or how you would handle a review where you spotted a suspicious image in a draft manuscript. The goal is to get a player to ask “what would a good scientist do.” Win or lose, the game also suggests reliable learning materials which reinforce good practice.

The varieties of features and approaches are many and relevant. Challenges could:

• Have disciplinary focus; i.e., “challenges for biologists,” “challenges for physical chemists.”
• Engage multidisciplinarity; statisticians develop questions on concepts they frequently see particle physicists misunderstand.
• Attack misconduct and misconceptions; pose ethical dilemmas and scenarios.
• Build progressively; a gastroenterologist is challenged over the course of the week with a series of case studies.
• Introduce emerging techniques and standards; quizzes lead to formal learning materials and highlighted tutorials.

Games should be fun, and there are a number of game features that could be incorporated:

• Leader boards allow players to share their scores.
• Aggregate scores could be proxies and points of pride for disciplines, departments, or institutions.
• Augmented reality and other cool gameplay features could be incorporated. Instead of Pokémon, animated geolocated mascots could provide informative (or disinformative) clues for players — The C4R “Rigorous Raven” could have cousins, “Risky Raven,” “Reckless Raven”.
• Cash prizes (used else where to highlight exemplary practices) might be an appropriate motivator.

Operationally and ethically, additional considerations would need to be developed including mechanisms to:

• Vet the question set and allow for moderated debate or correction.
• Incentives and attribution mechanisms to reward experts who develop question sets.
• Pedagogy expertise to refine game play efficacy and measure impact; trends amongst intended audiences could reveal misconceptions that require attention.
• Governance and collaboration mechanisms that select question areas based on needs and which are accountable for aching ethical, inclusive, and just outcomes.

Towards a ScienTest-index

At the end of this scenario, the goal of such a game is learning. An opportunity for an informal, low-threat environment where scientist can engage with the best practices of their field. Your ScienTest-index could be a much better indicator of your research quality than your h-index. A game of Wordle can teach me an obscure English word. A game of ScienTest could correct an outdated belief or expose me to new ways of thinking about my work.

Who’s ready to develop this?