What researchers think about Rapid Reviews?

Rapid Reviews (RRs) are lightweight secondary studies intended to provide evidence to support informed decision making. Different than Systematic Literature Reviews (SLRs) that usually take several months (or even years) to be done, RRs are conducted in time frames more likely to meet practitioners’ demands (in weeks or in a month, instead of months or a year) and demand the involvement of few, sometimes only one researcher, reducing the costs to deliver evidence.

Although RRs are a rising research method in the medical domain, they are barely known by the software engineering community. In a previous study we observed that practitioners were very supportive towards the use of RRs. Unfortunately, the perceptions of researchers, who usually are in charge of conducting the RRs, is still unknown. Considering the recent movements in software engineering to increase the quality of SLRs, we believe that the methodological concessions of the RRs could provoke some controversy in the software engineering research arena.

Therefore, it was not clear to us whether researchers would embrace RRs. On the one hand, although RRs can speed up the knowledge transfer process to practice (and initial results suggest that practitioners appreciate it), researchers may be skeptical to adopt RRs due to methodological concessions (or afraid to have their papers rejected when using a not that rigorous methodology) .

In order to investigate the software engineering researchers perception on the potential use of RRs, we invited a group of 37 SE researchers and asked them to analyze a predefined set of 50 statements regarding RRs, and rank them according to what extent they agree with each statement. To conduct this analysis, we relied on a Q-Methodology. This approach may sound like surveys, but it is intrinsically different. More concretely, using this method, the participants are required to prioritize their perceptions, avoiding bias in their responses (e.g., some participants might be more positive than others).

Using this methodology, we revealed four different perceptions about RRs. They are:

• The Picky: The SE researcher does not believe that RRs can deliver high quality evidence;
• The Unconvinced: The SE researcher needs more evidence about Rapid Reviews;
• The Pragmatic: The SE researcher might conduct some RRs in specific scenarios where they are appropriate.
• The Enthusiastic: The SE researcher is willing to conduct RRs, but only if minimum standards are meet;

With this paper, we understood that some researchers are willing to experiment with RRs, although some of them are still very conservative. However, with this study and our initial results (and eventual other research works that may go along these lines), we believe other researchers may become motivated to embrace this methodology.

If you want to know more about this study, you can read the preprint available here. The tool the implements the Q-Methodology is also available here.

Originally published at http://gustavopinto.org on July 26, 2019.