Rigor vs value in industrial research
Today marks my first year working as a Researcher for Zup Innovation, a Brazilian tech company. When I joined Zup, I often told people that the last time I had a contract with a company was like ten years ago. But, of course, 10 years in the tech industry is much more than a decade. Although my job is not in engineering per se, this year told me that even outdated programmers like me can be back on track.
My job at Zup is a mix of many things. A bit of engineering, a lot of management, a bit of writing, a lot of meetings, a bit of paperwork, and (sometimes a bit, sometimes a lot of) research. Even though I had some experiences collaborating with companies while I was a full-time professor, this was the first time that one of my main goals was to impact software engineering teams through research. Despite my academic background, this was not something crystal clear to me.
During this year, I’ve talked to many engineers/directors about how we could bring value to them. The more I talked to people, the more I understood that the research we did in the lab would hardly be translated to the company. Many variables impact this process (such as the scale of the experiments, the quality of the codebase, etc), but I believe the most critical variable here is time.
In the lab, we usually are OK to perform one experiment in, say, 2, 4, or 6 months. It takes time to understand the problem, train the students, set up the experiment, etc. It’s not uncommon to have studies going on for over a year, sometimes more. If the student leaves the lab, it’s OK to find and train a new student, which would take a few more months. Even in the best circumstances, if we believe that the results are not just right, it’s OK to redo the experiments, in order to make sure that what we see indeed makes sense. In the academic lab, time is our friend. Obviously, students have to graduate, projects need to be finished, and papers have to be written. But, generally speaking, it is OK to spend more time reflecting and refining the research before making it public. You know, people may make decisions based on our research, so we should value the rigor of our methodologies.
On the other hand, engineering teams operate at a fast pace. They ship code regularly, perhaps every day. They provide value to their clients through new features. Sometimes the value of a new feature is not always clear, so the teams have to experiment a bit. Experimentation requires early feedback, which is often much more appreciated than waiting for months to approach the clients. Teams have to move fast; otherwise, other teams will. And no problem if a new release comes with a bug. We can fix it in a minute and ship it again. Therefore, mottos like “move fast and break things” are not just an anecdote. In engineering teams, moving fast is often more desirable than having a bug-free product.
This is the dichotomy of industrial research.
While researchers favor methodological rigor, engineers favor delivering value fast to their clients.
In this context, it is not feasible to believe that we could just apply our favorite academic method in the industrial context. This will not work because the two teams operate on a different schedule.
If I tell to engineering teams that I could help them in understanding this particular problem they are facing, but it would take, say, 3 months, chances are they will look for the solution themselves. Depending on the context, in 3 months, maybe the whole project had changed its direction, and that particular problem may not relevant anymore.
By design, things like systematic literature reviews are just unfeasible to do. Creating that tool that will improve the precision and recall of state of the art might also not sound like an appropriate thing to do.
Not only it is not feasible to apply certain kinds of research methods, we also believe that we should relax parts of the methodological steps. For instance, instead of transcribing and coding dozens of interviews, we may conduct just a few ones, skip the transcription part, and report just a few insights of what we have heard here and there. Similarly, instead of writing one new tool from scratch, we may use (or adapt) an existing academic tool to the company’s context.
We can help the teams with their needed velocity by relaxing the research method. However, by relaxing the research method, our research becomes less likely to be well-received in the academic community. And it’s our interest to contribute with scientific knowledge to our community.
This is the dichotomy of industrial research.
At this point, I realized that our research team might be unable to contribute to the research community with new methods, techniques, and tools, just because they take too long to be polished.
And that’s OK.
I also realized that we have a unique opportunity to explore the behaviors of the engineering teams during the software development process. Since we are near to the teams, we can ask questions like: What are the team’s perceptions when adopting a research-oriented tool? To what extent our relaxed research methods could influence teams in their decision-making process? How do the teams find and share knowledge? And the like.
In summary, I learned that industrial research requires its own set of methodologies and tools. It is not feasible to translate our full-fledged academic methods to the industry in the hope they will work. Instead, we should tailor the methodologies and questions to the practitioners’ needs. Although writing this last sentence seems something obvious now, it was not evident in the first place 😊.
In an academic environment, most questions deserve investigation. In an industrial setting, just a subset of questions deserves investigation.
And perhaps this is the beauty of industry research.
During this first year, we made a few submissions, and one paper was accepted. For the next year, our plan is to improve our hability to conduct and share industrial research. Let’s see how far we can go.
BTW, if you are a research looking for an industry pattern to perform your experiment, drop me a line; we can make things happen at Zup!
There is much more to say about this first year at Zup, about other things I learned during this small time window. I also miss a bit writting blog posts. I really hope to keep this blog more active in the near future.
