LISTEN CASE STUDY
How Cognition uses Listen to build better AI products
Listen Labs & Cognition
Impact TLDR:
Engineers run research directly without a dedicated research team
Identified critical onboarding friction points within days
Shipped onboarding improvements in a two-week sprint
Doubled the percentage of users who progressed from landing page to merging a PR with Devin
Turned user interviews directly into product decisions and engineering tickets
The Challenge: Understanding Why Users Don’t Reach Value
For Cognition, customer understanding is not owned by a separate department. The people building the product are expected to understand the people using it.
“If you’re an engineer, the number one thing to think about is being obsessed with customers and making sure they’re having a good experience using your product,” said Adhyyan Sekhsaria, Founding Engineer at Cognition.
Cognition is the company behind Devin, an autonomous AI software engineer, and Windsurf, an AI-native development environment.
The company operates at the frontier of AI-powered software development, helping engineers build and ship software faster than ever before. But while AI has dramatically accelerated the speed of coding, Cognition believes the harder problem is understanding what users actually need.
“Cognition does not have a traditional market research team,” said Theodor Marcu, Head of Product Growth.
As Devin grew, Cognition became increasingly focused on onboarding.
The challenge wasn’t getting users to create accounts. It was helping them successfully reach the point where they experienced the value of an autonomous software engineer.
“We understand that even a little bit of friction can cause people to churn very quickly off the product and they wouldn’t even get to the part where the value is delivered,” said Sekhsaria.
Product analytics could show where users dropped off. But they couldn’t explain why.
The team needed to understand what first-time users were actually experiencing as they navigated Devin for the first time.
As Marcu shared, onboarding Devin is a uniquely difficult product challenge. Users must connect repositories, configure permissions, grant access to services, and trust an AI agent with their codebase before they can experience the product’s value.
The team needed to see the onboarding process through the eyes of new users
Company Background
The Study: Watching Users Experience Devin
“We noticed that there were too many buttons on the UI and people were overwhelmed with what to do,” said Sekhsaria.
The study also surfaced edge cases that are difficult to reproduce in internal testing.
Different browsers, operating systems, repository configurations, and developer workflows revealed moments of friction that weren’t apparent through analytics or internal testing alone.
“These are the kind of issues which are hard to find but very easy to fix,” said Sekhsaria.
In another study focused on Devin Review, Cognition uncovered a surprising discovery: users repeatedly requested a feature that already existed.
The product included a way to chat directly with Devin Review about a pull request, but users weren’t finding it.
The problem was discoverability, not functionality.
To better understand the onboarding journey, Cognition ran a Listen Labs study with first-time users of Devin.
The goal was simple: observe how users interacted with the product, identify moments of confusion, and understand what prevented them from reaching value.
The study combined interviews with screen recordings, allowing engineers to watch users navigate the onboarding flow in real time.
What they found was difficult to see through analytics alone.
Some users lacked the GitHub permissions required to connect repositories. Others were hesitant to grant access to their code. Many felt overwhelmed by the number of options available during onboarding.
Turning Insights Into Product Changes
Unlike traditional research workflows, there was no handoff between researchers and engineers.
The people watching the interviews were the same people shipping the fixes.
“We had a few engineers working on trying to optimize the onboarding flow,” said Sekhsaria.
Using findings from the study, the team prioritized onboarding improvements and redesigned key parts of the experience.
One recommendation was introducing a more guided onboarding experience, helping users understand what to do first rather than presenting every option at once.
The team also improved feature discoverability, addressed onboarding friction, and fixed numerous usability issues uncovered during the study.
Many of the changes were small.
But together, they removed barriers that prevented users from reaching value.
The Result: Activation Doubled
"Without Listen, we would have moved much slower. It would have taken us longer to get to those lessons and refine our thinking around what makes a great Devin onboarding."
Theodor Marcu
Why Customer Understanding Matters
For Cognition, one of the most important onboarding metrics is the percentage of users who progress from landing on the product to successfully merging a pull request through Devin.
After implementing onboarding improvements informed by the Listen study, that metric doubled.
“The main metric we care about is the percent of users who go from the landing page to merging a PR through Devin,” said Sekhsaria. “That metric doubled.”
The improvements weren’t the result of a major product overhaul.
They came from identifying and removing moments of friction that prevented users from succeeding.
More importantly, the team reached those insights far faster than they otherwise could have.
At Cognition, the team believes AI is fundamentally changing how software gets built.
As coding becomes increasingly automated, the challenge shifts from building products to improving them.
“Coding is becoming less of a bottleneck,” said Sekhsaria. “The hard part is figuring out what the user issue is, formulating a good solution for it, and then implementing it.”
For Cognition, that makes customer understanding a core part of engineering rather than a separate research function.
“The bottleneck after that becomes how fast can you iterate,” he added. “If you really want to maximize user quality and be obsessed with the customer, you have to have a customer in the loop.”
While Cognition is building increasingly autonomous software engineering agents, the company believes humans remain at the center of the development process.
Listen helps Cognition stay close to those users. Rather than relying exclusively on product metrics, the team can observe how engineers think, where they struggle, what they trust, and what ultimately helps them succeed.
"Listen has enabled more empathy for our users. We can cover and understand a lot more ground than we would have otherwise had a chance to."
Theodor Marcu
For Cognition, understanding the people who use and manage AI agents is as important as advancing the capabilities of the agents themselves.