Stack edit
Graphite, 2023
In the process of developing Stack Edit, we aimed to create a visual, streamlined approach for managing your PR stack, including bulk actions like adding reviewers. By bridging Graphite’s CLI with its web tool, we focused on driving user retention. This project marked my initial foray into zero-to-one endeavours at Graphite, where I, as a designer took the lead—from pinpointing user value and defining features to prototyping and setting key metrics.
This case study only lists high level decisions and considerations. Let's chat in detail – there's several design choices and technicalities to this project.
Impact
56%
of all net PR submissions
Role
Product design and prototyping
Timeframe
3 weeks for design and 6 weeks for implementation and QA
Target user
Graphite users primarily using the CLI
Our primary goal was to transition all CLI users, particularly those not yet reviewing code on Graphite, to our web tool. The number of such users was notably high, and many we spoke with were unaware that a web tool even existed.
Business goal
Our objective was to increase the number of users actively reviewing with Graphite. At that point, Graphite’s pricing model was based on active user engagement, meaning a user counted if they created, reviewed, or merged a PR using the platform. Data also indicated that users who performed multiple key actions tended to stay with Graphite longer, making retention the primary goal.
Success with this feature would pave the way for a future vision: the introduction of AI-generated titles and descriptions tailored to the context of each PR—a capability that was ultimately implemented.
Hypothesis
Introducing the web app to users will help them become more familiar with it, ultimately increasing their engagement with our web interface.
Identifying value through user research
The business goal was clear, but this wasn’t a feature users had asked for. They were content creating PRs through the CLI. My challenge was to identify the value for users and determine what functionality would make them want to use this feature.
An additional challenge was the tight timeline. I had just three weeks to go from research to hand-off while also learning a completely new domain: developer tools.
Insights
I conducted informal interviews with current Graphite users, particularly those who were CLI power users. From these discussions, several key capabilities emerged:
Adding reviewers across the stack
Adding labels across the stack
Marking all PRs in the stack as ‘merge-when-ready’
Adding titles and descriptions
Seamlessly navigating across the stack
Internal conversations with our dev-relations team also highlighted that many users were new to stacking and struggled to fully grasp the concept.
Another important user value was to utilize the UI and navigation of this feature to help users visualise their stack.
Kickstarting design
Once I had some direction, I began exploring various layouts.
Key flows
Eventually, we discovered that users who favoured this feature preferred not to navigate to the PR inbox after a stack. Instead, they wanted to go directly to the last PR in their stack to self-review and double-check their work. As a result, we revised the core CTA from the modal.
Key decisions
Opening stack edit with all PRs in draft mode – Since the main entry point was the CLI, I treated the entire interface as a draft stack. I assumed that this approach would make users feel secure and comfortable adding details before publishing. It also ensured that default reviewers from GitHub wouldn’t receive notifications immediately but only after the PR was published. However, I encountered technical issues where this setup wasn’t possible, either because users were on personal accounts or had a feature flag enabled.
Keyboard accessible – My intuition was that making the entire experience keyboard accessible would be crucial since users were coming from the CLI. To achieve this, I ensured that users could navigate the entire stack in either direction using keyboard shortcuts. Additionally, the title was pre-added based on the commit message to speed up the process, with the focus then shifting directly to the description in edit mode.
Templates in descriptions – PR descriptions are often pulled from templates provided by GitHub, reflecting team culture and requirements. I decided to pre-fill these relevant templates into the description. However, due to the broad scope of the project, we ultimately had to remove this feature from the implementation.
PR page to stack edit – My vision for the PR page was to create a stack view and ensure that the interface wasn't ephemeral. To complete the experience, I decided to add a link back to Stack Edit from the PR page, providing a consistent access point.
User testing and removing features
Since this was an experimental feature with many uncertainties around its visuals and effectiveness in conveying the stack concept, I conducted several rounds of user testing with prototypes.
Key findings
Users did not see value in applying the same description across the stack.
The original intention was to let users add a summary of the stack's content to each PR description, as part of our broader goal to make stacks a key feature in our product. However, users found no benefit in duplicating descriptions across the stack. Moreover, without a dedicated section for a stack description, this approach lacked clarity and would not have been intuitive.
Animation was crucial for helping users understand the stack's direction and navigate it effectively. The challenge here is that while a stack of PRs moves upward, they are technically referred to as downstack PRs. Therefore, it was essential to address both the keyboard shortcuts and the visual direction to ensure clarity.
Users generally publish/ keep as draft one PR at a time, instead of publish all at once.
Prototypes and stack direction
Below are various UI and animation explorations that I tested both internally and externally.
Technical workarounds
This project was an ambitious zero-to-one endeavor for a small, fast-moving team working on an early-stage product. To prevent exceeding our allocations, we had to cut several elements from the scope.
As noted, the description templates posed challenges. I also discovered that the description code was legacy code, which made modifications difficult (though this is no longer the case). Consequently, we opted for having the description selected but not editable. User feedback confirming the need for an editable description validated my original assumption, and we are now including this feature in our scope, as the description code is now clearer.
One of the biggest compromises was the removal of the scroll functionality. Initially, users could navigate through the stack using buttons or keyboard shortcuts one PR at a time. For users who wanted to scan or jump to different PRs in a large stack, I had added scroll functionality. However, the initial implementation was problematic, and fine-tuning it would have taken an unacceptable amount of time. Therefore, we decided to remove this feature during QA.
Data analytics
Almost every function was new to us. Product and data science were ramping up at the same time I was, which explains the limited product involvement in the design process. Our team’s data scientist had also just joined and was focused on setting up analytics for various projects. I provided a list of metrics I wanted to measure for this project, similar to what a product manager would. While most critical pieces were tagged, we could not track the review rate directly from Stack Edit due to how the data was structured, which would have made it unclear.
Core business & product
Total number of users who submitted on web
Of all CLI users, % of users who landed on our UI after being posed the question to submit on web
Web UI usage
Of all users who submitted on web, % of users who returned to our web UI
Of all users who submitted on web, % of users who took their first key action on our web UI after submitted through web
Number of users who took their first key action on our web UI after submitted through web
Drop-off
% of users who landed on the web submit UI, but didn't "publish all PRs" or take any action on all PRs in the stack (proxy for drop-off)
AI generate title + description
Graphite’s vision is to accelerate software development. Automating title and description generation leveraging falls well within this vision. The stack edit surface, as much as the PR page are apt canvases. Stack edit in particular is an author/ creator focussed surface (PR page is reviewer focussed), which makes it the ideal real estate for this feature.
Reduce negative space to improve description readability
Due to legacy code in the description, I couldn’t modify the split view design, which restricted its active real estate despite having more available space. After a user flagged this issue, I redesigned the margins and breakpoints to optimise the layout.
User reception and bold product changes
When users executed the 'gt submit' command in the CLI, they were asked if they wanted to enter the web UI, with the option to decline. Initially, we received feedback from several users who were frustrated by being automatically directed to the web UI and requested an option to disable this feature. To address their concerns, we implemented a flag that allowed users to bypass the web UI prompt entirely.
However, the product team suspected that we were hearing primarily from vocal critics and not from those who might support the feature. As a result, the team made the decision to remove the prompt entirely and automatically direct users to the web tool, unless they had a special flag to opt out. Since this change, approximately 50% of PRs have been submitted through this feature and we haven’t heard any more negative feedback.
