Automations
Graphite, 2023 – 2024
Automations are if/then rules that trigger actions when PRs meet certain criteria.
For example, an engineering manager might automatically add themselves as a reviewer to all PRs authored by their team, or a comment could be left on a PR using deprecated code.
This project began with the aim of boosting Graphite’s adoption in larger organisations by encouraging more engineers to use the product. Very quickly, it evolved into our first revenue-generating add-on after its rapid growth and success. I designed both the feature and the billing & pricing system, overcoming technical and business challenges along the way.
Impact
1100+
Automations created in <6 months
400+
Organisations using it
X0,000
5 figures in revenue
Timeframe
Project has been ongoing for 8 months (Nov 2023 – June 2024).
Original design: 2 weeks
Follow up tasks: 2 months ongoing
Pricing design: 1 week
Target user
Engineering ICs and managers
Ideal for those who want to automate repetitive tasks like assigning reviewers or getting notified about important PR updates.
Business goal
The initial goal was to grow the user base in enterprise organisations. Over time, this evolved into a product add-on that directly boosted revenue.
User assumption
We hypothesised that when an automation posted a comment to GitHub on affected PRs, non-Graphite users would click the Graphite link out of curiosity. This behavior could lead them to create accounts, thereby driving user growth. I modified the login screen for this user flow to highlight the value of Automations.
User research – identify core use cases and unravel behaviour insights
I conducted internal interviews to identify key use cases for this feature. I found team members with experience using Butterfly Bot, a similar tool in Phabricator, from their time at Facebook.
User stories
As an engineer in a core infra team, I want to be added as a reviewer every time someone touches a file I own.
As a manager, I want to be added as a reviewer to all PRs created and updated by my members in my team.
As an engineering lead, I want to be add as a reviewer when another team touches code owned by my team (will be handled through file detection).
Trigger based vs. state based model – technical complexity as Graphite is built atop GitHub
The ideal logic for a feature where there are triggers are and action are –
However, Graphite and GitHub interact via webhooks, meaning PRs are scanned for matching criteria only after each Graphite<>GitHub sync. This was a challenging mental model shift for me. I created several iterations of system flows to fully grasp this interaction.
Understanding this was crucial because it directly influenced how users would experience the results of their automations. For example, an automation would only apply to a PR once if the PR or automation was updated and rematched due to different triggers and/ or actions. It was also essential to reflect everything about how an automation would act on a PR accurately in the UI through copy.
Kickstarting design
Our PR inbox already featured an advanced filtering system that grouped PRs by factors such as author and status. Given the tight engineering timelines and the desire to minimize new surfaces and components, the product and design teams decided to repurpose this existing mechanism for Automations as well.
Jobs to be done based on persona
Automation creator
Create rule (Add name, pick repository, add trigger(s), add action(s))
View matching PRs to double check the automation
Understand how this rule would behave (through copy or visuals)
Delete a rule
Edit a rule
View all rules in an org
Filter automations (deprioritised)
Understand that you only create automations for synced repositories
Someone who’s PR is impacted by an automation
Understand that a bot has taken an action
Access rule through PR
Return back to PR easily
Someone who is impacted by an automation (eg. added as reviewer)
1. Get notified of the action
2. Understand that a bot has taken an action
3. Access rule
Critical workflows –
Our PR inbox already featured an advanced filtering system that grouped PRs by factors such as author and status. Given the tight engineering timelines and the desire to minimize new surfaces and components, the product and design teams decided to repurpose this existing mechanism for Automations as well.
Growing the feature –
Following the public beta launch and its swift success, we started enhancing features and adjusting the product based on user feedback. Fortunately, the lead engineer had built the Automations architecture with flexibility in mind, allowing us to quickly update and modify core functions. I adopted a similar approach in design,Somewhere along this process, I also redesigned
Graphite’s navigation system across web and mobile. leveraging existing components and workflows efficiently.
Templates
Templates became the most successful addition. Collaborating with the product and engineering teams, I helped define 6-8 templates that covered all five key actions and aligned with the most common automations in our data.
Shortly after launch, template-based automations quickly outnumbered custom-created ones.
Duplicate automations
Some users requested an easier way to create the same automation across all their synced repositories. Our solution was to introduce a duplication feature for automations.
Adding GIFs
Code review is a collaborative process that both shapes and is shaped by team culture. To add some joy to our product, we introduced a feature that lets users automatically add GIFs to PRs. This was powered by Giphy and GIFs are picked based on a user prompt.
Initially, I considered a solution where users could add a GIF link to a comment trigger. While this feature existed, it was hard to find. Since we only had four other actions, I introduced it as a new, more visible option.
Example use case - add a “thumbs up” GIF to approved PRs.
Posting a comment to GitHub
During the initial rollout, every time an Automation acted on a PR, Graphite would post or edit a comment on the PR in GitHub. This was meant to maintain transparency, inform non-Graphite users about the actions taken, and spread awareness of Graphite.
However, users found these comments cluttered and redundant. In response, we made it optional by adding a checkbox in the automation settings, allowing users to decide whether to include the comment.
Applying automation to existing PRs
"Once an automation is created, it applies to all open, published PRs that meet the criteria. This could result in an engineer suddenly being added as a reviewer to hundreds of PRs at once, leading to anxiety for both the creator and the affected engineer. To address this, we introduced an optional checkbox, giving users more control over this process."
Automations apply to open PRs only
Automations evolved with user feedback. While we added bigger features, we continued to iterate on more nuanced feedback as well.
Problem: It wasn’t clear to users that Automations only apply to open PRs. Solution: added copy in-situ explicitly
Layering pricing (and the demise of an original bet)
The initial assumption behind this feature was that non-Graphite users on GitHub would be curious about the automation and click on the comment to create a Graphite account. In hindsight, this expectation was overly optimistic and didn’t materialize.
However, given the high number of new automations created, the positive feedback during user check-ins, and our growing business goals, we decided to price this feature as an add-on to the regular membership. This decision followed a broader overhaul of Graphite's pricing strategy.
Billing page designs
Automation pricing was just one of many add-ons we planned to experiment with, so the design architecture needed to accommodate additional add-ons seamlessly.
We also had to account for various scenarios, including plan types (startup, enterprise, standard) and billing cadences (monthly and annual). The most challenging aspect was calculating costs across different billing cycles while ensuring the copy remained clear and accurate.
Pausing automations
With pricing being introduced, an obvious fear is churn, and in this context that means users deleting automations. There were two solutions proposed –
Ability to store and revive deleted automations
Pause automations (not billed for)
Pausing is a relatively clearer and more intuitive experience, compared to deleting and reviving. While the UI for this was a fairly simple task, complexity arose in the billing details.
For example, if an automation was created by a member of Team A, paused and then unpaused by a member of Team B, do we bill revive billing in Team A or move it to Team B.
This opened a can of worms around automation “ownership”. To solve for several cases like this, I looked at user data and noticed that 80% of automations were created for the benefit of the team and thus billing should belong to the team that creates or unpauses as automation.
Pause-unpause or Pause-resume?
There were a lot of opinions around nomenclature. My eventual explanation was this –
“Resume feels like it's going to pick up from where it left off, i.e. start running on all open PRs dating back to the moment the automation was paused. That's not true. Unpausing skips an entire period of time and starts running on currently open PRs. It's a subtle difference, but a critical one. Let's go with Unpause. ”
Impact
This case study is ongoing, but so far, the number of automations has remained steady even after the introduction of pricing.
Positive user reception
Number of automations growing fast and steady (Graphite’s fastest growing feature)
Several enterprise orgs created multiple automations
Users can immediately identify workflows they’d like to automate, eg: codeowner and reviewer assignment
Users constantly requesting new capabilities and sharing other workflows they’d like to automate
Some love from the team as we worked through this project
