At marketing automation platform Klaviyo, the data science team considers the customer at every turn and is open to any and all feedback.
Recently, the team launched a three-in-one, AI-powered tool called the Subject Line Assistant to help customers craft the ideal call-to-action to get marketing emails opened. Users have the option to generate subject lines, analyze performance data from previous campaigns, and learn best practices. But the interesting part is where the development process began.
“Customers were the primary focus from the very beginning,” said Sarah Fathelbab. As the product designer on the project, Fathelbab introduced an empathetic lens to the development process, interviewing clients to understand exactly who would be using the tool and what problems it could solve.
“It helps to understand the story behind their role, the company they work for or how they chose to start their company,” said Fathelbab. Depending upon whether the user is a working parent or an entrepreneur, daily work flows often look different — and that specification plays into both the machine learning and UX pieces of the tool’s design. “When you hear how much it actually means to them to come up with a good subject line, you understand that that’s their livelihood.”
Radical candor with clients translates to the collaborative nature of the company, and the synergy that happens among teams during product development. “Our engineers and data scientists do not operate in a silo,” said Senior Product Manager Royce Yap. “They are actively involved in the product development process and are made to think like customers when building for them.”
That’s why they understand that email subject lines carry a lot of weight in the success of a campaign, and what might seem to a marketer like a simple task could be a far more time-consuming, daunting feat for solo business owners already wearing a number of hats. “It’s almost like writing the title of a paper. It’s the last thing you want to touch because you know it should say a lot about what you’re going to read,” said Fathelbab. “It’s the same thing when it comes to the subject line, and that was a huge revelation on our part. It’s not random text copy. It should be very meaningful.”
And that quest for personalized meaning translated to a more nuanced understanding of the role that Subject Line Assistant should play in the process. “The point of the tool is to be a source of inspiration, which allows you the flexibility to tweak wording,” said Data Scientist Josh Villarreal. In fact, re-wording was a specific behavior that the team picked up on as they reviewed how customers were using the assistant.
Between Villarreal’s focus on machine learning elements, Fathelbab’s understanding of customer needs and pain points, product manager support from Yap, and the front-end and back-end expertise of Senior Data Scientist Philip Ekfeldt, Klaviyo’s Subject Line Assistant project certainly had a passionate team behind it. Built In sat down with the aforementioned team members to learn more about how the product development process operates in tandem with the company’s customer-first mindset.
“There is so much at stake with a single subject line that this often leads to writer’s block.”
Where did the ideation process begin on the Subject Line Assistant project?
Fathelbab: The idea came from talking to customers and learning what their general pain points were when it came to crafting good subject lines. The more we interviewed, the more problems we identified. It was a very customer-first approach.
Yap: Klaviyo customers struggle with writing subject lines for a variety of reasons. Most often, they either lack inspiration for something that is attention-worthy or they are not sure of what works best for their audience. There is so much at stake with a single subject line that this often leads to writer’s block. With that in mind, we decided that having a one-dimensional subject line tool was not going to work for a customer base with varying needs. That’s why we designed a multifunctional assistant that is able to generate customized marketing content while allowing users to reference success metrics relating to past subject lines.
How did working with team members from different areas of expertise enhance the final product?
Ekfeldt: Cross-functionality works well within the data science team. No matter what your role is, you get to work end to end on features and products. You can better understand what everyone’s doing, which makes it easier to collaborate and offer insight and feedback. It helps make the whole product come together.
Villarreal: What’s really great about Klaviyo is that everyone’s happy to help. To make sure that the generator was working as expected, we released surveys asking people whether this group of subject lines was better than that group or to pick their favorite subject line out of the bunch. It was valuable to have input from not only data scientists and engineers, but also from people who were working on different aspects of the product.
Fathelbab: The highly collaborative environment is really nice. You’re not carrying all the weight on your shoulders. For example, I was doing the interface, but I got to sit in with Josh as he was working in the back end, which gave me a chance to understand how he was doing the AI portion. You get to build it all together. It’s very cool.
“It was valuable to have input from not only data scientists and engineers, but also from people who were working on different aspects of the product.”
Josh, what machine learning elements did you need to consider on this project?
Villarreal: Machine learning is the bread and butter of the generator piece of the Subject Line Assistant. ML is generally an optimization problem where you want to get the best possible performance in one area, subject to some constraints. In this case, we weren’t necessarily looking to use it to come up with the one perfect subject line for these customer inputs; we wanted to come up with a set of subject lines that were diverse enough to give customers ideas and alleviate their writer’s block. ML was also folded into how we determined which subject lines were better than others and how we evaluated the constraints of the problem. If a subject line is witty and references a sale but it’s over 70 characters, then it’s a lost cause, because when it shows up in your inbox you’ll see ellipses. We had to make sure we were paying attention to the customer’s tone and content without it being too long. It required a little bit of finicky work.
Plagiarism was also a big hurdle. We needed to make sure that the model wasn’t coming up with the same subject line twice. If you’re having a 99 percent off liquidation sale, there are only so many ways to say that. The biggest weapons we had to combat that were natural language processing and our own judgment.
Philip, how did you ensure the Subject Line Assistant was designed to operate at scale?
Ekfeldt: Before launching to all of our users, we wanted to make sure the experience was stable and fast. For the generator, we got the speed down to two to three seconds per request. We also wanted to make sure that the past campaign performance page queried quickly. We have a lot of data, and some companies have a lot of campaigns to query for. When we did a proof of concept, depending on the company, it took 20 to 30 seconds. We were able to optimize that to take one second, which was a great improvement.
Sarah, what role did design play in the current UX, and how are you keeping future features in mind?
Fathelbab: It goes back to understanding the customer’s subject line creation process. You’re introducing something that wasn’t part of their workflow before. How does this new tool fit into that process? We had to make sure that the design didn’t inundate the functionality to the point where it makes the customer feel like they don’t know what to do. When we were designing the generator, each tool went through numerous iterations. We needed to be conscious of the customer’s day-to-day and ensure each feature was helpful enough to include.
There’s a tricky balance to strike. You want to give the user more functionality, but you don’t want to give them so much that it takes up too much time or disrupts their flow. You also want to ensure that what you’re adding to the tool is intentional. Did a customer say that they needed this, or do we think that they’ll need it? After a certain period of time, it’s worth asking, “How has this feature helped?” But my favorite question to ask is, “If you could wave a magic wand and design this however you want, what would it look like?” The first thing that pops into their head might be something we didn’t think of.
“You want to give the user more functionality, but you don’t want to give them so much that it takes up too much time or disrupts their flow.”
Royce, how does Klaviyo’s solution stand out compared to what the rest of the market offers?
Yap: Based on our research, we are the first among our direct competitors to allow end users to access advanced AI technology within their content creation workflow. Standalone AI-powered copywriting tools are proliferating online; however the most advanced generative language technologies have not yet made their way into marketing platforms. Technology moves fast though, so I would not be surprised to see the rest of the market launch something similar.
How did you land on the solution that ultimately went on to launch?
Ekfeldt: It came from a lot of experimentation. We had the pain points from customers, so a lot of last year was spent iterating on our ideas and folding them into the final product. We first tested the three solutions we had as separate features, but having different options for different types of users worked well in parallel. When we figured that out, we said, “Why not combine them?”
Fathelbab: We also looked at the data, which informed our placement of the three tools. Rearranging them allowed us to surface the feature that would be the most useful for the largest body of customers.
Villarreal: AI models for text generation are, frankly, really cool. We’ve talked about writer’s block and how Klaviyo customers need a little bit of inspiration or a place to start with writing subject lines, and like Philip mentioned, we had been experimenting with AI models early on. This was a perfect, wholesome application to put an exciting piece of technology in front of customers.
