Every industry has its own use for software development, from cybersecurity to banking to medicine. And with more opportunities come even more diverse challenges.
So what does an engineer do when a problem arises? Take, for example, a small team’s platform growing a user base bigger than their system can handle.
Like any good experiment, hurdles exist to be overcome through trial-and-error and putting skills gained along the way to good use. We spoke to two engineers from different backgrounds about their most recent projects and exciting breakthroughs that rose up from the ashes.
At WHOOP, Staff Technical Lead Bobby Johansen is using the power of biometric data to measure an individual’s performance levels and reach optimum fitness efficiency on their healthtech platform. Keeping track of one user’s data might seem manageable, but with devices around the globe, the team at WHOOP has learned how to divide and conquer.
What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?
WHOOP ingests biometric data from users all around the world every second of the day in order to provide real-time insights into your body’s performance. As you can imagine, this is an incredible amount of data to receive and prepare for consumption by the applications, which turn data into insights for our end users.
At the time, we were a small team dealing with a growing user base that was outpacing our existing technology’s capability of handling data at scale. This led to outages, sometimes as frequently as weekly, that would take hours on end to recover from. We had to find a solution that would keep up with our growth, increase stability, and we had to find it fast. As a result, we began evaluating our technology and architecture choices from the ground up.
How did you and your team overcome this challenge in the end?
We concluded that if we split apart our data processing pipeline into multiple applications, it would be much easier to scale. Each application would adhere to the single-responsibility principle and take charge of one specific task. When it came to storing such a vast amount of time series data, we had a lot of pain maintaining it in PostgreSQL. We evaluated other storage mediums but ultimately landed on Cassandra for ease of horizontal scalability and high write throughput.
On the other hand, streaming the dense amount of data from application to application took a few iterations. We were experiencing pain from limits on the amount of data we could publish and consume over streaming medium Kinesis. At the time, AWS had conveniently come out with Amazon Managed Streaming for Apache Kafka (Amazon MSK). We utilized Kafka for a while until we outgrew the throughput it could provide us.
Our last iteration was to move our dense data out of Kafka and into a Kafka cluster of our own, providing 600 plus gigabit throughput to our applications. Our focus on scalability in our design choices created stability across a large piece.
“Our focus on scalability in our design choices created stability across a large piece.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
This challenge really shaped how our team approaches problems of scalability. We have learned to come at them from multiple perspectives taking into account the operational complexity of managing a system as well as the impact that our technology choices have on the stability of our services at scale. The team now approaches every technology choice with a well-rounded lens, weighing each decision carefully so that we can increase maintainability, performance, and stability as WHOOP grows.
Artificial intelligence company MORSE Corp. is teaching its clients a new language to decipher the customer-focused algorithmic solutions they provide. With so many diverse systems performing at once and a floodgate of data regularly pouring in, both their customers and their teams rely on constant communication to keep programs running smoothly.
What’s the biggest technical challenge you’ve faced recently in your work? What made this particular challenge so tricky?
MORSE is rooted at the heart of a vast web of co-contractors, vendors and research institutes. Our role is to further the bleeding edge of capabilities in the computer vision (CV) space for our customers at the Department of Defense. Our work requires multi-domain expertise to integrate the players in our network into a cohesive product.
My team is responsible for architecting systems to run at scale, to run in the cloud, to perform statistical analysis, and to execute experiments/simulations for our clients. Our technical specialties span Python, distributed computing with Spark, and real-time systems engineering.
Constructing these systems has its own set of challenges but one of the more difficult aspects of our job is elucidating the insights the systems provide for our customer. Distilling numerous metrics, observed behaviors, and the implications of detected aberrations in-system for decision makers is a daunting task – but one we’ve gotten good at.
How did you and your team overcome this challenge in the end?
Stemming from the doctrines of domain-driven design, we heavily focus on perpetuating a ubiquitous and consistent language across our programs. A shared language and understanding allows us to communicate effectively with decision makers. We supply simple but thorough elucidation packages that teach readers everything they need to know to navigate the results from our experiments.
Once our clients learn to speak a common language with domain experts, they are then able to set precise goals and targets for the program. These carefully calibrated goals drive vendors to invest in the area that had the greatest impact, ultimately furthering the program’s capabilities.
“The complexity inherent to the nature of our work mandates good communication.”
How did this technical challenge help you grow as an engineer or help you strengthen a specific skill?
The exposure to the systems, technical problem spaces, and people we work with have been transformative; not only for me but for the MORSE team as a whole. The complexity inherent to the nature of our work mandates good communication. Communication is pivotal. My takeaway is communicate clearly, early and often.
