Originally designed in Switzerland, Scala was released to the public in 2004 to address Java’s famous verbosity, its focus on object-oriented programming and other issues. Since then, the language has grown out of its status as a niche skill favored by hobbyists and is now used by startups large and small to build a wide range of tools and services.
What is Scala Used for?
- Data pipelines
- Microservices
- Video transcoding systems
- Real-time information processing
- Ad-serving
- Data processing
- Communications platforms
- Big data projects
With a number of tech companies looking for engineers with Scala skills, we decided to investigate what prompts them to choose functional programming over more traditional object-oriented languages. We spoke to techies at seven tech startups to learn why they have adopted functional programming, and the challenges and benefits associated with it.
1. WHAT HAVE YOU BUILT USING SCALA, AND WHAT CRITERIA DO YOU USE WHEN DETERMINING WHETHER YOU’LL USE IT IN ANY GIVEN CIRCUMSTANCE?
ESENTIRE
TIM STEINBACH, SENIOR SOFTWARE DEVELOPER
By combining artificial intelligence technology with professional experience, eSentire builds cybersecurity solutions designed to hunt for, investigate and respond to threats in real time. The company opened its Seattle office after acquiring local startup Versive, which used artificial intelligence to search for intruders within a system.
Senior Software Developer Tim Steinbach said the company has used Scala to build big data pipelines at the core of their product.
When it comes to deploying high-quality, dependable software, functional programming in Scala has proven itself to be invaluable in our environment. We have access to the wealth of the JVM ecosystem while using a high-velocity, high-quality programming language. There are very few ecosystems in the functional programming community that provide us with the tools and libraries required for our day-to-day business.
CURALATE
MIKE COOK, DIRECTOR OF ENGINEERING
Curalate helps businesses with their marketing efforts through building and engaging social media followers, stimulating the production of user-generated content, driving traffic to websites and helping to grow influencer networks. The company is headquartered in Philadelphia, and has offices in New York, London and downtown Seattle.
Director of Engineering Mike Cook says his team enjoys the conciseness of coding in Scala.
We use Scala for everything server-side at Curalate, including building and deploying hundreds of microservices. These range from very high-scale stream listeners responsible for processing social media posts or web metrics events (at over one billion per week) in real time, down to infrequently occurring event triggers where we build small serverless Scala apps triggered by AWS lambdas.
HIYA
NODIR YULDASHEV, PRINCIPAL SOFTWARE ENGINEER
Hiya uses a highly complex set of technologies in its ongoing defense of smartphones against spam and robocalls. By integrating with smartphone providers, Hiya is able to tell users who is calling them and allow them to block unwanted calls.
For adopting Scala within an organization, Principal Software Engineer Nodir Yuldashev recommends starting with smaller coding projects that will ship.
I have built a variety of systems using Scala, ranging from consumer-facing web applications to petabyte-scale big data platforms. The one that stands out is a live digital video transcoding system which connects news broadcasters and local government agencies to thousands of traffic cameras across the country.
There are a number of factors that help to determine which functional language to choose for a given project, starting with strong community support on platforms like StackOverflow and GitHub. Being able to find an answer to a specific challenge in the language — and not having to reinvent the wheel — is a huge productivity boost.
AXON
TAMAS WEISZ, SENIOR SOFTWARE ENGINEER
Axon started life as TASER International and builds a range of software and hardware tools designed for use by law enforcement and public safety institutions. These include everything from TASER devices to body cameras to digital evidence management systems.
Senior Software Engineer Tamas Weisz said his team benefits from bringing a functional programming mindset even when coding in non-functional languages like Java.
One of Axon’s products is Evidence.com, which allows our customers to access and manage their digital evidence and manage their Axon devices. A large part of the backend services for this system is written in Scala, including real-time information processing from Axon body cameras worn in the field, AI training, analytics, reporting, core evidence workflow and access control logic, as well as services used by other Axon teams such as user and session management.
While we don’t widely use other languages that one would commonly think of as “functional programming,” some functional principles (immutability and map-filter-reduce, for example) have become second nature for us, so lots of our other codebases end up with similar code. In particular, lots of our front end JavaScript code is written in a somewhat functional style — we use React, which encourages this style anyway.
ADCOLONY
JOSH CONNER, DIRECTOR OF ENGINEERING
AdColony’s mobile advertising and marketing platform reaches around 1.4 billion people worldwide. Headquartered in Los Angeles, the company was one of the earliest app developers on iOS. The company’s local office is located in Bellevue.
Director of Engineering Josh Conner said he generally hires engineers who are new to Scala, and as such must anticipate his staffing needs before they become acute.
Almost all of our ad-serving and data-processing backend services are written in Scala. We have a fair number of folks who came from the JS world, so we’ll write quick one-off things in node.js sometimes. However, we’ve found that our inflection point where we start to wish we’d written something in Scala is less than 100 lines of code, so more Scala than anything else these days — especially now that sttp has a curl-based backend for Scala Native.
PORCH
JIM MCCABE, STAFF ENGINEER
Porch’s simple interface belies complex technology that assesses the needs of homeowners based on a simple questionnaire, then connects them with service providers that can repair, maintain or add to their home. Those services include everything from garden mulching and lawn mowing to ceiling fan installation and deck repairs.
Staff Engineer Jim McCabe said Porch uses Scala for many of its back end services.
Some of our backend Scala services include our communication platform, which manages email, SMS and web-based chat; our real-time phone call routing platform; our event scheduling system; our file storage platform; our partner-facing API layer; and our project service, which manages the lifecycle of a project between a homeowner and pros.
We love Scala as a general purpose back end language. The main criteria is, “is this a backend service? Great, let’s use Scala instead of Java.”
NORTAL
NATHAN SKONE, SOLUTIONS ARCHITECT
Estonian software company Nortal builds a wide range of tools and services for enterprise and mid-market clients. The company arrived in the Pacific Northwest with its acquisition of Kirkland-based Dev9 last year, and maintains that office to focus on building custom cloud and software products.
Because Nortal works with a variety of customers, Solutions Architect Nathan Skone said his company must consider the practicality of adopting Scala-based tools within a client organization.
We have built some really cool Apache Spark data pipelines that use machine learning algorithms to remove human bottlenecks from our clients’ business processes. As consultants, we consider ease-of-adoption and our clients’ skill sets when we are choosing a language to use. Scala is often a perfect fit for our big data projects.
2. WHAT BENEFITS AND CHALLENGES DOES YOUR TEAM SEE IN USING A FUNCTIONAL LANGUAGE LIKE SCALA?
ESENTIRE
TIM STEINBACH, SENIOR SOFTWARE DEVELOPER
The entry barrier into functional programming as a whole is still surprisingly high. Most developers have had very limited exposure to proper functional programming and are hesitant to explore its benefits. We have been lucky to have a group of developers and stakeholders that were excited about the journey, and convinced of the eventual benefits.
During the entire time that eSentire has had Scala applications in production, we have not had any major outage of said applications — and we largely attribute this to the transparency and “code cleanliness” of principled functional programming. The Scala community has created a rich set of tooling to ensure adherence to proper conventions and quality guidelines.
CURALATE
MIKE COOK, DIRECTOR OF ENGINEERING
Our team loves how concise and compact code written in Scala can be. More concise code means that it’s easier for developers to see the ideas in the code in their entirety more often. Concepts that would otherwise need to be pulled into a separate method can often be articulated by a single line of functional code. Scala’s more powerful language features and elegant syntax make it easier to spot defects in our code since there’s less context to read within.
On the challenges front, it’s a lot less often that we come across software engineering candidates with Scala or FP experience than the more common imperative programming experience.
The conciseness of the language comes to mind as one of the major benefits. Being able to express large algorithmic meaning and not having to write repetitive boilerplate code makes teams productive.
Managing dependency versions is a common challenge across different programming languages. The issue is prominent with Scala, where you have to account for different Scala versions as well — especially during Scala version upgrades.
AXON
TAMAS WEISZ, SENIORE SOFTWARE ENGINEER
Scala’s rich type system allows us to succinctly and precisely express the state of the entities for any situation. For example, sealed traits are great for safely describing situations where a limit exists on some kind of value versus the value being unlimited. In other languages, the options might have been to use magic values or null.
Immutability is a useful tool as well. Immutable locals help reason about the code locally, while case classes and immutable collections help ensure that no surprises happen to the data as it travels down and up the call chain. I think everyone who had ever had to debug a rogue setter appreciates how much this can simplify things!
Data processing with maps, filters and reducers helps remove lots of the cruft that exists in the imperative counterparts of these idioms, and clarify the intent of the developer. Less cruft means less opportunities to make a mistake, and focusing on the core of the data transforms generally makes understanding the code easier.
ADCOLONY
JOSH CONNER, DIRECTOR OF ENGINEERING
Types and a good type system are just the best. We had some organizational churn in the beginning of 2018, and inherited about 100,000 lines of Scala code with very little test coverage — but thanks to the Scala compiler having our back, we’ve been able to be very ambitious in refactoring and back-filling test coverage while still delivering stability and features for the company. Library support is also generally quite good: circe, Finch, PureConfig, kantan.csv, Monocle, ScalaCheck, Monix and refined — to name just a few — have all been a pleasure to work with.
As for challenges, our experience in hiring has been trickier than for more “mainstream” languages. We mostly hire developers who are new-ish to Scala but eager to learn, and so have had to invest more in the “long haul” — hiring a little before we actually need more people, building a team culture that is supportive of some learning and exploring.
PORCH
JIM MCCABE, STAFF ENGINEER
The benefits of Scala accumulate over time, as it slowly changes the patterns we use to solve problems. Our first early steps were mostly just trying to avoid clumsy Java syntax, but gradually we started to appreciate other things. At this point, I would say that we’re a good way down the path but not all the way. Our Scala code is asynchronous, uses immutable data structures with a declarative style instead of imperative, and with functional error handling instead of throwing exceptions.
The big challenge is that most new hires don’t have any background in this stuff, and there’s a steep learning curve. We organically learned these things over a few years, but a new hire has to dive into the deep end all at once. It can take a few weeks or even months until they feel totally comfortable.
NORTAL
NATHAN SKONE, SOLUTIONS ARCHITECT
Functional languages like Scala fit big data workflows very well. Functional concepts such as immutability, pure functions and separating program data from behavior help us reason about data transformations and ensure data correctness. One of the main challenges we have faced is training developers that have been taught and raised in an object-oriented programming world.
3. HOW CAN LARGER ORGANIZATIONS IMPROVE THEIR ADOPTION OF FUNCTIONAL LANGUAGES?
ESENTIRE
TIM STEINBACH, SENIOR SOFTWARE DEVELOPER
All of our projects make use of an internal SBT plugin, which propagates code conventions, test coverage goals, linting rules and much more. This has allowed developers to get automated feedback as soon as they write code on whether the code is formatted properly, or if there’s proper test coverage.
The biggest benefit, however, is the mental model that functional programming brings. Once understood, the benefits of a simpler, constrained, easy-to-follow programming paradigm will make developers more productive. Here at eSentire, we have accepted that it is quicker to produce a quick script in Python or Bash. But with principled Scala, the maintenance cost and confidence that the application can withstand error cases, invalid data and high throughput has certainly paid off.
CURALATE
MIKE COOK, DIRECTOR OF ENGINEERING
Invest in infrastructure to support your team. A good set of core libraries and service scaffolds has made for a fantastic developer experience here at Curalate. This drums up enthusiasm within the team and encourages further learning and adoption of the language and its concepts.
HIYA
NODIR YULDASHEV, PRINCIPAL SOFTWARE ENGINEER
In larger organizations, there are multiple barriers whenever you transition to a new technology or language. In those scenarios, I always advise teams to start with code that will not ship. For example, start with writing unit and integration tests with Scala in traditional Java projects. That helps build confidence in management level, and also helps developers get comfortable with the language.
AXON
TAMAS WEISZ, SENIOR SOFTWARE ENGINEER
It’s possible to reap some of the benefits of the functional principles in a non-functional language. If a project decides to go with Java instead of Scala, you can still bring the functional mindset with you and stick to immutable data classes and collections (Lombok and Guava are two libraries that I found useful in the past), use stream processing instead of hand-written for loops, or use lightweight lambdas instead of fully-fledged function classes.
The important thing is to keep things to a level that the entire team is comfortable with, because ultimately functional programming is just a means to help write better software, not the end itself.
ADCOLONY
JOSH CONNER, DIRECTOR OF ENGINEERING
We had strong top-down buy-in originally, which is kind of like the “easy mode” version of adding new things to your tech stack. Beyond that, Scala’s a big language, so having a vision of “this is how we write Scala” — and then writing it down using a combination of a scalac flags, formatter rules and a style guide — has been super valuable.
How do you use implicits? How do you inject dependencies? Keep a list of “blessed” libraries for common tasks, like how to do data interchange and which test runner/libraries you use. That kind of thing. We don’t have any secret sauce — mostly, like many things, you just have to show up and do the work — and we’re definitely still learning every day.
PORCH
JIM MCCABE, STAFF ENGINEER
This happened organically at Porch, where a few early adopters were able to get a skeletal system running. The enthusiasm spreads before long, and as our skills grew we also fine tuned that platform.
Code reviews also play a big role here at Porch, and that continues to be a big factor because it is a natural place to teach and learn and grow. We also have a Scala chat room where people ask questions throughout the day. If a question can’t be answered in that forum, people frequently pair up to solve the problem together.
NORTAL
NATHAN SKONE, SOLUTIONS ARCHITECT
It is critical to focus on more than just language syntax when teaching a functional language like Scala to developers. Developers can end up trying to apply their OOP techniques instead of taking advantage of the functional benefits of the language. For this reason, we are careful to teach general functional concepts and skills along with the specific language syntax.
