Artificial intelligence was almost exclusively the domain of academic research for decades. In the past ten years, however, machine learning (ML) techniques have finally achieved sufficient effectiveness and practicality for large-scale adoption in companies and institutions. This adoption, however, remains incipient. Most organizations are still in the early stages gaining proficiency in these technologies and growing them at enterprise scale.
The potential for professionals in this area, therefore, is enormous, evinced by the steady increase in ML job openings and courses. Given the proliferation of ML jobs postings out there, what are the roles and positions in the field? What should you study? What will be the future of a career in ML?
Let’s start by examining what makes ML different from other computing fields.
What Is Machine Learning?
Throughout the evolution of the computer, we’ve thought of it as a machine that executes our commands in an exact and infallible way. Computer scientists have focused on algorithms developed via their own analytical skills. Mathematical proofs guaranteed their accurate results.
This approach, while extremely successful, has limitations. There are still many problems too complex for computers because they’re based on skills humans do well (not machines) like identifying objects or understanding natural language.
So we had to abandon a purely algorithmic approach and help computers develop skills in the same way humans do: learning by example.
The behavior of the software no longer depends only on the code, but also on the data we provide during the training process.
Therein lies the fundamental difference between machine learning and other domains of computer science: the behavior of the software no longer depends only on the code, but also on the data we provide during the training process.
This reliance on data creates a set of needs in addition to those we face in traditional computer science; as a result, new roles and careers have also emerged. Let's take a look at the current ML job market.
Applications and Market
Today, there are already many commercial ML applications Some of the most common are:
-
Customer profile analysis to make decisions about granting credit or prioritizing marketing actions
-
Recommendation systems for products, services and content
-
Fraud detection systems
-
Demand forecasting and logistics optimization
-
Natural language analysis for routing or automatic response (chatbots)
-
Image analysis for object and face detection
-
Text recognition and document structure extraction from images
-
Speech to text transcription
-
Translation between languages
-
Screening for medical imaging exams
Some of the markets that use ML most are online services, marketing and financial services, but there's potential in all industries. The biggest bottleneck for more universal adoption today is not the capacity of algorithms or systems, but in identifying and modeling problems, gathering the data needed for training, and integrating the models with systems in production.
An important point is that these fields of application have different degrees of generality or specificity. Applications such as facial recognition, for example, are more likely to be generalized and therefore use models sold as products by technology companies. Natural language processing (NLP) benefits greatly from culture-specific knowledge and training data, so there’s more room for regional development. Problems involving structured data, such as risk analysis, are highly linked to the specific characteristics of a business and are usually developed in a customized way.
Roles and Careers
The ML job market is very new, dynamic and will certainly continue to evolve, but we can already find some common roles in several organizations. These roles aren’t necessarily performed by different people; it's not uncommon to see a single ML specialist performing several of them. That said, as ML-based solutions become more complex and critical to businesses, specialization will increase.
Machine Learning Roles and Careers
- Data Scientist
- ML/MLOps Engineer
- Analyst
- ML Researcher
It's also clear that specialization shouldn’t come at the risk of rigidity; good professionals must always be knowledgeable of adjacent areas to be prepared for career shifts. Below I’ve listed common paths a career can take.
Data Scientist
Data scientist is currently the most common title for those working with ML. The term has a long history but has become popular in the last decade. Basically, data science is the union of computer science with statistics to process, understand and transform large volumes of data. Originally, the role of data scientist focused on descriptive analysis, statistical models and experiment design, but the role naturally extended to the creation of ML models.
Today, many people associate the role of data scientist primarily with machine learning, but analytical work remains very important, and often even more applicable and useful to the day-to-day operations of an organization.
Because of her great analytical skills, a data scientist can naturally move into areas of business analysis, or delve into the computational aspects of her job to transition to a career in software engineering. Data engineering, in particular, has a great deal of overlap with data science.
ML/MLOps Engineer
A machine learning model doesn’t do much by itself. It needs to be integrated with existing systems, both to consume training data and to provide predictions about new data. If we want the model to maintain its performance, we also need to monitor and automate its training, in addition to ensuring it's reproducible.
These and other aspects are closer to disciplines such as data engineering, DevOps and software engineering than to mathematics and statistics, and collectively they’re called MLOps. In many organizations, the role of the ML or MLOps engineer exists to address these aspects.
With the increasing maturity and automation of ML algorithms, this role stands to gain even more prominence. In smaller or less mature organizations, this person can also embrace the responsibilities that would otherwise belong to the data scientist. The reverse is also common, resulting in so-called “full stack data scientists.”
ML engineering is a specialization of software engineering, so a person in that area can transition to other software domains without much difficulty, such as data engineering or backend development.
Analyst
Before making the decision to use machine learning, someone needs to identify, understand and quantify the business problem. In many companies, that responsibility lies with the business analyst, a person who combines knowledge of the business domain with analytical mathematical skills and (increasingly) software development tools.
There’s no specific training for analysts, but these professionals generally have training in an area with a strong STEM component, such as engineering, economics or administration. The scope of an analyst's work is broad and requires comfort with ambiguity. Analysts willing to deepen their technical knowledge can become data scientists or engineers. Another natural transition is to specialize in a business or management domain.
ML Researcher
There’s still a lot of room for developing new machine learning algorithms, which is what ML researchers do. This integrates academia and industry; many of the most important articles feature co-authors from technology companies and universities.
This reality, however, is much more common in countries that are at the forefront of machine learning development, such as the United States, Canada and China. In most other countries private investment in research is much more limited. Those who decide to enter the private sector after pursuing higher education in ML generally assume roles in data science or ML engineering.
What Should I Study?
Since ML as a profession is a relatively new domain, there’s no established consensus regarding knowledge and curricula in formal academic education or professional training. There are, however, a few natural paths for those who want to follow a career in this space.
What Should I Study for a Career in Machine Learning?
- Computer Science and Software Engineering
- Statistics
- Deep Learning
- Humanities
Computer Science and Software Engineering
ML models are software, so knowledge of computer science and software engineering are necessary foundations for working with these technologies. In particular, deep learning (neural networks with complex structures, which caused the current explosion of ML applications) emerged within the domain of computer science.
Most of the practical bottlenecks for the evolution and adoption of machine learning are fundamentally engineering and software issues, such as integration, maintenance and scalability. These problems are at the core of ML engineering but data scientists also need a computational knowledge base to be effective in their work.
Statistics
The vast majority of ML algorithms need a large amount of data to train (and the more data, the better the result). This means that working with massive amounts of data becomes a fundamental skill, and it's often the most time-consuming step in the process. The best tool ever developed to understand and transform data is statistics, and that brings us to an important point: statistics is to machine learning what Boolean logic is to traditional computing.
ML is a fundamentally statistical technique, so no model is 100% percent accurate (if you developed a model like that, you probably made a mistake, or the problem doesn't need ML to be solved in the first place).
Statistics are not only essential for manipulating training data, but the discipline is even more critical for understanding an ML model’s results and its effects when applied in practice. In addition, many models widely used in practice, such as logistic or linear regression and decision trees, have origins in statistics.
Interestingly, many people embark on careers in machine learning and data science from backgrounds in science, such as physics, neuroscience or mathematics. This makes a lot of sense when taking into account all the techniques used in these disciplines, which aim to extract knowledge from mountains of experimental data. The scientific process has a lot in common with machine learning, a viable alternative for those who don’t want to pursue a career in academia.
Deep Learning
The current boom in machine learning came from the last decade’s advances in deep learning techniques, especially those in neural network architecture. Despite being part of computer science, deep learning is a vast, complex and constantly evolving domain, and therefore deserves specific attention.
Many people, when thinking about studying machine learning, are actually imagining becoming proficient in deep learning techniques. These techniques, however, aren’t always the most suitable for all problems. Deep learning is crucial to solve problems involving unstructured data (e.g. images) or natural language processing (NLP), but problems involving structured data can often be solved more efficiently using simpler, cheaper algorithms.
There's a growing trend to reuse existing models through transfer learning of models pre-trained for more general problems, or using AutoML tools, for which you only need to provide the data and the target. Therefore, broadening the low-level knowledge of deep learning techniques is more worthwhile for people who intend to follow a path in research and advanced applications. For those who want to focus on immediate practical applications, understanding the principles behind these algorithms is still important, but keeping up to date with the latest and most advanced techniques may not necessarily be your highest priority.
Humanities
Technology must always exist to serve human needs, but sometimes it's easy to lose sight of that fact when we get excited by technical challenges or breakthroughs. As technology becomes more powerful and advances into increasingly complex domains, it becomes imperative to keep the human factor in focus. Otherwise, we run the risk of solving the wrong problem or making it worse.
What’s more, as some of lower-level concerns become more automated, ML professionals tend to be more valuable if they can understand the interactions between their systems and people who will use or be affected by them.
One way to avoid getting too lost in the technical maze is to study some disciplines that deal with the nature of human experience, such as philosophy, psychology or social sciences like sociology. If you're anything like me, you may be thinking, “But I chose a technical career because I don't like dealing with people!” You probably think that, like I did in the past, because you think interacting with others is difficult or unpredictable. Studying a little psychology, for example, helps a lot in this regard, and you may even end up enjoying it. Courses in the humanities will also give you the tools you need to communicate effectively in writing and present your findings to a group of stakeholders!
Domain-Specific Knowledge
Machine learning problems don't happen in a vacuum. Instead, they’re typically closely linked to the business, science or other field in which you’re working. Therefore, having specific knowledge of the domain in which you operate will be extremely valuable. For example, you’ll want to know basic economics when working in the financial market, the foundations of mechanical engineering when you need to optimize industrial processes, or pedagogy for educational applications.
Today, the most common scenario is that you absorb some of this knowledge in practice, as needed, while you interact with specialists through teamwork. But there are also many who take the opposite journey and learn ML techniques to solve problems in their area of expertise.
As ML techniques become more flexible and automated, the tendency is for this type of specialization to become more and more common. To work with natural language models, for example, it's very useful to know linguistics, or have medical training in order to develop diagnostic models.
Ethics
Today, technology already has an enormous impact on all areas of society, from entertainment to politics, and, of course, on the labor market. With ML's ability to tackle new problems, its influence will only grow. For this reason, ethics is no longer a distant academic abstraction for technology professionals, but a central part of our work.
In ML, specifically, there's the issue of bias. Since ML needs to use existing data to guide model training, it's very easy for them to reproduce the asymmetries of the society in which they are inserted, such as sexism and structural racism. Tackling bias isn’t easy and sometimes the only solution to avoid perpetuating an asymmetry may be to discard a whole set of data, or even decide that the problem should not be solved with ML. Professionals in the field will increasingly need to prepare themselves to participate in discussions and strive to make ethical decisions.
What About the Role of Traditional Computing?
One thing we shouldn't forget is that ML isn’t a substitute, but a complement to traditional computing, an extremely relevant and rapidly evolving field in its own right. Having at least a basic knowledge of ML will be necessary for professionals of the future, but not everyone will need to specialize in it.
The only certainty we can have about the future of the AI job market is that it’ll continue to change, and quickly. Many of these roles didn’t even exist a few years ago and this process will continue. Some new roles, which we can’t even imagine today, will emerge, and others will cease to exist. To prepare, it's always worthwhile to have a solid foundation in fundamentals such as computer science, statistics and software engineering, while keeping up with the constant evolution of the field.
Finally, it’s critical we don’t lose sight of the human factor. The best way to predict the future is to create it, and our responsibility in that future continues to grow.
