The Role
Design, build, and optimize scalable data pipelines and architectures using SQL and AWS big-data technologies. Assemble complex datasets, automate processes, enable analytics and stream processing, secure cross-region data, and support cross-functional stakeholders including data scientists and product teams.
Summary Generated by Built In
Responsible for
expanding and optimizing the data and data pipeline architecture, as well as
optimizing data flow and collection for cross functional teams. The ideal
candidate is an experienced data pipeline builder and data wrangler who enjoys optimizing
data systems and building them from the ground up.
The Big Data Engineer
will support software developers, database architects, data analysts and data
scientists on data initiatives and will ensure optimal data delivery
architecture is consistent throughout ongoing projects.
Responsibilities for Big
Data Engineer
- Create and maintain
optimal data pipeline architecture,
- Assemble large,
complex data sets that meet functional / non-functional business requirements.
- Identify, design,
and implement internal process improvements: automating manual processes,
optimizing data delivery, re-designing infrastructure for greater scalability,
etc.
- Build the
infrastructure required for optimal extraction, transformation, and loading of
data from a wide variety of data sources using SQL and AWS ‘big data’
technologies.
- Build analytics
tools that utilize the data pipeline to provide actionable insights into
customer acquisition, operational efficiency and other key business performance
metrics.
- Work with
stakeholders including the Executive, Product, Data and Design teams to assist
with data-related technical issues and support their data infrastructure needs.
- Keep our data
separated and secure across national boundaries through multiple data centers
and AWS regions.
- Create data tools
for analytics and data scientist team members that assist them in building and
optimizing our product into an innovative industry leader.
- Work with data and
analytics experts to strive for greater functionality in our data systems.
Qualifications for Big
Data Engineer
- 5+ years of
experience in a Data Engineer role
- Graduate degree in
Computer Science, Statistics, Informatics, Information Systems or another
quantitative field.
- Advanced working
SQL knowledge and experience working with relational databases, query authoring
(SQL) as well as working familiarity with a variety of databases.
- Experience building
and optimizing ‘big data’ data pipelines, architectures and data sets.
- Experience
performing root cause analysis on internal and external data and processes to
answer specific business questions and identify opportunities for improvement.
- Strong analytic
skills related to working with unstructured datasets.
- Build processes
supporting data transformation, data structures, metadata, dependency and
workload management.
- A successful
history of manipulating, processing and extracting value from large
disconnected datasets.
- Working knowledge
of message queuing, stream processing, and highly scalable ‘big data’ data
stores.
- Strong project
management and organizational skills.
Experience using the following software/tools
- Experience
supporting and working with cross-functional teams in a dynamic environment.
- Experience with big
data tools: Hadoop, Spark, Kafka, etc.
- Experience with
relational SQL and NoSQL databases, including Postgres and Cassandra.
- Experience with
data pipeline and workflow management tools: Azkaban, Luigi, Airflow, etc.
- Experience with AWS
cloud services: EC2, EMR, RDS, Redshift
- Experience with
stream-processing systems: Storm, Spark-Streaming, etc.
- Experience with
object-oriented/object function scripting languages: Python, Java, C++, Scala,
etc.
Skills Required
- 5+ years of experience in a Data Engineer role
- Graduate degree in Computer Science, Statistics, Informatics, Information Systems or another quantitative field
- Advanced working SQL knowledge and experience with relational databases and query authoring
- Experience building and optimizing big data pipelines, architectures and datasets
- Experience performing root cause analysis on data and processes to answer business questions
- Strong analytic skills related to working with unstructured datasets
- Build processes supporting data transformation, data structures, metadata, dependency and workload management
- History of manipulating, processing and extracting value from large disconnected datasets
- Working knowledge of message queuing, stream processing, and highly scalable big data stores
- Strong project management and organizational skills
- Experience with big data tools such as Hadoop, Spark, Kafka
- Experience with relational SQL and NoSQL databases including Postgres and Cassandra
- Experience with data pipeline and workflow tools such as Azkaban, Luigi, Airflow
- Experience with AWS cloud services including EC2, EMR, RDS, Redshift
- Experience with stream-processing systems such as Storm or Spark-Streaming
- Experience with object-oriented or scripting languages: Python, Java, C++, Scala
- Experience supporting and working with cross-functional teams in a dynamic environment
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
KMR Search Group is an executive search and recruitment firm specializing in the Healthcare, Pharmaceutical, Biotech, and IT Services industries. Operating with a nationwide reach, they help companies identify and place top-tier leadership talent in specialized roles, as seen in their history of conducting high-level searches for positions such as General Managers and Chief Financial Officers across various industrial sectors.
