The Role
Designs, builds, and maintains scalable ETL/ELT pipelines and distributed data-processing systems. Develops Python, SQL, Airflow, Spark, and Redis solutions for large datasets, with responsibilities spanning data integration, validation, monitoring, observability, performance optimization, troubleshooting, and documentation. Collaborates with BI, Product, Data Science, and Engineering teams to deliver reliable production datasets and data products.
Summary Generated by Built In
Devsinc is looking for a highly skilled Senior Data Engineer with 4–6 years of professional experience to design, build, and maintain scalable data pipelines and processing systems that support analytics, data products, and AI/ML capabilities.
The ideal candidate will have strong hands-on experience with ETL/ELT pipelines, Python, SQL, Apache Airflow, Apache Spark, and Redis, along with the ability to develop reliable and maintainable workflows for large, complex, and continuously growing datasets. You will collaborate with Product, Business Intelligence, Data Science, and Engineering teams to deliver production-ready datasets and high-performance data solutions.
Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines for data ingestion, transformation, validation, and delivery.
- Build, schedule, and orchestrate production-grade data workflows using Apache Airflow.
- Develop distributed data-processing jobs using Apache Spark.
- Write efficient, reusable, and maintainable Python code for data processing, automation, and pipeline development.
- Develop and optimize complex SQL queries for data transformation, analysis, and quality validation.
- Design pipelines capable of processing large-scale structured, semi-structured, and unstructured datasets.
- Implement Redis for caching, high-performance data access, and data-intensive application requirements.
- Integrate data from APIs, relational databases, files, third-party providers, and other internal and external sources.
- Implement data validation, monitoring, logging, error handling, alerting, and pipeline observability.
- Optimize pipeline performance, data storage, processing time, and infrastructure costs.
- Develop reusable data-ingestion and transformation frameworks instead of one-off scripts.
- Troubleshoot pipeline failures, performance bottlenecks, and data-quality issues to ensure timely resolution.
- Collaborate with BI, Product, Data Science, and Engineering teams to deliver reliable, production-ready datasets.
- Establish and maintain data-engineering standards, technical documentation, and development best practices.
Requirements
- Bachelor’s degree in Computer Science, Software Engineering, Data Science, or a related field.
- 4–6 years of professional experience in Data Engineering or a closely related role.
- Strong hands-on experience designing and implementing production-grade ETL/ELT pipelines.
- Strong proficiency in Python for data processing, automation, and data-engineering workflows.
- Advanced SQL skills and a strong understanding of relational databases.
- Practical production experience with Apache Airflow for workflow orchestration.
- Hands-on experience with Apache Spark and distributed data processing.
- Strong understanding of data modelling, transformation patterns, and data-pipeline architecture.
- Experience with Redis, caching strategies, and high-performance data-access patterns.
- Experience processing large datasets and optimizing pipeline and query performance.
- Strong understanding of data quality, validation, monitoring, observability, and pipeline reliability.
- Familiarity with Linux, Git, Docker/containers, and modern software-engineering practices.
- Strong analytical, troubleshooting, communication, and cross-functional collaboration skills.
Preferred Qualification
- Experience with cloud data platforms and object storage services such as AWS, Azure, or GCP.
- Experience working with PostgreSQL, data warehouses, or analytical databases.
- Experience processing geospatial data or large-scale location-based datasets.
- Familiarity with DuckDB, Apache Sedona, Trino, Presto, or similar analytical technologies.
- Experience processing high-volume event, mobility, transactional, or geospatial data.
- Familiarity with CI/CD pipelines and infrastructure-as-code practices.
- Experience supporting data products, analytics platforms, or AI/ML pipelines.
Skills Required
- Bachelor’s degree in Computer Science, Software Engineering, Data Science, or a related field.
- 4–6 years of professional experience in Data Engineering or a closely related role.
- Production-grade ETL/ELT pipeline design and implementation experience.
- Strong proficiency in Python for data processing, automation, and data-engineering workflows.
- Advanced SQL skills and strong understanding of relational databases.
- Production experience with Apache Airflow.
- Hands-on experience with Apache Spark and distributed data processing.
- Understanding of data modeling, transformation patterns, and data-pipeline architecture.
- Experience with Redis, caching strategies, and high-performance data-access patterns.
- Experience processing large datasets and optimizing pipeline and query performance.
- Understanding of data quality, validation, monitoring, observability, and pipeline reliability.
- Familiarity with Linux, Git, Docker or containers, and modern software-engineering practices.
- Strong analytical, troubleshooting, communication, and cross-functional collaboration skills.
- Experience with cloud data platforms and object storage services such as AWS, Azure, or GCP.
- Experience with PostgreSQL, data warehouses, or analytical databases.
- Experience processing geospatial data or large-scale location-based datasets.
- Familiarity with DuckDB, Apache Sedona, Trino, Presto, or similar analytical technologies.
- Experience processing high-volume event, mobility, transactional, or geospatial data.
- Familiarity with CI/CD pipelines and infrastructure-as-code practices.
- Experience supporting data products, analytics platforms, or AI/ML pipelines.
Devsinc Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Devsinc and has not been reviewed or approved by Devsinc.
-
Healthcare Strength — OPD and IPD medical coverage, along with health insurance mentions, indicate a meaningful healthcare offering.
-
Flexible Benefits — Work-from-home options, workation opportunities, and paid leave point to flexible arrangements for where and when work is done.
-
Wellbeing & Lifestyle Benefits — Access to gyms and salons, plus various allowances and occasional trips or events, add lifestyle value beyond base pay.
Devsinc Insights
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
We integrate global leaders in web development with passionate Asian talent to get a unique blend of Quality and Affordability. We are headquartered in California and work consistent eastern and pacific standard hours. We like ad hoc pairing as necessary, TDD, and working with other agencies to make things happen. We contribute to open source projects and genuinely enjoy coding. We are also committed to teaching, and spreading knowledge!
