While technology is the heart of our business, a global and diverse culture is the heart of our success. We love our people and we take pride in catering them to a culture built on transparency, diversity, integrity, learning and growth.
If working in an environment that encourages you to innovate and excel, not just in professional but personal life, interests you- you would enjoy your career with Quantiphi!
Role: Senior Data Engineer - Python, Spark & Databricks
Experience Level: 4-7 Years
Work Shift: 6:00 PM - 3:00 AM IST (US Business Hours Shift)
Work location: Remote
Notice Period: 0-15 days(Looking for immediate joiners or currently serving Notice period)
Role & Responsibilities:
- ETL/ELT Data Pipeline Engineering: Design, build, and optimize scalable real-time and batch data pipelines using Python, SQL, PySpark, Spark Structured Streaming, Delta Live Tables, and Medallion Architecture (Bronze, Silver, Gold layers) for high-performance analytics.Must have hands-on experience working with terabyte-scale (TB) datasets and designing distributed data processing solutions.
- Databricks & Lakehouse Management: Manage Databricks workflows, clusters, and Delta Lake. Implement Unity Catalog for metadata management, fine-grained access control, OAuth-based authentication, and enterprise security compliance.
- Cloud Infrastructure & Migration: Experience working with at least one major cloud platform (GCP, AWS, or Azure). Lead cloud migrations from legacy databases (Oracle, Teradata, SQL Server) to cloud environments (GCS, AWS S3, Azure Data Lake Storage, Snowflake, Redshift).
- CI/CD & DevOps Automation: Build end-to-end CI/CD pipelines using Terraform and GitHub Actions to automate data pipeline deployments, infrastructure provisioning, and proactive monitoring.
- Web Services & Application Scripting: Develop and deploy custom data processing scripts and data-driven APIs using Python and web frameworks like Flask.
- Containerization & Innovation: Utilize Docker for containerized processing and evaluate emerging technologies for data-intensive machine learning tasks.
- Stakeholder Collaboration: Work closely with business stakeholders and cross-functional engineering teams to translate business requirements into technical architectures and data-driven insights.
Skills expectations:
- Core Data Engineering & Big Data: Advanced hands-on experience in Python, SQL, PySpark, and expert-level SQL query optimization for massive datasets.
- Databricks Expertise: Strong proficiency in Databricks functionalities, including Workflows, Unity Catalog, Delta Lake, and Delta Live Tables.
- Cloud Platform Services: Deep familiarity across major cloud environments (GCP, AWS S3/Redshift, Azure ADLS Gen2).
- DevOps & Infrastructure: Solid experience in Terraform, GitHub Actions, Docker, and CI/CD best practices for data platforms.
- Web Frameworks: Practical experience with Python web frameworks such as Flask.
- Collaborative & Agile Mindset: Strong problem-solving, communication, and ownership skills to guide projects and mentor team members in a fast-paced environment.
If you like wild growth and working with happy, enthusiastic over-achievers, you'll enjoy your career with us!
Skills Required
- 4-7 years of data engineering experience
- Advanced hands-on experience with Python, SQL, PySpark, and SQL query optimization
- Experience designing distributed data processing solutions for terabyte-scale datasets
- Strong proficiency with Databricks Workflows, Unity Catalog, Delta Lake, and Delta Live Tables
- Experience with at least one major cloud platform: GCP, AWS, or Azure
- Experience with cloud migration from legacy databases to cloud platforms
- Experience with Terraform, GitHub Actions, Docker, and CI/CD practices
- Practical experience with Python web frameworks such as Flask
- Strong problem-solving, communication, collaboration, and ownership skills
- Ability to guide projects and mentor team members
- Immediate availability or notice period of 0-15 days
Quantiphi Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Quantiphi and has not been reviewed or approved by Quantiphi.
-
Wellbeing & Lifestyle Benefits — Wellbeing initiatives such as monthly meeting-free AMA-Zen Days, health check-ups, and wellness counseling are designed to reduce burnout and support day-to-day balance. Broader wellness programs reinforce both physical and mental health.
-
Flexible Benefits — Remote/hybrid options with flexible working hours provide meaningful autonomy over where and when work gets done. Flexible leave constructs, including sabbaticals and special day leaves, add practical adaptability to the package.
-
Parental & Family Support — Paid parental leave in the U.S., alongside maternity and childcare support, signals solid backing for families. These family-oriented policies integrate with a wider health and wellness focus.
Quantiphi Insights
What We Do
Quantiphi is an award-winning AI-first digital engineering company driven by the desire to solve transformational problems at the heart of business. Quantiphi solves the toughest and complex business problems by combining deep industry experience, disciplined cloud, and data-engineering practices, and cutting-edge artificial intelligence research to achieve quantifiable business impact at unprecedented speed.
