Role Overview:
This role will be responsible for conducting data ingestion and platform management on the Databricks platform. This role requires a deep understanding of:
- Data Lake ingestion processes and best practices
- ETL/ELT implementation
- CI/CD
- System integration tools
- Data pipeline management
About the role:
- Ingest data from a variety of source systems and tailor ingestion approaches on a per-system basis
- Manage, maintain, and oversee ETL/ELT pipelines on the Databricks platform
- Optimize data pipelines for scalability and speed
- Document ingestion and integration flows and pipelines
- Use Airflow to schedule and automate ingestion jobs
- Manage metadata and master data in technical data catalog
- Ensure ELT/ETL design meets required security and compliance guidelines, and ensure PII management, flagging and risk assessment during ingestion
- Maintain ETL/ELT pipeline infrastructure and implement automated monitoring strategies
- Ensure adherence to SDLC best practices
About you:
- Bring 3+ years of experience in data engineering, ingestion pipelining, and ETL/ELT
- Hold a bachelor’s degree in computer science, engineering, statistics, or related field
- Have hands-on experience with and understanding of the following:
- Spark/Scala
- SQL
- Python/PySpark or similar programming language
- Databricks
- Unity Catalog
- ETL/ELT development, monitoring and pipelining using tools such as Apache Airflow
- Ingestion tools such as Dell Boomi
- Data quality guidelines
- CI/CD pipelines
- Agile
- Git and version control
About Encora
Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.
At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
Top Skills
What We Do
Headquartered in Santa Clara, California, and backed by renowned private equity firms Advent International and Warburg Pincus, Encora is the preferred technology modernization and innovation partner to some of the world’s leading enterprise companies. It provides award-winning digital engineering services including Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering. Encora's deep cluster vertical capabilities extend across diverse industries, including HiTech, Healthcare & Life Sciences, Retail & CPG, Energy & Utilities, Banking Financial Services & Insurance, Travel, Hospitality & Logistics, Telecom & Media, Automotive, and other specialized industries.
With over 9,000 associates in 47+ offices and delivery centers across the U.S., Canada, Latin America, Europe, India, and Southeast Asia, Encora delivers nearshore agility to clients anywhere in the world, coupled with expertise at scale in India. Encora’s Cloud-first, Data-first, AI-first approach enables clients to create differentiated enterprise value through technology
