Important Information
Location: Chennai
Experience: 8+ years
Job Mode: Full-time
Job Summary:
Lead Data Eng with Scala, Spark, Pyspark & Databricks
Skills
Job Title: Data Engineer (Scala, Spark, Pyspark, Databricks)
Responsibilities:
- Design, develop, and maintain robust and scalable data pipelines using Apache Spark and Scala on the Databricks platform.
- Implement ETL (Extract, Transform, Load) processes for various data sources, ensuring data quality, integrity, and efficiency.
- Optimize Spark applications for performance and cost-efficiency within the Databricks environment.
- Work with Delta Lake for building reliable data lakes and data warehouses, ensuring ACID transactions and data versioning.
- Collaborate with data scientists, analysts, and other engineering teams to understand data requirements and deliver solutions.
- Implement data governance and security best practices within Databricks.
- Troubleshoot and resolve data-related issues, ensuring data availability and reliability.
- Stay updated with the latest advancements in Spark, Scala, Databricks, and related big data technologies.
Required Skills and Experience:
- Proven experience as a Data Engineer with a strong focus on big data technologies.
- Expertise in Scala programming language for data processing and Spark application development.
- In-depth knowledge and hands-on experience with Apache Spark, including Spark SQL, Spark Streaming, and Spark Core.
- Proficiency in using Databricks platform features, including notebooks, jobs, workflows, and Unity Catalog.
- Experience with Delta Lake and its capabilities for building data lakes.
- Strong understanding of data warehousing concepts, data modeling, and relational databases.
- Familiarity with cloud platforms (e.g., AWS, Azure, GCP) and their data services.
- Experience with version control systems like Git.
- Excellent problem-solving and analytical skills.
- Ability to work independently and as part of a team.
Preferred Qualifications (Optional):
- Experience with other big data technologies like Kafka, Flink, or Hadoop ecosystem components.
- Knowledge of data visualization tools.
- Understanding of DevOps principles and CI/CD pipelines for data engineering.
- Relevant certifications in Spark or Databricks.
About Encora :
Encora is the preferred digital engineering and modernization partner of some of the world’s leading enterprises and digital native companies. With over 9,000 experts in 47+ offices and innovation labs worldwide, Encora’s technology practices include Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering.
At Encora, we hire professionals based solely on their skills and qualifications, and do not discriminate based on age, disability, religion, gender, sexual orientation, socioeconomic status, or nationality.
Top Skills
What We Do
Headquartered in Santa Clara, California, and backed by renowned private equity firms Advent International and Warburg Pincus, Encora is the preferred technology modernization and innovation partner to some of the world’s leading enterprise companies. It provides award-winning digital engineering services including Product Engineering & Development, Cloud Services, Quality Engineering, DevSecOps, Data & Analytics, Digital Experience, Cybersecurity, and AI & LLM Engineering. Encora's deep cluster vertical capabilities extend across diverse industries, including HiTech, Healthcare & Life Sciences, Retail & CPG, Energy & Utilities, Banking Financial Services & Insurance, Travel, Hospitality & Logistics, Telecom & Media, Automotive, and other specialized industries.
With over 9,000 associates in 47+ offices and delivery centers across the U.S., Canada, Latin America, Europe, India, and Southeast Asia, Encora delivers nearshore agility to clients anywhere in the world, coupled with expertise at scale in India. Encora’s Cloud-first, Data-first, AI-first approach enables clients to create differentiated enterprise value through technology
