About the Role We are looking for a hands-on Data Engineer (7+years) to build and optimize scalable data pipelines and analytical datasets on the Databricks platform. You will work closely with Analytics/BI, Product, and Business teams to enable data-driven decision-making. Retail / eCommerce domain exposure is a strong plus, along with the ability to translate business needs into reliable and performant data solutions.
Key Responsibilities
• Design, develop, and maintain robust ETL/ELT pipelines using Databricks (Spark) and Python (PySpark). • Develop and optimize complex transformations using SQL (joins, window functions, CTEs, query tuning). • Build curated datasets and data models to support reporting, dashboards, and advanced analytics use cases. • Implement pipeline reliability best practices: data quality checks, monitoring, alerting, and reconciliation. • Optimize Databricks workloads for performance and cost (cluster sizing, partitioning strategies, caching, file formats). • Work with structured and semi-structured data (JSON, CSV, Parquet/Delta) and handle schema evolution. • Collaborate with stakeholders to understand business KPIs and deliver data solutions aligned to retail/eCommerce metrics (sales, orders, returns, inventory, customer cohorts). • Follow engineering best practices for version control (Git), documentation, reusable code patterns, and testing. • Good to have: Support or migrate Alteryx workflows into Python/Databricks pipelines.
Must-Have Skills & Qualifications
• 7-13 years of experience in Data Engineering / Data Warehousing / Big Data. • Strong hands-on experience with Databricks (Jobs/Workflows, notebooks, cluster concepts, Spark tuning fundamentals). • Strong programming skills in Python (PySpark preferred). • Excellent SQL skills, including performance tuning and writing complex analytical queries. • Experience building scalable pipelines and working with large datasets in distributed environments. • Strong understanding of data engineering concepts: ETL/ELT, orchestration, data validation, and observability. • Familiarity with modern data storage formats and practices (Delta/Parquet, partitioning, incremental loads).
Good-to-Have Skills
• Retail / eCommerce domain knowledge (customer behavior, funnel metrics, pricing/promotions, inventory, catalog, order lifecycle). • Alteryx (workflow development, optimization, scheduling, or migration to Databricks). • Experience with Lakehouse patterns and Delta Lake features (e.g., MERGE, OPTIMIZE, Z ORDER). • Experience with orchestration tools (e.g., Airflow, ADF, Databricks Workflows). • Cloud experience: AWS / Azure / GCP (S3/ADLS/GCS, IAM basics, security controls). • CI/CD exposure for data pipelines, code reviews, and automated deployments. Preferred Traits • Strong problem-solving skills and a mindset for root-cause analysis. • Ownership and accountability for production-grade pipelines. • Ability to communicate with both technical and non-technical stakeholders. • Comfort working in fast-paced environments with evolving requirements.
Skills Required
- 7-13 years of experience in data engineering, data warehousing, or big data
- Strong hands-on experience with Databricks Jobs, Workflows, notebooks, cluster concepts, and Spark tuning
- Strong programming skills in Python, preferably PySpark
- Excellent SQL skills, including performance tuning and complex analytical queries
- Experience building scalable pipelines and working with large datasets in distributed environments
- Strong understanding of ETL/ELT, orchestration, data validation, and observability
- Familiarity with Delta or Parquet formats, partitioning, and incremental loads
- Retail or eCommerce domain knowledge
- Alteryx workflow development, optimization, scheduling, or migration to Databricks
- Experience with Lakehouse patterns and Delta Lake features such as MERGE, OPTIMIZE, and Z ORDER
- Experience with orchestration tools such as Airflow, Azure Data Factory, or Databricks Workflows
- Cloud experience with AWS, Azure, or GCP, including storage, IAM, and security controls
- CI/CD exposure for data pipelines, code reviews, and automated deployments
eClerx LLC Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about eClerx LLC and has not been reviewed or approved by eClerx LLC.
-
Healthcare Strength — Feedback suggests U.S. employees have access to medical, dental, and vision coverage, with HSA/FSA options and life insurance included. An Employee Assistance Program also appears in the public benefits profile supporting wellbeing.
-
Retirement Support — Feedback suggests a 401(k) is part of the U.S. package, with some mentions of an employer match. Retirement provision is consistently cited across company profiles and listings.
-
Leave & Time Off Breadth — Feedback suggests paid holidays, sick leave, and PTO are part of the offering, alongside parental leave. Parental leave is also highlighted in public benefits profiles.
eClerx LLC Insights
What We Do
eClerx provides business process management, automation and analytics services to a number of Fortune 2000 enterprises, including some of the world's leading financial services, communications, retail, fashion, media & entertainment, manufacturing, travel & leisure, and technology companies. Incorporated in 2000, eClerx is today traded on both the Bombay and National Stock Exchanges of India. The firm employs 16,000+ people across Australia, Canada, Germany, India, Italy, Netherlands, Philippines, Singapore, Thailand, UK, and the USA.
