We are seeking a highly skilled AWS Data Engineer with deep expertise in AWS cloud architecture, big data processing, real-time streaming, and modern data lake technologies. The ideal candidate will have strong hands-on experience in Spark (PySpark), Iceberg, EMR, Starburst/Trino, and event-driven architectures, along with experience building real-time and API-driven data applications who can design and build generic solutions for one of our Fortune 500 Client programs in the realm of Financial Master & Reference Data Management. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.
ResponsibilitiesKey Responsibilities
- Design and implement scalable, secure, and cost-optimized AWS data architectures.
- Develop and maintain ETL pipelines using AWS Lambda and AWS Glue ETL.
- Configure and manage AWS Glue Crawlers, Glue Data Catalog, and schema evolution.
- Build, optimize, and unit test applications on the Apache Spark framework using PySpark.
- Design and optimize data lakes using Apache Iceberg on AWS, including table compaction and Iceberg performance tuning.
- Work extensively with data formats such as Avro, Parquet, JSON, XML, and CSV.
- Orchestrate event-driven workflows using AWS Step Functions and Amazon EventBridge.
- Connect and integrate Starburst from Lambda and Glue ETL jobs for federated querying.
- Implement CI/CD pipelines for automated testing and deployment.
- Perform unit testing using PyTest, and performance tuning of Spark and Python applications
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization strategies.
- Strong hands-on experience with AWS services including Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS.
- Deep experience in Apache Spark (PySpark/Scala) development, unit testing, and performance optimization.
- Strong Python programming skills using libraries such as pandas, requests, json, and awswrangler.
- Experience on Apache Kafka and Confluent Kafka.
- Experience designing and optimizing data lakes using Apache Iceberg, including compaction and Iceberg optimization techniques.
Skills Required
- Strong understanding of AWS architecture best practices, scalability, security, and cost optimization
- Hands-on experience with AWS Lambda, Glue ETL, Athena, S3, DynamoDB, Step Functions, EventBridge, SNS, and SQS
- Deep experience developing, unit testing, and performance optimizing Apache Spark applications using PySpark or Scala
- Strong Python programming skills, including pandas, requests, json, and awswrangler
- Experience with Apache Kafka and Confluent Kafka
- Experience designing and optimizing Apache Iceberg data lakes, including compaction and optimization techniques
- Experience with AWS cloud architecture, big data processing, real-time streaming, and modern data lake technologies
- Experience building real-time and API-driven data applications
EXL Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about EXL and has not been reviewed or approved by EXL.
-
Healthcare Strength — Medical plan options include HSA, HRA, and PPO choices, with additional health-related programs such as disability coverage and an employee assistance program. Feedback suggests these options provide meaningful flexibility and coverage breadth for many roles.
-
Leave & Time Off Breadth — Up to 20 days of paid time off, plus federal holidays and paid sick and safe time, are highlighted. Feedback suggests this leave structure is a notable benefit alongside core compensation.
-
Retirement Support — A traditional and Roth 401(k) with up to a 3% company match and full vesting after one year is described. Feedback suggests this match and vesting timeline help bolster long-term savings.
EXL Insights
What We Do
Choosing a digital partner is about more than capabilities — it’s about collaboration and character. Unrealistic overhauls and off-the-shelf products ignore what matters most — your unique needs, culture, goals, and your legacy data and technology environments. At EXL, our collaboration is built on ongoing listening and learning to adapt our methodologies. We’re your business evolution partner—tailoring solutions that make the most of data to make better business decisions and drive more intelligence into your increasingly digital operations. Whether your goals are scaling the use of AI and digital, redesign operating models, or driving better and faster decisions, we’re here to partner with you to help you gain—and maintain—competitive advantage with efficient, sustainable models at scale. Our expertise in transformation, data science, and change management helps make your business more efficient and effective, improve customer relationships and enhance revenue growth. Instead of focusing on multi-year, resource- and time-intensive platform designs or migrations, we look deeper at your entire value chain to integrate strategies with impact. We use our specialization in analytics, digital interventions, and operations management—alongside deep industry expertise — to deliver solutions that help you outperform the competition. At EXL, it’s all about outcomes—your outcomes—and delivering success on your terms. Share your goals with us and together, we’ll optimize how you leverage data to drive your business forward. For more information, visit www.exlservice.com.
