The Role
Design, build, and optimize cloud data pipelines using PySpark, Snowflake, and AWS. Migrate legacy ETL to cloud, implement deduplication and data quality, tune performance, manage Snowflake schemas, and collaborate with architects and stakeholders while documenting designs and runbooks.
Summary Generated by Built In
PLEASE NOTE:
- IT IS 100 % ON SITE POSITION IN Mc lean VA
- PySpark & Python for data pipeline development, Snowflake & AWS
DESCRIPITION:
We are seeking a hands-on, delivery-focused Senior Data Engineer to help build and scale our cloud data platform. In this role, you will design and develop modern data pipelines using PySpark, Snowflake, and AWS to optimize cloud data workloads. The ideal candidate combines strong engineering fundamentals with cloud-native data expertise and is capable of translating complex business needs into robust, performant, and well-documented data solutions. Experience within Fannie Mae, Freddie Mac, or equivalent GSE/mortgage enterprise environments is highly valued.
RESPONSIBILITES:
- Scalable Architecture: Design and build scalable batch and streaming data pipelines using PySpark for large-scale data processing.
- Modernization: Migrate legacy, on-premises ETL workloads (e.g., IBM DataStage, Informatica) to high-performing PySpark and Snowflake cloud pipelines.
- Data Transformation: Write production-grade PySpark code to read from Amazon S3 (Parquet/Delta files), execute complex transformations, and process massive datasets efficiently.
- Deduplication: Design and implement robust deduplication strategies for high-volume datasets using PySpark.
Platform Engineering: Build and manage Snowflake warehouses, schemas, and data models optimized for enterprise analytics and business intelligence reporting. - Iceberg Tables: Design and implement Apache Iceberg tables in Snowflake to support open lakehouse architectures and data interoperability.
Incremental Processing: Build and maintain Snowflake Dynamic Tables and Materialized Views to enable near real-time analytics and query acceleration. - PySpark Tuning: Optimize distributed Spark jobs by leveraging partitioning, caching, broadcast joins, and shuffle optimization.
Snowflake Optimization: Tune Snowflake workloads using clustering keys, micro-partition pruning, query profiling, precise warehouse sizing, and strategic result caching. - Cost Management: Continuously monitor and optimize Spark jobs, Snowflake queries, and AWS infrastructure to balance speed and cloud expenditure.
- Data Quality: Implement data validation, lineage tracking, and monitoring solutions across all pipeline stages to ensure high data integrity.
Cross-Functional Collaboration: Partner closely with data architects, business analysts, Technical Program Managers (TPMs), and corporate stakeholders to deliver dependable data products. - Technical Documentation: Author comprehensive technical designs, data schemas, and operational runbooks to ensure every pipeline is maintainable and audit-ready.
Requirements
REQUIRED QUALIFICATION:
- Experience: 6+ years of hands-on data engineering experience in large-scale enterprise environments.
- PySpark Expertise: Deep proficiency in building distributed data processing pipelines, handling S3 Parquet/Delta files, and implementing complex transformations and deduplication logic.
- Snowflake Proficiency: Strong hands-on experience with SnowSQL, Snowpipe, Streams, Tasks, and Role-Based Access Control (RBAC). Proven track record establishing Iceberg tables, Dynamic Tables, and Materialized Views.
- AWS Cloud Ecosystem: Robust working knowledge of AWS services, including S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
- Advanced SQL & Python: Mastery of advanced SQL techniques (window functions, CTEs, complex joins) alongside strong Python programming skills for automation, scripting, and orchestration utilities.
- Orchestration & Architecture: Solid understanding of data warehousing, ELT/ETL patterns, data lakes, and lakehouse architectures using tools like Airflow or AWS Step Functions.
- Communication: Strong verbal and written communication skills with the ability to articulate technical decisions clearly to both technical peers and business leaders.
PREFERRED QUALIFICATION:
- Industry Experience: Prior experience working within heavily regulated environments such as financial services, mortgage banking, or GSE programs (Fannie Mae / Freddie Mac).
- ETL Migration: Hands-on experience with legacy ETL frameworks (e.g., IBM DataStage) to support modernization initiatives.
- DevOps & CI/CD: Familiarity with continuous integration and continuous deployment pipelines for data infrastructure (Git, Jenkins, GitHub Actions, Terraform).
- Data Quality Frameworks: Exposure to automated data quality and validation frameworks (e.g., Great Expectations, dbt testing suites).
- Streaming Analytics: Knowledge of real-time streaming platforms like Apache Kafka or AWS Kinesis.
- Professional Certifications: AWS Certified Data Analytics, AWS Certified Solutions Architect, or SnowPro Core/Advanced certifications.
Skills Required
- 6+ years of hands-on data engineering experience in large-scale enterprise environments.
- Deep proficiency in PySpark for distributed data processing, transformations, and deduplication.
- Strong hands-on experience with Snowflake (SnowSQL, Snowpipe, Streams, Tasks, RBAC, Dynamic Tables, Materialized Views) and Iceberg tables.
- Working knowledge of AWS services: S3, Glue, EMR, Lambda, IAM, Step Functions, CloudWatch, and Redshift.
- Advanced SQL (window functions, CTEs, complex joins) and strong Python programming skills.
- Solid understanding of data warehousing, ELT/ETL patterns, lakehouse architectures, and orchestration tools (Airflow or AWS Step Functions).
- Strong verbal and written communication skills to articulate technical decisions to technical and business stakeholders.
- Experience migrating legacy ETL workloads (e.g., IBM DataStage, Informatica) to cloud pipelines.
- Familiarity with DevOps/CI-CD for data infrastructure (Git, Jenkins, GitHub Actions, Terraform).
- Exposure to data quality frameworks (Great Expectations, dbt testing suites) and streaming platforms (Apache Kafka, AWS Kinesis).
- Prior experience in regulated financial services, mortgage banking, or GSE environments (Fannie Mae/Freddie Mac).
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Omm IT Solutions is a leading provider of IT consulting, analysis, and advisory services, specializing in delivering technical solutions for federal and state government agencies. Their offerings include application development, cybersecurity services, cloud advisory, network modernization, and data analytics.
.png)
