ECS is seeking a Senior Databricks Migration Engineer to work onsite for the US Postal Services' Inspector General's Office (OIG) at our Arlington, VA office. This role leads technical data migration efforts from legacy SQL systems, warehouses and servers to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design. This also includes building large data structures and dashboards and reporting on data migration projects.
Responsibilities include:
- Lead the technical migration from legacy SQL Server stored procedures and ADF pipelines to Databricks Lakehouse (Delta Lake), ensuring best practice Lakehouse design.
- Translate traditional relational data warehousing paradigms into scalable, distributed Lakehouse frameworks (Bronze, Silver, Gold).
- Design robust, reusable ETL/ELT frameworks using PySpark, Delta Live Tables (DLT), and Databricks Workflows.
- Architect and refine the Gold Layer (dimensional models, star schemas) specifically to maximize Power BI performance.
- Optimize Databricks SQL Warehouses to support high-concurrency, low-latency Power BI queries (DirectQuery and Import modes).
- Implement advanced optimization techniques, including Z-Ordering, data skipping, liquid clustering, and materialized views.
- Define and enforce governance standards for cluster sizing, auto-scaling policies, and serverless SQL compute to balance performance with cost.
- Implement proactive monitoring dashboards to track Databricks Unit (DBU) consumption and identify cost-saving opportunities.
- Establish best practices for partition strategies and file size management within Delta Lake.
- Design and implement a robust data security model using Unity Catalog for centralized governance.
- Enforce row-level and column-level security policies to ensure compliant data access for Power BI consumers and internal analysts.
- Align the Lakehouse security architecture with existing enterprise Azure Active Directory (Microsoft Entra ID) and RBAC standards.
- Act as the primary technical lead, conducting dedicated pair-programming sessions, workshops, and code reviews to transition the team from SQL-centric to Spark-centric thinking.
- Create comprehensive technical documentation, including architecture diagrams, design patterns, and optimization playbooks.
- Build a foundational knowledge transfer framework to ensure the internal team is fully self-sufficient post-migration.
- Communicate effectively verbally and in written form to both technical and non-technical audience
- Work in an organized fashion, completing tasks timely while paying close attention to details
Please Note: This position is contingent upon additional funding and requires a Public Trust background investigation. This entails an in-depth background check & either US Citizenship or Permanent Resident (Green Card) status.
Preferred Qualifications- Bachelor's degree or higher from an accredited college or university in Computer Science, Engineering, or a related technical field
- 5+ years’ experience in data engineering, data system development or related roles
- 5+ years’ experience with cloud platforms (e.g. Azure, AWS, GCP)
- 1+ year leading complex, cross-functional data projects and technical teams
- Experience with Databricks Lakehouse, Apache Spark, Delta Lake, cloud-native databases, storage solutions, and distributed compute platforms
- Experience with data warehousing, dimensional modeling, enterprise data lakes, incremental data loads, and metadata-driven ingestion and data quality frameworks using PySpark
Note: This position requires working onsite 4 days a week at out USPS customer site in Arlington, VA.
Salary Range: $130,000 - $158,000 annually
General Description of Benefits
Skills Required
- Must be US Citizen or Permanent Resident (Green Card) and pass Public Trust background investigation
- Work onsite 4 days per week at USPS site in Arlington, VA
- Lead migration from SQL Server stored procedures and ADF pipelines to Databricks Lakehouse
- Expertise with Databricks Lakehouse, Delta Lake, and Apache Spark (PySpark)
- Experience designing ETL/ELT frameworks using Delta Live Tables (DLT) and Databricks Workflows
- Experience optimizing Databricks SQL Warehouses and Power BI performance (DirectQuery and Import)
- Implement data governance and security using Unity Catalog and integrate with Azure AD / Microsoft Entra ID and RBAC
- Experience with cloud platforms (Azure, AWS, GCP) and cloud-native storage/compute
- Bachelor's degree in Computer Science, Engineering, or related field
- 5+ years' experience in data engineering or related roles; 1+ year leading complex cross-functional data projects
ECS Compensation & Benefits Highlights
The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about ECS and has not been reviewed or approved by ECS.
-
Healthcare Strength — ECS advertises multiple national-network medical plan options with HSA eligibility alongside dental and vision coverage. Coverage generally begins quickly and is paired with company-paid short- and long-term disability, adding stability to the health package.
-
Retirement Support — A 401(k) with Safe Harbor and immediate vesting on employer contributions is emphasized, with an employer match available. Access to an employee stock purchase plan via the parent company provides an additional savings avenue.
-
Parental & Family Support — Paid parental leave up to 30 days, adoption assistance, and other family-oriented leaves are highlighted. Feedback suggests these offerings add meaningful value beyond base pay for many roles.
ECS Insights
What We Do
ECS, a segment of ASGN (NYSE: ASGN), delivers advanced solutions and services in cloud, cybersecurity, artificial intelligence (AI), machine learning (ML), application and IT modernization, and science and engineering. The company solves critical, complex challenges for customers across the U.S. public sector, defense, intelligence and commercial industries. ECS maintains partnerships with leading cloud, cybersecurity, and AI/ML providers and holds specialized certifications in their technologies. Headquartered in Fairfax, Virginia, ECS has more than 3,400 employees throughout the U.S. and has been recognized as a Top Workplace by The Washington Post for the last five years.
