The Role
Designs, builds, deploys, and sustains enterprise ETL/ELT pipelines using Azure Databricks, SQL, and Python. Responsibilities include batch and streaming ingestion, data transformation, orchestration, quality controls, monitoring, failure recovery, CI/CD, documentation, query optimization, Power BI dataflows, and secure data movement across DoD and Air National Guard systems.
Summary Generated by Built In
Data Engineer / Databricks Engineer-Senior LocationHybrid - Air National Guard Readiness Center (ANGRC), Joint Base Andrews, MD; up to 3 telework days per week. Remote work may be authorized by the Government on an exception basis. Occasional travel may be required.Security Clearance RequirementActive SECRET security clearance Position SummaryNationwide IT Services, NIS, is seeking a Senior Data Engineer / Databricks Engineer to design, build, test, deploy, optimize, and sustain enterprise data pipelines supporting the ANG CDAO data ecosystem for a potential opportunity. The engineer will work primarily with Azure Databricks, SQL, Python, Git-based CI/CD, and integrated DoD/DAF/ANG source systems to provide reliable ingestion, transformation, orchestration, data quality, monitoring, and technical documentation.
Key Responsibilities
Our benefits package includes medical, dental, and vision insurance, life and disability insurance, a 401(k) plan with employer match, paid holidays, PTO (sick/vacation), commuter benefits, employee assistance program (EAP), educational reimbursement, and pet insurance.
Salary range $125,000 to $178,000 annually
The salary range is a general guideline. We consider several factors when determining base salary offers, including the position's scope and responsibilities, the candidate's experience, education, and skills, and current market conditions.
Key Responsibilities
- Design, build, test, deploy, and maintain automated and manual ETL/ELT pipelines supporting approximately 15–20 DAF, ANG, and DoD source systems.
- Implement ingestion patterns for structured, unstructured, API, batch, and streaming data; support near-real-time data currency requirements where applicable.
- Develop SQL and Python transformation logic to cleanse, standardize, deduplicate, validate, and enrich enterprise data in accordance with CDAO data standards and business rules.
- Build and operate Azure Databricks jobs, orchestration sequences, triggers, and workload schedules while optimizing execution performance and cloud-compute utilization.
- Implement automated data-quality checks, reconciliation controls, invalid-record quarantine, monitoring, logging, alerting, and recoverable error-handling patterns.
- Triage critical pipeline failures, perform root-cause analysis, and restore service within required response windows.
- Maintain pipeline code, configuration, infrastructure/deployment scripts, and test assets in the official Git repository and execute automated CI/CD build and security testing before release.
- Create and maintain data lineage, source-to-target mappings, architecture documentation, operational runbooks, and quarterly technical-documentation updates.
- Support data warehouse/lake administration, query optimization, Power BI dataflows, and secure cross-platform data movement as needed.
- Demonstrated hands-on experience developing production-grade ETL/ELT pipelines using Azure Databricks or comparable Spark-based data platforms.
- Advanced SQL and Python skills for data transformation, validation, automation, troubleshooting, and performance optimization.
- Experience with batch and streaming ingestion, APIs, schema management, orchestration, monitoring, data quality, and failure recovery.
- Experience with Git-based version control and CI/CD practices for data-engineering solutions.
- Knowledge of data warehousing/lakehouse patterns, indexing/query optimization, access control, encryption, and secure handling of sensitive/CUI data.
- Must satisfy applicable DoD/DAF security, cybersecurity-awareness, CUI, OPSEC, and access-training requirements.
- Experience extracting and transforming data from DAF/DoD systems such as MILPDS, DCPDS, AFRISS, CHRIS, CMS, CRIS, GFEBS, DEAMS, REMIS, AROWS, M4S, iEMS, or Lockheed Martin Tableau.
- Experience integrating Databricks workloads with Palantir Foundry, Microsoft Azure, Advana/WDP, Power BI, or AskSage.
Our benefits package includes medical, dental, and vision insurance, life and disability insurance, a 401(k) plan with employer match, paid holidays, PTO (sick/vacation), commuter benefits, employee assistance program (EAP), educational reimbursement, and pet insurance.
Salary range $125,000 to $178,000 annually
The salary range is a general guideline. We consider several factors when determining base salary offers, including the position's scope and responsibilities, the candidate's experience, education, and skills, and current market conditions.
Skills Required
- Hands-on experience developing production-grade ETL/ELT pipelines using Azure Databricks or comparable Spark-based data platforms
- Advanced SQL and Python skills for data transformation, validation, automation, troubleshooting, and performance optimization
- Experience with batch and streaming ingestion, APIs, schema management, orchestration, monitoring, data quality, and failure recovery
- Experience with Git-based version control and CI/CD practices for data-engineering solutions
- Knowledge of data warehousing and lakehouse patterns, indexing and query optimization, access control, encryption, and secure handling of sensitive or CUI data
- Ability to satisfy DoD/DAF security, cybersecurity-awareness, CUI, OPSEC, and access-training requirements
- Active Secret security clearance
- Experience extracting and transforming data from DAF/DoD systems such as MILPDS, DCPDS, AFRISS, CHRIS, CMS, CRIS, GFEBS, DEAMS, REMIS, AROWS, M4S, iEMS, or Lockheed Martin Tableau
- Experience integrating Databricks workloads with Palantir Foundry, Microsoft Azure, Advana/WDP, Power BI, or AskSage
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Nationwide IT Services, Inc. (NIS) is designated 8(a) by the SBA, and a CVE verified Service Disabled Veteran Owned Small Business (SDVOSB) Information Technology (IT) and Management consulting company. NIS has been providing Information Technology Solutions, program management support services and subject matter expertise within the federal government since 2006. NIS works with each client to deploy a mission-specific solution that: Complies with the Regulatory Environment Captures the Client’s Mission, Vision & Values Aligns Objectives with Deliverables Applies Industry Best Practices Creates Measurable, Sustainable Change.
