Senior Staff Data Engineer, Databricks (R6508)

Posted 26 Minutes Ago
Be an Early Applicant
Hiring Remotely in USA
Remote
Senior level
Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Our mission is to protect service members and civilians with intelligent systems.
The Role
Build and operate a governed Databricks enterprise lakehouse, including Bronze and Silver ingestion pipelines, batch and streaming processing, Delta Lake workflows, data quality controls, access models, CI/CD, and deployment automation. Apply data classification, segregation, security, retention, and compliance requirements in a regulated environment. Establish reusable engineering standards, document data assets, and collaborate with analytics and domain teams to support trusted downstream modeling.
Summary Generated by Built In

Shield AI is a venture-backed defense-tech company with the mission of protecting service members and civilians with intelligent systems. Its products include Hivemind autonomy software, V-BAT and X-BAT aircraft, and Aechelon simulation and synthetic reality technologies. With offices and facilities across the U.S., Europe, the Middle East, and Asia-Pacific, Shield AI’s technology actively supports operations worldwide. For more information, visit www.shield.ai. Follow Shield AI on LinkedIn, X, Instagram, and YouTube. 

 

Role Summary


The Senior Staff Data Engineer will help build and operate the enterprise lakehouse on Databricks, creating the governed data foundation that supports multiple business domains and downstream analytics. This is a hands-on role, responsible for scalable ingestion, reliable data processing, and strong technical controls across the Bronze and Silver layers of the medallion architecture.

 

This role is not limited to moving data from point A to point B. The Data Engineer is expected to understand the meaning, sensitivity, classification, and intended use of the data flowing through the pipelines they build, and to apply that understanding when designing controls, access patterns, data quality checks, and segregation boundaries appropriate for a highly regulated environment.

What You'll Do:

  • Design and build ingestion pipelines, batch and streaming where appropriate, from enterprise source systems into the Databricks lakehouse using Delta Lake.

  • Own Bronze-layer ingestion, including raw landing patterns, metadata capture, load traceability, and recoverable ingestion design.

  • Build Silver-layer pipelines for cleansing, standardization, deduplication, conformance, and quality enforcement without embedding unauthorized or undocumented KPI logic.

  • Define and evolve reusable ingestion and transformation patterns, templates, and engineering standards that other domains can adopt as they onboard to the platform.

  • Implement and maintain Databricks platform constructs needed for secure delivery, including catalogs, schemas, service principals, job orchestration, and environment-aware deployment patterns.

  • Build and maintain CI/CD pipelines for data platform assets, including ingestion code, transformation logic, workflow definitions, tests, and environment promotion across dev, test, and prod.

  • Apply data classification, segregation, and handling requirements within the pipeline design, ensuring sensitive and regulated data is processed in accordance with enterprise controls and access policies.

  • Build data quality controls that do more than detect technical failures, including checks that reflect actual business meaning, record integrity, completeness, and expected domain behavior.

  • Quarantine, flag, and route problematic records or datasets according to defined quality and compliance rules rather than silently dropping or obscuring issues.

  • Maintain documentation for source objects, ingestion logic, applied transformations, data quality rules, and known limitations so downstream teams can trust and use the data correctly.

  • Partner with the Analytics Engineer and domain teams to ensure Silver-layer data is reliable, well-governed, and suitable for trusted Gold-layer modeling.

  • Collaborate with domain engineering teams to align on ownership boundaries, onboarding patterns, data contracts, and support expectations as new domains are enabled onto the platform.

Preferred Qualifications:

  • Databricks certification such as Data Engineer Associate or Professional.

  • Experience standing up or maturing Unity Catalog structures and access models in an enterprise setting.

  • Experience with CI/CD, infrastructure-as-code, and deployment automation for data platforms.

  • Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment.

  • Experience enabling multiple business domains on a shared data platform while maintaining strong governance and ownership boundaries.

Required Qualifications:

  • 12+ years of data engineering experience, including hands-on ownership of production data pipelines.

  • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL.

  • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts.

  • Experience with batch, CDC, and/or streaming ingestion patterns and the operational trade-offs associated with each.

  • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled promotion across environments.

  • Strong SQL skills and solid grounding in data modeling fundamentals, even if dimensional modeling is not the primary responsibility of this role.

  • Demonstrated ability to understand the business and regulatory context of the data being processed, not just the mechanics of pipeline development.

  • Experience applying data classification, segregation, security, retention, or compliance requirements in data engineering workflows within a regulated or security-sensitive environment.

  • Ability to design pipelines with awareness of the actual data domains involved, including sensitivity, ownership, permitted use, and downstream impact.

  • Comfort operating in a fast-moving platform build where patterns are still being established and engineers are expected to shape standards, not just follow them.

#LI-KE1

#LE

 

Full-time regular employee offer package:

Pay within range listed + Bonus + Benefits + Equity

 

Temporary employee offer package:

Pay within range listed above + temporary benefits package (applicable after 60 days of employment)

 

Salary compensation is influenced by a wide array of factors including but not limited to skill set, level of experience, licenses and certifications, and specific work location. All offers are contingent on a cleared background and possible reference check. Military fellows and part-time employees are not eligible for benefits. Please speak to your talent acquisition representative for more information.

 

###

 

Shield AI is proud to be an equal opportunity workplace and is an affirmative action employer. We are committed to equal employment opportunity regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, marital status, disability, gender identity or Veteran status. If you have a disability or special need that requires accommodation, please let us know. 

Skills Required

  • 12+ years of data engineering experience, including hands-on ownership of production data pipelines
  • Strong Databricks experience, including Delta Lake, Databricks Workflows or Jobs, and Spark with PySpark and/or Spark SQL
  • Working knowledge of Unity Catalog, including catalogs, schemas, tables, lineage, and access control concepts
  • Experience with batch, change data capture, and/or streaming ingestion patterns and their operational trade-offs
  • Experience with CI/CD and deployment automation for data pipelines and platform assets, including version control, testing, and controlled environment promotion
  • Strong SQL skills and grounding in data modeling fundamentals
  • Ability to understand the business and regulatory context of processed data
  • Experience applying data classification, segregation, security, retention, or compliance requirements in regulated or security-sensitive data engineering workflows
  • Ability to design pipelines with awareness of data sensitivity, ownership, permitted use, and downstream impact
  • Comfort working in a fast-moving platform build and shaping engineering standards
  • Databricks certification, such as Data Engineer Associate or Professional
  • Experience establishing or maturing Unity Catalog structures and access models in an enterprise setting
  • Experience with CI/CD, infrastructure as code, and deployment automation for data platforms
  • Experience in defense, aerospace, financial services, healthcare, or another highly regulated environment
  • Experience enabling multiple business domains on a shared data platform while maintaining governance and ownership boundaries

What the Team is Saying

Dylan
Mo
Willy
Michael
Vibhav
Kirby
Ryan
Evan

Shield Compensation & Benefits Highlights

  • Healthcare Strength — Medical, dental, and vision coverage are described as strong and affordable, and offerings include a mental-health assistance program. Overall impressions point to good plan value for many employees.
  • Retirement Support — A 401(k) with employer matching is highlighted positively, with accounts describing a company match and favorable experiences. Current match formulas and vesting are noted as worth confirming.
  • Equity Value & Accessibility — Full-time hires receive equity (often via RSUs), positioned as a meaningful component of total rewards. A double-trigger structure is noted as potentially deferring taxes until a liquidity event.

Shield Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: San Diego, CA
Year Founded: 2015

What We Do

At Shield you won't wait years to see your work reach the field. You'll build hardware and software that operates in the real world right now, in the hands of the people who depend on it. Hivemind, our AI pilot, has been flying since 2018. It has flown more than 30 platforms, including an F-16, and it now sits under a U.S. Air Force production contract for Collaborative Combat Aircraft. When you write code or shape a system here, you contribute to technology with a proven flight record and a clear production future. V-BAT flies intelligence, surveillance, and reconnaissance missions with an operational record that stretches from Ukraine to the Indo-Pacific. It delivers eyes where they matter most, in the most demanding conditions on earth. The teams behind it watch their work get tested where the stakes are real. X-BAT takes its first flight this year. It's an AI-piloted fighter that needs no runway, built to operate where traditional aircraft can't. Join now and you help shape a program at its earliest, most formative stage. That's the kind of ground-floor work that defines a career. Do the most impactful work of your life, on problems that matter. Autonomy at this level asks a lot of you. You'll take on problems in perception, planning, and control that few teams anywhere are equipped to solve. You'll work across disciplines, from aerospace and robotics to machine learning and systems engineering, alongside people who hold themselves to an exacting standard and expect the same from you. Our mission is clear: protect service members and civilians with intelligent systems. That purpose runs through every decision, every design review, and every deployment. It's why the work here carries a weight you can feel. Ready to join our mission? Explore our open roles and find where you fit.

Why Work With Us

Founded in 2015 by a former Navy SEAL, Shield builds AI pilots and uncrewed aircraft. Veterans aren't an afterthought here, they're at every level. It's why the work carries weight: AI pilots and uncrewed aircraft flying real missions, from Ukraine to the Indo-Pacific, protecting service members and civilians.

Gallery

Gallery
Gallery
Gallery

Shield Teams

Team
Aircraft
Team
Growth (Sales and Marketing)
Team
Hivemind
About our Teams

Shield Offices

Hybrid Workspace

Employees engage in a combination of remote and on-site work.

Typical time on-site: Flexible
HQSan Diego, CA
United Arab Emirates
Arlington, VA
Company Office Image
Frisco, Texas
Ukraine
London
Melbourne, Victoria
Taiwan, Province of China
Waltham, MA
Learn more

Similar Jobs

Shield  Logo Shield

Engineer I, Software - Autonomy Capabilities - New Grad - 6129

Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Remote
USA

Shield  Logo Shield

Capture Manager

Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Remote
USA

Shield  Logo Shield

Reliability Engineer

Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Remote
USA
180K-270K Annually

Shield  Logo Shield

Staff Devops Engineer

Aerospace • Artificial Intelligence • Machine Learning • Robotics • Software • Defense Technology
Remote
USA

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account