Senior Data Engineer (Databricks)

Posted Yesterday
Hiring Remotely in United States
Remote
Senior level
Artificial Intelligence • Information Technology • Software
The Role
Design, build, and migrate large-scale batch data pipelines into Databricks. Develop and optimize production-grade Python/Spark/SQL solutions, support parallel legacy and cloud systems during migration, lead technical design discussions, use AI tools to improve workflows, ensure pipeline reliability through testing/monitoring/tuning, and participate in a rotating on-call schedule.
Summary Generated by Built In

At Sparq, we help companies solve the right problems—not just build more technology.


We’re a modern product engineering partner blending strategy, craftsmanship, and speed to help organizations modernize confidently in the age of AI. From data ecosystems to digital products and AI acceleration, we turn complexity into clarity and ideas into impact.


If you’re driven to build what’s next, lead with empathy, and deliver excellence without ego, you’ll feel right at home at Sparq.


C2C is not available

Must be authorized to work in the U.S. without sponsorship

Why you will enjoy Mondays again:

  • Opportunity to work alongside a talented, diverse team in a collaborative and creative environment
  • Ongoing investment in your growth through training, mentorship, and industry certifications
  • Hands-on exposure to cutting-edge technologies across multiple industries
  • Recognition programs and employee awards within an inclusive culture that celebrates great work, including our AI Innovators Program
  • Remote or hybrid work options
  • Competitive salary + bonus opportunities
  • Robust benefits package, matching 401(k) plan, and substantial PTO
  • Tuition reimbursement

A Day in the Life:

  • Build and migrate large-scale batch data pipelines from legacy data platforms into Databricks.
  • Develop and optimize production-grade solutions using Python, Spark, and advanced SQL.
  • Support parallel legacy and cloud data environments throughout the migration and validation process.
  • Serve as a subject matter expert while remaining hands-on with day-to-day development and troubleshooting.
  • Actively use AI tools, including Claude through our Anthropic partnership, to streamline tasks, improve the quality of your work, and share best practices with teammates while continuously seeking new ways to integrate AI into your everyday workflows
  • Lead technical design conversations and guide implementation within an established architecture and project framework.
  • Collaborate with developers, architects, and business stakeholders to communicate technical decisions, risks, and progress.
  • Improve pipeline reliability and performance through testing, monitoring, tuning, and production support.
  • Use GitHub Copilot when helpful to support development productivity.
  • Participate in a rotating on-call schedule approximately every four to six weeks.

What it takes:

  • Recent hands-on experience building production batch solutions in Databricks, ideally within the past 12–18 months.
  • Strong development experience with Python and Apache Spark.
  • Advanced SQL skills, including complex query development and performance tuning.
  • Consultative approach and problem solving skills to successfully align digital solutions with long-term business goals of the client
  • Experience delivering large-scale migrations from legacy data platforms to cloud-based environments, preferably involving Teradata or a comparable platform.
  • Experience supporting legacy and cloud systems operating in parallel during a migration.
  • The ability to lead design and technical-direction discussions while remaining actively involved in coding and implementation.
  • Hands-on experience using AI tools to enhance daily work, or a strong desire to do so, including a willingness to experiment, learn, and champion AI adoption within your role
  • Strong communication skills across technical teams, architects, business partners, and leadership.
  • Availability to participate in an on-call rotation.
  • Experience with Azure data services—such as Data Factory, Synapse Analytics, Event Hubs, Delta Lake, Cosmos DB, and Azure DevOps—is preferred.
  • Exposure to Big Data technologies, NoSQL databases, Git, Jenkins, CI/CD practices, and Agile delivery methodologies is a plus.

Regardless of your specific role, we seek individuals who are excited to explore, adopt, and evangelize AI tools and methodologies. If you have experience in AI or a proven track record of rapidly learning and mentoring others on emerging tech, you’ll fit right in.


Equal Employment Opportunity Policy: Sparq is proud to offer equal employment opportunity without regard to age, color, disability, gender, gender identity, genetic information, marital status, military status, national origin, race, religion, sexual orientation, veteran status, or any other legally protected characteristic.


We are committed to providing equal employment opportunities and believe in an inclusive workplace. If you require reasonable accommodations to participate in the job application or interview process, please let us know by contacting [email protected]



#LI-REMOTE

Skills Required

  • Hands-on experience building production batch solutions in Databricks within the past 12-18 months
  • Strong development experience with Python
  • Strong development experience with Apache Spark
  • Advanced SQL skills, including complex query development and performance tuning
  • Experience delivering large-scale migrations from legacy data platforms to cloud-based environments
  • Experience with Teradata or comparable legacy platforms
  • Experience supporting legacy and cloud systems operating in parallel during a migration
  • Ability to lead technical design and direction while remaining hands-on with coding and implementation
  • Hands-on experience using AI tools to enhance daily work, or a strong desire to learn and champion AI adoption
  • Strong communication skills across technical teams, architects, business partners, and leadership
  • Availability to participate in a rotating on-call schedule
  • Must be authorized to work in the U.S. without sponsorship
  • Experience with Azure data services (Data Factory, Synapse Analytics, Event Hubs, Delta Lake, Cosmos DB, Azure DevOps)
  • Exposure to Big Data technologies, NoSQL databases, Git, Jenkins, CI/CD practices, and Agile delivery methodologies

Sparq Compensation & Benefits Highlights

The following summarizes recurring compensation and benefits themes identified from responses generated by popular LLMs to common candidate questions about Sparq and has not been reviewed or approved by Sparq.

  • Leave & Time Off Breadth PTO with floating holidays and seven U.S. holidays is part of the package, and flexibility around taking time off is highlighted positively. These elements support work–life balance for many roles.
  • Retirement Support A 401(k) program is offered and some postings reference a company match. This provides baseline long‑term savings support even though specific match details are not publicly listed.
  • Inclusive Benefits Coverage Participation in LGBTQ+ inclusion benchmarks and the presence of colleague resource groups signal attention to inclusive policies and benefits. External recognitions further reinforce a focus on equitable workplace practices.

Sparq Insights

Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Atlanta, GA
723 Employees

What We Do

In the age of AI, differentiation isn’t in what you build - it’s in the problems you choose to solve and the outcomes they unlock. That’s why we help leaders cut through the noise, focus on what matters most, and solve it right the first time. By fusing problem-first thinking, deep technical craftsmanship, and fast, flawless delivery, we de-risk transformation and deliver solutions that stick, scale, and prove their worth. With a deep bench of end-to-end technologists, architects, and engineers, no challenge is too complex and no solution is half-built. At Sparq, your mission is our mission. We’re modular by design - meeting you where you are and accelerating you toward where you’re meant to be. We don’t just guide - we climb with you. Embedded alongside your teams, we chart the course, build with precision, and navigate complexity until your outcomes are achieved. Built to solve, not just to build.

Similar Jobs

Onebridge Logo Onebridge

Senior Data Engineer

Information Technology • Analytics
Remote
United States
500 Employees

Solvd, Inc. Logo Solvd, Inc.

Senior Data Engineer

Information Technology • Software • Consulting
Remote
6 Locations
708 Employees

Dynatrace Logo Dynatrace

Solutions Engineer

Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • Automation
Remote or Hybrid
Phoenix, AZ, USA
5600 Employees
130K-160K Annually

Dynatrace Logo Dynatrace

Data Governance Lead

Artificial Intelligence • Big Data • Cloud • Information Technology • Software • Big Data Analytics • Automation
Remote or Hybrid
Boston, MA, USA
5600 Employees
130K-160K Annually

Similar Companies Hiring

Golden Pet Brands Thumbnail
Digital Media • eCommerce • Information Technology • Marketing Tech • Pet • Retail • Social Media
El Segundo, California
178 Employees
Kepler  Thumbnail
Fintech • Software
New York, New York
6 Employees
Onshore Thumbnail
Artificial Intelligence • Fintech • Software • Financial Services
New York, New York
60 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account