Senior Software Engineer - Data Pipelines

Your Role:

• You'll own the whole lifecycle of data pipelines running in 15+ regions, from design through production, and set the technical direction for how we build them.
• You'll write and optimize big, distributed processing jobs to handle massive datasets.
• You'll build and maintain pipelines that feed our product's access layer—think threat intelligence, assets, and detections—and evolve the data models underneath them.
• Quality is huge: you'll design frameworks to check data health, run conditional checks, and keep everything observable.
• You'll lead our observability across all pipelines.
• You'll raise the engineering bar through design reviews, code review, and mentorship.

Team & Culture:

• We believe in owning your work end-to-end without annoying bottlenecks.
• We're AI-native, using custom plugins and protocols to make development smoother.
• We take engineering rigor seriously (think dead-code detection, constant security scans, and always-on production monitoring).
• Yes, we have on-call, but we've built tools to automate the pain out of it—and we expect you to keep automating.

What You'll Need:

• 7+ years building and operating production data pipelines at scale, including systems you designed yourself and then ran.
• Deep Python 3.10+ chops (Pydantic, async, decorators, packaging, profiling).
• Apache Airflow in production DAG authoring at scale, dynamic task mapping, deferrable operators, SLAs, MWAA or self-hosted operations and upgrades.
• PySpark mastery job authoring plus real optimization (partitioning, shuffle and skew mitigation, join strategy, reading Catalyst plans), EMR or Databricks.
• Snowflake depth (Snowpark, schema and clustering design, query tuning, warehouse sizing, cost attribution).
• AWS knowledge (S3, SQS/SNS, EMR, IAM, infrastructure-as-code).
• Data quality tooling experience designing frameworks (Great Expectations or equivalent), not only configuring them.
• Observability: distributed tracing, structured logging, metrics.
• CI/CD with artifact packaging and staged deployments across many environments and regions.
• Strong testing discipline: pytest, moto, integration tests against real or mocked cloud services.

#LI-SK1

#LI-Hybrid