You will own how data gets into our platform and how it gets served back out — ingestion, the lakehouse, and the query layer underneath everything analytics and product depend on.
The problem is specific. Data arrives from CDC streams, transactional databases, event topics, partner APIs, and files, and today each source carries its own pipeline, its own failure modes, and its own on-call story. Your mandate is to collapse that into one ingestion framework and one open lakehouse — reliable enough to publish SLOs against, fast enough to serve interactive query, and cheap enough to defend line by line.
This is a hands-on role. You will set technical direction, hire, and grow the team — and you will also be in design reviews, in code review, and in the pipeline when a stateful stream will not recover from its checkpoint. Expect roughly half your time in technical work.
You own the full path: how data lands, the table format and its lifecycle, the transformation layer, the engines that serve it, and the SLOs on top of all of it. When a dataset is late or wrong, it is your team's phone that rings — and you are expected to have already built the thing that catches it first.
What you will do:
- Lead, hire, and grow a team of 10+ data engineers — set the technical bar through design and code review, not through status meetings.
- Own the architecture and delivery of a unified ingestion framework: one configuration-driven path for batch, CDC, and streaming sources, with schema evolution, replay and backfill, idempotency, dead-letter handling, and data contracts built into the framework rather than reimplemented per pipeline.
- Own production Spark Structured Streaming pipelines — watermarking, stateful joins and aggregations, checkpoint and restart discipline, exactly-once sinks, lag and backpressure management.
- Own the Apache Iceberg lakehouse: partition and sort strategy, file sizing and compaction, snapshot and orphan-file lifecycle, schema and partition evolution, and multi-engine interoperability.
- Set the dbt modeling standard — layering conventions, tests, contracts, CI enforcement, and lineage that stakeholders trust.
- Own Trino catalog design, workload isolation, and query performance for interactive and federated access.
- Define and meet freshness, completeness, and latency SLOs. Run a 24x7 on-call rotation with a short mean time to restore.
- Own cost: a defensible cost-per-pipeline and cost-per-dataset number, and the levers to move it.
- Partner with product, analytics, and architecture to sequence the roadmap, and bring rigor to decisions — collect the data, seek dissent, and run pilots rather than arguing from opinion
Required Qualifications
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience.
- 8+ years in data engineering, including 3+ years leading engineers as a manager or tech lead — and you are still hands-on in code and design.
- Production experience with Spark Structured Streaming at scale: state store growth, checkpoint recovery, watermark tuning, and late or out-of-order data.
- Deep Apache Spark and PySpark performance work — diagnosing and fixing skew, shuffle pressure, small-file problems, and executor memory failures on real workloads.
- Experience building or substantially owning a reusable ingestion framework serving multiple source types — not a collection of individual pipelines.
- Production experience with an open table format (Apache Iceberg preferred) including schema and partition evolution, compaction strategy, and migration from an existing format.
- Experience with dbt as a team-wide modeling standard, including testing and CI.
- Experience with Trino or Presto operations and query optimization.
- Experience running reliable, high-scale platform systems — 24x7 on-call, availability targets, and fast restoration of service
Preferred Qualifications
- Apache Flink, Kafka or MSK internals, or high-throughput stream-join design.
- CDC tooling in production (Debezium, DMS, GoldenGate, Qlik) and integrating legacy or mainframe sources into modern pipelines.
- Data contracts, catalog, and lineage tooling (DataHub, OpenMetadata, Glue, Unity).
- Iceberg REST catalog implementations and multi-engine interoperability.
- AWS, Kubernetes, and Terraform fluency — you can debug below the framework layer.
- Multi-tenant B2B data platforms with per-tenant cost attribution and isolation.
- Open-source contribution to the projects in this stack.
What Success Looks Like
A stable, well-led SRE organization with clear ownership, career paths, and low regrettable attrition among your managers and their teams.
Consistent, Datadog-driven observability and SLOs in place across the organization, with measurable reduction in Sev1/Sev2 incidents and mean time to detect/resolve.
Modern, standardized infrastructure practices — GitOps delivery via Argo, IaC via Terraform/OpenTofu, and reliable CI/CD via Jenkins — adopted consistently across teams and clouds.
A mature, blameless incident-management culture with strong postmortem follow-through.
Strong cross-functional trust with engineering, product, and security/compliance stakeholders.
Skills Required
- Bachelor's degree in Computer Science, Engineering, or equivalent practical experience
- 8+ years of experience in data engineering
- 3+ years leading engineers as a manager or technical lead while remaining hands-on in code and design
- Production experience with Spark Structured Streaming at scale, including state store growth, checkpoint recovery, watermark tuning, and late or out-of-order data
- Deep Apache Spark and PySpark performance experience, including diagnosing skew, shuffle pressure, small-file problems, and executor memory failures
- Experience building or substantially owning a reusable ingestion framework for multiple source types
- Production experience with an open table format, preferably Apache Iceberg, including schema and partition evolution, compaction, and migration from an existing format
- Experience with dbt as a team-wide modeling standard, including testing and continuous integration
- Experience with Trino or Presto operations and query optimization
- Experience running reliable, high-scale platform systems with 24x7 on-call, availability targets, and rapid service restoration
- Experience with Apache Flink, Kafka or MSK internals, or high-throughput stream-join design
- Production CDC tooling experience with Debezium, DMS, GoldenGate, or Qlik, including legacy or mainframe source integration
- Experience with data contracts, catalog, and lineage tooling such as DataHub, OpenMetadata, Glue, or Unity
- Experience with Iceberg REST catalog implementations and multi-engine interoperability
- Fluency with AWS, Kubernetes, and Terraform, including debugging below the framework layer
- Experience with multi-tenant B2B data platforms, per-tenant cost attribution, and isolation
- Open-source contributions to technologies in the data platform stack
What We Do
PDI Technologies resides at the intersection of productivity and sales growth, delivering powerful solutions that serve as the backbone of the convenience retail and petroleum wholesale ecosystem. By “Connecting Convenience” across the globe, we empower businesses to increase productivity, make more informed decisions, and engage faster with their customers. www.pditechnologies.com
