Senior Data Engineer

Job Title:

Contract Type:

Time Type:

Job Description:

Main Responsibilities

Data Engineering Leadership

• Contribute to the definition of the target data architecture (lake/lakehouse, streaming, event-driven) and technology choices for high-volume time-series, market, and transactional data.
• Build and operate end-to-end pipelines (ingest → quality → transform → serve) with strong SLAs/SLOs, lineage, and observability.
• Establish coding, testing, CI/CD, and infrastructure-as-code standards; drive adoption of data governance, cataloging, and access controls.
• Own capacity, performance, reliability, and cost efficiency of data platforms across environments (dev/test/prod).
• Ensure data quality, lineage, and observability across all layers of the data stack.

Strategic Collaboration & Business Alignment

• Partner with trading desks, quantitative teams, and risk functions to translate business needs into data solutions that enhance decision-making and operational efficiency.
• Act as a senior liaison between engineering and business stakeholders, ensuring alignment on data priorities and delivery timelines.
• Prioritize a value-based backlog (e.g., faster close/settlement, improved forecast accuracy, reduced balancing penalties) and measure business impact.
• Align data models and domain ownership with business processes (bids/offers, nominations, positions, exposures,outages).
• Coordinate with Cybersecurity, Compliance, and Legal on sector-specific controls (e.g., REMIT/NERC-CIP considerations, data retention, segregation).

Innovation & Product Development

• Incubate and industrialize data products: curated marts, feature stores, real-time decision APIs, and event streams for forecasting and optimization.
• Introduce modern patterns (CDC, schema evolution, Delta/Iceberg, stream–batch unification) to improve freshness and resilience.
• Evaluate and integrate external data (weather, fundamentals, congestion, capacity postings), internal and external vendor systems (ETRM) safely and at scale.
• Collaborate with quantitative analysts to productionize ML pipelines (forecasting load/renewables, anomaly detection, etc.. ) with monitoring and rollback.

Mentorship & Technical Oversight

• Lead incident reviews and architectural forums; provide pragmatic guidance on trade-offs (latency vs. cost, simplicity vs. flexibility).
• Develop growth paths and learning plans focused on energy domain fluency and modern data engineering practices.

Operational Excellence

• Implement robust monitoring/alerting, runbooks, and on-call rotations; drive MTTR down and availability up for critical data services.
• Enforce data quality contracts (SLAs/SLOs), lineage, and reconciliation for market submissions, settlements, and reporting.
• Optimize cloud spend and storage/compute footprints; plan capacity for market events and seasonal peaks.
• Ensure security and compliance by design: least-privilege access, secrets management, encryption, auditability, and disaster recovery testing.

Profile

• Master’s or Bachelor’s degree in Computer Science, Data Engineering, Applied Mathematics, or a related technical field.
• 5+ years of experience in data engineering, with at least 3 years in a senior role.
• Proven experience in the energy trading sector, ideally with exposure to Natural Gas and Power markets, balancing mechanisms, and regulatory frameworks (e.g., REMIT, EMIR).
• Azure: ADLS Gen2, Event Hubs, Synapse Analytics, Azure Databricks (Spark), Azure Functions, Azure Data
• Factory/Databricks Workflows, Key Vault, Azure Monitoring/Log Analytics; IaC with Terraform/Bicep; CI/CD with Azure DevOps or GitHub Actions.
• Snowflake (on Azure or multi-cloud): Warehousing design, Streams & Tasks, Snowpipe/Snowpipe Streaming, Time
• Travel & Fail-safe, RBAC & row/column security, external tables over ADLS, performance tuning & cost governance.
• Programming & Engineering Practices: Strong OOP in Python and Java/Scala; SDLC leadership, DevOps mindset, TDD/BDD, code reviews, automated testing (unit/integration/contract), packaging and dependency management, API design (REST/gRPC).
• Web & Data Acquisition: Robust web scraping and ingestion with Scrapy, Requests, Playwright/Selenium; scheduling, retries/exponential backoff, change-data capture; ethical/legal collection practices (robots.txt, terms).
• Orchestration & Quality: Airflow/ADF/Databricks Jobs, data contracts, Great Expectations (or similar), lineage/catalog (e.g., Purview), metrics/observability (Prometheus/Grafana/Application Insights).
• Dataframe oriented programming: pandas, spark/snowpark dataframes, SQL data transformation
• Additional Skills
• Designing low-latency pipelines for sub-second to minute-level telemetry, weather and market data; tuning Spark
• Structured Streaming/Flink/Kafka Streams.
• Quality & Reconciliation for telemetry and market submissions (gap fill, resampling, deduplication, anomaly detection, schema evolution).
• Serving Patterns: time-series stores and query layers (e.g., Delta Lake over ADLS, Iceberg, materialized views in Snowflake), APIs and event streams for downstream consumption.
• English (fluent), any additional language is an asset

If you think the open position you see is right for you, we encourage you to apply!

Our people make all the difference in our success.