Senior Data Engineer

ABOUT THE COMPANY:

OAG is a leading data platform for the global travel industry offering an industry - first single source for supply, demand, and pricing data.  

We empower the global travel industry with high-quality, relevant datasets covering the whole journey from planning to customer experience.  

Headquartered in the UK, with operations in the USA, Denmark, France, Germany, Singapore, Japan, China, and Lithuania. For more information visit www.oag.com  

ABOUT THE PROJECT/ROLE:

You will join a diverse team of 6 multi-skilled software and data engineers, tasked with the modernisation of Flight Status data ingestion, processing, and aggregation mechanisms. Our initiative is targeted towards the implementation of a platform based on industry standards and best practices, thus ensuring crucial enablement of OAG’s core business. Our focus is on data pipeline.

YOU WILL: 

• Design, build, and optimize scalable ETL and Structured Streaming pipelines in Azure Databricks for real-time and batch ingestion of Flight Status data.
• Design and implement data ingestion and processing pipelines that consolidate heterogeneous data sources, including APIs, event streams, and file-based feeds, into the OAG lakehouse (Azure Databricks + Delta Lake), ensuring data consistency, reliability, and scalability.
• Implement and monitor data quality using automated validation, alerting, and observability practices.
• Develop and maintain orchestration workflows in Apache Airflow, coordinating ingestion and transformation processes across multiple data flows.
• Build reusable frameworks for schema evolution, error handling, deduplication, and auditing.
• Collaborate with data platform, analytics, and product teams to define SLAs, data contracts, and performance targets.
• Optimize Spark and Delta Lake performance for scalability, latency, and cost efficiency.
• Implement CI/CD pipelines and automation for data workflows using Azure DevOps or equivalent tools.
• Mentor engineers, review code, contribute to platform design discussions and planning, and help grow data engineering competencies in the team and across OAG.

ESSENTIAL:

• Proven track record in data engineering, with a strong focus on ETL development and streaming data architectures.
• Experience with Azure Databricks, Apache Spark (Structured Streaming), and Delta Lake.
• Proficiency in Python (PySpark) and SQL, with experience transforming large-scale, complex datasets.
• Hands-on experience in data orchestration and workflow automation (e.g., Apache Airflow or similar).
• Experience working in a cloud data environment (preferably Azure) across storage, compute, and pipeline services.
• Familiarity with streaming or messaging technologies (e.g., Kafka, Event Hubs).
• Strong understanding of data quality, validation, and observability practices.
• Ability to deliver production-grade solutions with a results-oriented and ownership-driven mindset.
• Experience implementing CI/CD and version-control practices using Azure DevOps, GitHub Actions, or similar tools.
• Excellent analytical, communication, and collaboration skills.
• Strong understanding of modern data engineering patterns and ability to design scalable, modular, and reliable data systems.

DESIRABLE:

• Knowledge of data governance and lineage tools (e.g., Unity Catalog).
• Experience with data-quality frameworks (e.g., Great Expectations, Soda).
• Understanding of real-time ingestion design patterns and schema evolution techniques.
• Exposure to containerized environments, infrastructure-as-code practices (Terraform).

WE WILL OFFER YOU:

• A dynamic work environment at OAG, fostering innovation in a progressive, non-hierarchical culture, where passionate tech enthusiasts continuously develop new products and solutions with a focus on improvement for the aviation industry.
  
• Company-provided free lunch every day.
• A modern office in a convenient location, with flexible working arrangements that allow remote work or office attendance.
  
• Work remotely from anywhere in the world for 1 month each year.
• An attractive compensation and benefits package, including private health insurance, a company bonus scheme, and voluntary participation in a company-supported retirement scheme.
  
• A generous annual leave policy, growing with each year of service, and a day off during your birthday month.
• Participation in team-building activities, team workshops, and group learning sessions.

We will ensure that the exact salary offered for you will be based on your qualifications, competencies, professional experience and requirements for the corresponding job function (salary starts from 4100 EUR gross per month).

OAG is an Equal Opportunity Employer. We ensure all applicants are considered for employment without discrimination.