Our client is looking for a Senior Data Engineer to design, scale, and maintain high-performance ingestion, Change Data Capture (CDC), and delivery pipelines. In this role, you will architect enterprise-grade data platforms to process large-scale metadata, algorithmic feature extraction, and licensable asset datasets.
This team sits at the foundation of the data processing and feature enrichment pipelines. Driving consistency and performance across an ever-growing data lake, we provide guardrails and sound system design patterns to inform the creation and fine-tuning of the industry's cutting-edge foundational large language models.
You will leverage Google Cloud Platform (GCP) as the primary compute foundation of a multi-cloud data processing ecosystem, alongside hybrid data warehouses and query engines including Snowflake and ClickHouse to provide low-latency data exposure, incremental delta tracking, and seamless integration endpoints for downstream AI/ML consumption.
Primary Tech Stack: GCP, ClickHouse, Snowflake, dbt Core, Apache Airflow, Python, SQL.
Responsibilities
1. Ingestion Architecture & GCP Foundation:
- Architect, build, and optimize scalable job creation, batching, and high-throughput ingestion pipelines hosted on GCP.
- Implement serverless and containerized data pipelines using GCP infrastructure (BigQuery, Dataflow, Pub/Sub, Cloud Storage, GKE).
2. Enterprise ETL, CDC & Incremental Processing:
- Design robust Change Data Capture (CDC) and incremental update workflows to enable fast delta synchronization across platforms.
- Build dbt transformation models and Airflow orchestrations following Data Vault / dimensional modeling patterns.
- Implement delta tracking methods (e.g., hash keys, hash diffs) to track row-level asset modifications and metadata events.
3. Multi-Engine Analytics & Query Optimization:
- Maintain and optimize Snowflake integration layers, handling deduplication logic, canonical content filtering, and policy restrictions.
- Deploy and manage ClickHouse as a high-concurrency analytical engine to deliver sub-second metadata retrieval and indexing.
- Construct dedicated integration views and API query layers allowing internal and external clients to consume real-time delta updates.
4. Data Quality, Governance & Scalability:
- Enforce metadata schema standardization, canonical asset mapping, and automated validation layers across diverse asset types (images, video, 3D models, audio).
- Collaborate closely with cross-functional teams to integrate metadata pipelines with feature extraction tools, search indexes, and licensing platforms.
Requirements
- Google Cloud Platform (GCP): 4+ years of production experience with core GCP data technologies (BigQuery, Dataflow, Cloud Pub/Sub, GKE, Google Cloud Storage).
- Database & Query Engines: Deep technical proficiency with both Snowflake (warehousing, semi-structured queries, integration layers) and ClickHouse (real-time aggregation, low-latency columnar storage, indexing).
- Enterprise ETL / CDC: Proven experience architecting CDC pipelines and incremental delta loading for multi-terabyte dataset scale.
- Data Transformation & Orchestration: Hands-on mastery of dbt Core/Cloud and Apache Airflow in an enterprise CI/CD setting (e.g., GitHub Actions).
- Programming Skills: Advanced skills in Python and SQL; familiarity with Go, Java, or Scala is a strong plus.
Nice-to-Have Qualifications:
- Experience with Data Vault 2.0 architecture, hash-key generation, and raw/staging/integration layer design.
- Background handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations.
- Familiarity with media metadata schemas (images, 3D model stems, video derivatives).
- Experience with other major cloud providers such as AWS, Azure, or Oracle Cloud.
Key Success Metrics (First 90 Days)
- GCP Pipeline Optimization: Standardize ingestion batching and pipeline compute infrastructure on GCP.
- ClickHouse & Snowflake Integration: Establish streamlined integration views exposing low-latency metadata delta changes.
- Automated CDC Workflows: Finalize automated dbt delta transformations with hash tracking across primary warehouse layers.
We offer*:
- Flexible working format - remote, office-based or flexible
- A competitive salary and good compensation package
- Personalized career growth
- Professional development tools (mentorship program, tech talks and trainings, centers of excellence, and more)
- Active tech communities with regular knowledge sharing
- Education reimbursement
- Memorable anniversary presents
- Corporate events and team buildings
- Other location-specific benefits
*not applicable for freelancers
Skills Required
- 4+ years of production experience with GCP data technologies, including BigQuery, Dataflow, Cloud Pub/Sub, GKE, and Cloud Storage
- Deep technical proficiency with Snowflake and ClickHouse
- Experience architecting CDC pipelines and incremental delta loading at multi-terabyte scale
- Hands-on mastery of dbt Core or Cloud and Apache Airflow in an enterprise CI/CD environment
- Advanced Python and SQL skills
- Familiarity with Go, Java, or Scala
- Experience with Data Vault 2.0 architecture, hash-key generation, and raw, staging, and integration layer design
- Experience handling AI/ML metadata, computer vision feature vectors, C2PA provenance headers, or semantic search configurations
- Familiarity with media metadata schemas
- Experience with AWS, Azure, or Oracle Cloud
What We Do
N-iX is a global software solutions and engineering services company that helps world’s leading organizations turn challenges into lasting business value, operational efficiency, and revenue growth using advanced technology. Whether you need to build a custom solution, modernize your digital product or acquire extra tech expertise - we have the experience and capabilities to ensure your success. With over 2,000 professionals in 25 countries across Europe and the Americas, N-iX offers expert solutions in cloud, data analytics, embedded software, IoT, AI, machine learning, and other tech domains. Being in business for over two decades, we have worked with dozens of industry-leading enterprises and Fortune 500 companies creating value across a wide variety of sectors, including finance, manufacturing, supply chain, retail, e-commerce, healthcare, and more. Our unique combination of business domain expertise and technical know-how enables us to effectively collaborate with ISVs, tech companies, and enterprises of all sizes. Thanks to the strong tech ecosystem and partnerships with AWS, GCP, Microsoft, SAP, OpenText, Snowflake, and others, we bring extra speed, scale and efficiency to more than 160 organizations across the globe. N-iX is recognized by numerous industry awards, such as CRN Solution Provider 500, Global Outsourcing 100 by IAOP, ISG Provider Lens™, Modern Application Development services providers by Forrester, etc
