We are seeking a Principal Data Architect to define and drive enterprise data architecture, modernization, and governance strategies across complex multi-cloud (Google Cloud Platform and Microsoft Azure) and hybrid environments. This role will design scalable, secure, and high-performing data platforms while partnering with business and engineering leaders to translate strategic objectives into impactful data solutions.
A key focus of this role will be spearheading scalable, agent-driven data migration pipelines to transition legacy on-premises data assets to the cloud, setting the technical direction for engineering teams, and actively managing the customer's technical backlog and delivery roadmap.
Key Responsibilities
Multi-Cloud Lakehouse Architecture: Lead the end-to-end architecture, design, and implementation of scalable Data Lakehouse solutions across Google Cloud Platform (BigQuery, GCS, BigLake, Dataplex) and Microsoft Azure (Azure Synapse Analytics, Microsoft Fabric, ADLS Gen2, Azure Databricks).
On-Prem to Cloud Migration & Agentic Pipelines: Architect and deploy automated, scalable, agent-driven data pipelines (leveraging GenAI agents and metadata-driven frameworks) to orchestrate, validate, and accelerate complex migrations from on-premises legacy warehouses/databases (Oracle, Teradata, SQL Server) to GCP and Azure.
Client Engagement & Advisory: Collaborate directly with customers to understand business goals, data challenges, and technical requirements; translate them into robust architectural blueprints, technical roadmaps, and actionable delivery plans.
Customer Technical Backlog Management: Own and actively manage the customer technical backlog—translating business epics into technical user stories, prioritizing technical debt, defining sprint milestones, and aligning backlog items with business ROI.
Modern Ingestion & Streaming: Design and implement data pipelines supporting both real-time and batch ingestion using modern orchestration (Cloud Composer, Azure Data Factory, Airflow) and streaming/messaging frameworks (Kafka, GCP Pub/Sub, Azure Event Hubs).
Governance & Lineage: Establish and enforce best practices for unified data cataloging, metadata management, automated lineage, and data quality across multi-cloud and on-prem systems using Dataplex, Microsoft Purview, or open-source governance tools.
Security & Compliance: Define and implement robust data security, identity management (IAM / Azure Entra ID), encryption, row/column-level access control, and regulatory compliance standards across clouds.
Team Technical Leadership & Mentorship: Steer the overarching technical direction of engineering teams; establish architectural standards, mentor senior and mid-level engineers, conduct architecture review, and ensure engineering excellence.
Hands-on Delivery & Prototyping: Balance strategic architecture discussions with hands-on solutioning, architectural spikes, Proof-of-Concepts (POCs), and performance tuning of complex multi-cloud distributed workloads.
Cross-Functional Partnership: Partner with enterprise architects, delivery leads, DevOps engineers, and customer stakeholders to drive solution adoption, operational scalability, and long-term platform maintainability
Technical Experience & Tooling
Cloud Platforms: Deep expertise in both Google Cloud Platform (GCP) and Microsoft Azure enterprise data ecosystems.
Processing Engines: Apache Spark, PySpark, Databricks, Azure Synapse / Fabric Spark, GCP Dataproc.
Modern Migration & AI Automation: Building automated, agentic, or metadata-driven data migration pipelines, automated schema conversion, and data validation agents for on-prem to cloud lift-and-shift/re-platforming.
Messaging & Streaming: Apache Kafka, GCP Pub/Sub, Azure Event Hubs.
Orchestration: Apache Airflow, Google Cloud Composer, Azure Data Factory (ADF).
Analytical Storage & Warehouses: GCP BigQuery, Azure Synapse, Microsoft Fabric, Delta Lake/Parquet.
NoSQL / Cache Storage: MongoDB, Redis, Bigtable, Azure Cosmos DB.
Containerization & Deployment: Kubernetes (GKE and AKS), Cloud Run, Azure Container Apps.
API Frameworks: Python (FastAPI, Flask) for data service endpoints.
Databases & Querying: Advanced SQL validation and extraction from legacy/RDBMS systems (SQL Server, Oracle, PostgreSQL).
CI/CD & Version Control: Git, GitHub, GitLab CI/CD, Azure DevOps, and Infrastructure as Code (Terraform).
Candidates should demonstrate hands-on architecture and design experience across the following domains:
Required Qualifications & Skills
15+ years of progressive experience in Software Engineering, Data Engineering, and Enterprise Data Platform architecture.
5+ years architecting, migrating, and deploying production data platforms across GCP and/or Azure, with proven multi-cloud integration patterns.
Demonstrated track record leading on-premises/cloud to cloud modernization/migration initiatives, including data ingestion, transformation, automated data reconciliation, and cutover strategies.
Experience designing or integrating autonomous, agent-driven, or LLM-augmented data workflows (e.g., automated schema mapping, code conversion, data quality validation agents).
Proven hands-on experience designing and deploying Lakehouse platforms incorporating Medallion Architecture (Bronze/Silver/Gold) and Data Mesh / Data Product principles.
Deep knowledge of multi-cloud data governance, access management, and cataloging (Dataplex, Microsoft Purview, Unity Catalog).
Strong leadership capability in technical backlog management, agile scoping, sprint planning, and setting technical strategy for distributed teams.
Exceptional stakeholder communication skills: capability to articulate architectural trade-offs to C-level executives as smoothly as reviewing code with engineers.
Preferred Qualifications (Optional / Good to Have)
Multi-cloud certifications: Google Cloud Professional Data Engineer / Cloud Architect and/or Microsoft Certified: Azure Data Engineer Associate / Azure Solutions Architect Expert.
Hands-on knowledge of Snowflake and Databricks
Familiarity with AWS data services (Amazon S3, Redshift, Glue, EMR).
Experience with LangChain, LlamaIndex, or agentic framework integrations in data operations.
Education
Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or equivalent practical experience.
Skills Required
- 15+ years of progressive experience in software engineering, data engineering, and enterprise data platform architecture
- 5+ years architecting, migrating, and deploying production data platforms across GCP and/or Azure
- Experience leading on-premises or cloud-to-cloud modernization and migration initiatives
- Experience with data ingestion, transformation, automated reconciliation, and cutover strategies
- Experience designing autonomous, agent-driven, or LLM-augmented data workflows
- Hands-on experience with lakehouse platforms, Medallion Architecture, Data Mesh, and Data Product principles
- Knowledge of multi-cloud data governance, access management, and cataloging
- Technical leadership in backlog management, agile scoping, sprint planning, and distributed-team strategy
- Exceptional stakeholder communication skills across executive and engineering audiences
- Bachelor's or Master's degree in Computer Science, Information Technology, Engineering, or equivalent practical experience
- Google Cloud Professional Data Engineer or Cloud Architect certification and/or Microsoft Azure data or solutions architect certification
- Hands-on knowledge of Snowflake and Databricks
- Familiarity with AWS data services
- Experience with LangChain, LlamaIndex, or agentic framework integrations
What We Do
Egen is a data engineering and cloud modernization firm partnering with leading Chicagoland companies to launch, scale, and modernize industry-changing technologies. We are catalysts for change who create digital breakthroughs at warp speed. Our team of cloud and data engineering experts are trusted by top clients in pursuit of the extraordinary. Our mission is to be an enabler of amazing possibilities for companies looking to use the power of cloud and data. We want to stand shoulder to shoulder with clients, as true technology partners, and make sure they succeed at what they have set out to do. We want to be disruptors, game-changers, and innovators who have played an important part in moving the world forward.
