Position: Data Engineer (Platform Modernization & Migration)
Location: Fully Remote (EMEA & APAC Preferred)
Contract Type: Full-time, Permanent
Time Zone Alignment: Requires overlap until at least 12:00 PM Pacific Time (PT)
🧭 About Inallmedia.com Inallmedia.com is a global technology and design firm focused on building impactful digital solutions through remote, distributed teams. We partner with international clients across industries, providing long-term technical expertise, product innovation, and team augmentation.
🚀 Project Overview You will join a core Data Engineering organization, working in close partnership with the Data Platform team to drive a major data platform modernization initiative. The primary focus of this project is executing pre-designed migration blueprints for approximately 1,000 datasets to Snowflake with Airflow 3 orchestration. You will play a crucial role in accelerating time-sensitive infrastructure goals by evolving legacy Lakehouse formats to Apache Iceberg, modernizing streaming infrastructure across cloud regions, and adapting existing pipelines to uphold strict data quality and performance standards.
🔍 Key Responsibilities
- Pipeline Migration & Upgrades: Execute the end-to-end migration of approximately 1,000 datasets to Snowflake, including upgrading orchestration workflows from Apache Airflow 2 to Airflow 3.
- Storage & Data Lake Modernization: Migrate legacy Hive data structures and datasets to Apache Iceberg format using the Snowpark Connect framework/connectors.
- Streaming Infrastructure & Region Migration: Reconfigure, update, and migrate Confluent/Apache Kafka streaming connections across cloud regions (e.g., migrating from US East 1 to US East 2).
- Pipeline Adaptation & Performance Tuning: Adapt existing data pipelines to target Snowflake environments while optimizing SQL queries (utilizing CTEs, window functions, and complex joins) to ensure high performance and data reliability.
- Workflow Scheduling & Monitoring: Manage, test, and debug DAGs (Directed Acyclic Graphs) and workflow scheduling within Apache Airflow.
- Testing & Quality Assurance: Implement testing strategies and conduct hands-on code debugging to guarantee data quality and consistency throughout the platform modernization.
💡 Must-Have Skills
- Industry Experience: Minimum 2+ years of hands-on data engineering experience working with large-scale datasets, Apache Airflow, and Apache Spark.
- Core Languages & SQL: Strong software engineering fundamentals with hands-on Python development experience and advanced SQL skills (complex joins, CTEs, window functions, and query tuning).
- Data Warehousing & Modeling: Proven experience with Snowflake data modeling and schema architecture.
- Data Processing & Orchestration: Direct experience building and testing pipelines using Apache Airflow, Apache Kafka, and Apache Spark / PySpark.
- Data Formats & Cloud Storage: Expertise working with Apache Iceberg and Apache Parquet data formats, alongside AWS S3 (data lakes, bucket partitioning, and lifecycle policies).
- Testing & Debugging: Ability to define testing strategies and independently debug complex distributed data pipelines.
- Communication & Autonomy: Strong communication skills with a proven track record of working independently in a fast-paced environment.
- AWS WorkSpace Requirement: This role requires the use of a client-provisioned AWS WorkSpace. Candidates must demonstrate good connectivity to an AWS WorkSpace region by completing the connectivity test at https://clients.amazonworkspaces.com/Health. Please note: You will be required to submit a screenshot of these results as supporting evidence with your application.
✨ Nice-to-Have Skills
- Snowflake Migration: Experience with enterprise Snowflake migrations (warehouse modernization, cost and performance tuning).
- Airflow Migration: Direct experience with Apache Airflow platform migrations (version upgrades, executor changes, or platform moves).
- AI Tooling: Hands-on experience using AI-assisted coding/development tools to accelerate workflow velocity.
🌐 Time Zone & Collaboration The role is fully remote, with a strong preference for candidates located in the EMEA (Europe, Middle East, Africa) or APAC (Asia-Pacific) regions. Time zone coverage is required until at least 12:00 PM PT (Pacific Time) to ensure seamless overlap with the core engineering team.
💬 Language All interviews, documentation, and daily communication will be in English.
#LI
Skills Required
- At least 2 years of hands-on data engineering experience with large-scale datasets, Apache Airflow, and Apache Spark
- Hands-on Python development experience
- Advanced SQL skills, including complex joins, CTEs, window functions, and query tuning
- Experience with Snowflake data modeling and schema architecture
- Experience building and testing pipelines with Apache Airflow, Apache Kafka, and Apache Spark or PySpark
- Expertise with Apache Iceberg and Apache Parquet data formats
- Experience with AWS S3 data lakes, bucket partitioning, and lifecycle policies
- Ability to define testing strategies and independently debug complex distributed data pipelines
- Strong communication skills and ability to work independently in a fast-paced environment
- Ability to use a client-provisioned AWS WorkSpace and submit a connectivity test screenshot
- Experience with enterprise Snowflake migrations, warehouse modernization, and cost and performance tuning
- Experience with Apache Airflow platform migrations, version upgrades, executor changes, or platform moves
- Hands-on experience using AI-assisted coding or development tools
- English proficiency for interviews, documentation, and daily communication
- Availability for time-zone overlap until at least 12:00 PM Pacific Time
What We Do
In All Media is a cutting-edge technology and innovation consulting organization, facilitating digital transformation in multiple industries. We have partnered with top-tier organizations such as NBCUniversal, Sony DADC, Aclara Technologies, and Dun & Bradstreet, among other leading companies worldwide. In All Media harnesses the collaborative synergy of all our international talent pool using Agile methodologies, to deliver excellence efficiently and with the utmost quality. At In All Media, we believe in diversity and autonomy. Our new model brings together a global managed community, with experts guiding and managing talented engineers; working to bring together the best talent in the world, regardless of where they are. We are not only evolving in terms of technology, but also in terms of In All Media as a re-imagined technology consultancy in step with the Fourth Industrial Revolution. Our community of 200+ experts and engineers are very motivated by different challenges in their field, and apply continuous learning to stay ahead of the curve in their expertise on new developments and innovative technologies. Our delivery model is based on Agile, On-site, and Near Shore capabilities to maximize team integration and provide the best value to our clients. We partner holistically with our clients in the co-creation of services and products, and follow streamlined Agile processes throughout. This makes In All Media a truly global company driven by a community of dedicated, talented individuals self-organizing into multi-disciplinary teams, producing the best possible output for our world-class clients. We're hiring! Email us your CV: [email protected] Contact Us: [email protected]
