Data Engineer
Position: Data Engineer
Location: Remote from LATAM
Contract Type: Full-time vendor
Time Zone Alignment: CT
About In All MediaIn All Media is a nearshore managed service provider focused on team augmentation and digital product delivery. We assemble senior, LATAM-based squads from our vetted Coderfull community (500+ engineers) that integrate seamlessly with client teams to deliver software, data, cloud, and AI initiatives with speed and rigor.
Project OverviewJoin a specialized data squad dedicated to a large-scale modernization initiative. This project focuses on the migration of approximately 1,000 datasets to Snowflake, utilizing Apache Airflow for orchestration. While the core architecture and technical approach are pre-defined by the internal team, you will be responsible for the execution, adaptation of pipelines, and ensuring the reliability of the new data ecosystem.
Key Responsibilities
- Data Migration: Execute the migration of high-volume datasets to Snowflake, ensuring architectural alignment and data integrity.
- Pipeline Adaptation: Modify and optimize existing data pipelines to leverage Snowflake’s native capabilities and performance.
- Orchestration & Workflow: Develop, test, and monitor complex DAGs in Apache Airflow to ensure seamless scheduling.
- Data Reliability: Implement comprehensive testing strategies to validate data quality throughout the migration lifecycle.
- Cloud Infrastructure: Manage and interact with AWS services, specifically S3 data lakes, partitioning strategies, and lifecycle policies.
Must-Have Skills
- Advanced SQL: Mastery of complex joins, CTEs, window functions, and performance tuning for large-scale data sets.
- Snowflake Proficiency: Hands-on experience with Snowflake data modeling and warehouse management.
- Python for Data: Strong ability to develop and test pipelines using Python and Airflow.
- Big Data Tools: Practical experience with Apache Spark and Apache Kafka for data processing and streaming.
- AWS Ecosystem: Deep understanding of S3 (data lakes, partitioning, and storage optimization).
- Testing Mindset: Proven experience in designing and executing testing strategies for data pipelines.
Nice-to-Have Skills
- Experience in high-velocity migration projects (legacy systems to cloud).
- Familiarity with data governance and security best practices within Snowflake.
- Certifications in AWS or Snowflake.
Common Requirements
- Solid understanding of data structures, algorithms, and software design principles.
- Commitment to CI/CD best practices and documentation.
- Fluent English for daily technical collaboration and project reporting.
- Ability to work within an Agile environment and follow pre-defined technical roadmaps.
#LI
Top Skills
What We Do
In All Media is a cutting-edge technology and innovation consulting organization, facilitating digital transformation in multiple industries.
We have partnered with top-tier organizations such as NBCUniversal, Sony DADC, Aclara Technologies, and Dun & Bradstreet, among other leading companies worldwide. In All Media harnesses the collaborative synergy of all our international talent pool using Agile methodologies, to deliver excellence efficiently and with the utmost quality.
At In All Media, we believe in diversity and autonomy. Our new model brings together a global managed community, with experts guiding and managing talented engineers; working to bring together the best talent in the world, regardless of where they are. We are not only evolving in terms of technology, but also in terms of In All Media as a re-imagined technology consultancy in step with the Fourth Industrial Revolution.
Our community of 200+ experts and engineers are very motivated by different challenges in their field, and apply continuous learning to stay ahead of the curve in their expertise on new developments and innovative technologies.
Our delivery model is based on Agile, On-site, and Near Shore capabilities to maximize team integration and provide the best value to our clients. We partner holistically with our clients in the co-creation of services and products, and follow streamlined Agile processes throughout.
This makes In All Media a truly global company driven by a community of dedicated, talented individuals self-organizing into multi-disciplinary teams, producing the best possible output for our world-class clients.
We're hiring! Email us your CV: [email protected]
Contact Us: [email protected]
