The Role
Build and operate reliable end-to-end data pipelines that integrate structured, semi-structured, document, and time-series sources into a lakehouse or data fabric. Responsibilities include ingestion, transformation, cleansing, batch and streaming processing, monitoring, testing, data quality, and preparing curated datasets for visualization and MLOps. The role collaborates with architects and cross-functional teams to align pipelines with platform standards and support initial pilot use cases.
Summary Generated by Built In
Phase: Initial phase (foundational)
Required experience: 7+ years in data engineering, building and operating production data pipelines.
Role summary
Builds the end-to-end data pipelines that bring every source into the platform reliably. Responsible for source integrations, ingestion, transformation, and data processing that feed the lakehouse and, later, the graph and ML layers. This is the core delivery role for the initial pilot use cases.
Key responsibilities
- Build and operate end-to-end data pipelines from source systems into the lakehouse / data fabric.
- Integrate structured, semi-structured, document, and time-series sources into a common platform.
- Implement data processing, cleansing, and transformation for the priority pilot use cases.
- Ensure pipeline reliability, monitoring, and data quality across all feeds.
- Work with the architect to align pipelines to platform standards and modeling conventions.
- Prepare curated datasets for the visualization and MLOps workstreams.
Must-have skills and experience
- Strong data engineering background building production pipelines end to end.
- Hands-on experience with Azure data services and readiness to work with Snowflake.
- Experience integrating diverse sources: Oracle ERP, MongoDB, PostgreSQL, Cassandra, Redis, InfluxDB, and time-series data.
- Solid data processing skills (batch and streaming) and strong SQL.
- Experience with object and file storage such as MinIO / NFS.
- Data quality, testing, and pipeline observability practices.
Nice to have
- Snowflake production experience.
- Exposure to OT / IoT data feeds.
- Familiarity with orchestration frameworks and infrastructure-as-code.
Relevant stack
Azure Data Factory / Synapse, Snowflake, Oracle ERP, MongoDB, PostgreSQL, Cassandra, Redis, InfluxDB, time-series sources, MinIO / NFS, APIs.
General attributes
- Proactive and self-driven, able to take ownership and move work forward without waiting to be told.
- AI-enabled in day-to-day work, comfortable using AI tools and copilots to accelerate delivery and quality.
- Strong self-learner who stays current with evolving tools, platforms, and practices.
- Good team player who collaborates well across engineering, operations, and stakeholder groups.
Skills Required
- 7+ years of experience in data engineering
- Experience building and operating production data pipelines end to end
- Hands-on experience with Azure data services
- Readiness to work with Snowflake
- Experience integrating Oracle ERP, MongoDB, PostgreSQL, Cassandra, Redis, InfluxDB, and time-series data
- Strong data processing skills for batch and streaming workloads
- Strong SQL skills
- Experience with object and file storage such as MinIO or NFS
- Experience with data quality, testing, and pipeline observability practices
- Snowflake production experience
- Exposure to OT or IoT data feeds
- Familiarity with orchestration frameworks and infrastructure-as-code
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
Coditude stands out as a rapidly growing force in the digital realm, offering straightforward, impactful tech capabilities. Our team, both seasoned and savvy, is the perfect ally to thrive in the digital age. We excel in Product Development, SaaS Solutions, Enterprise Mobile Applications, AI, Cloud Solutions, Browser Extension Development, and Digital Commerce Solutions, not to mention our prowess in Infrastructure Modernization and Management
