Job ID: RQ11099 - DataOps/Cloud Data Engineer - Senior
Job Description:
Responsibilities:
Design, develop, test, implement, and troubleshoot: data pipelines using various tools such as Databricks, Azure Data Factory, AWS Gluecomplex data transformation procedures, data models for data warehouses and lakehouses
Experience and Skill Set Requirements:
Programming & scripting: Python, SQL, Linux shell, PowerShell.
Data manipulation/analysis using pandas and pyspark.
Working with XLSX, CSV, JSON files, relational databases, cloud storage, structured and unstructured data
AWS and/or Azure services.
Cloud data warehouse solutions such as AWS Redshift.
Data lakehouse solutions such as Databricks (Delta Lake).
Data processing orchestration/automation
ETL tools (such as Azure Data Factory, AWS Glue) as well as cloud agnostic tools such as Informatica IDMC.
Data modeling & architecture: relational and dimensional modeling.
Data reporting/visualization.
Full SDLC from requirements gathering, design, implementation, testing to deployment and production support
Project management (agile/scrum and waterfall)
Change and Incident management
Communication, presentation and negotiation skills
Consulting, problem-solving and decision-making skills
Experience working in OPS and/or public sector in general
MUST HAVES:
Experience using AWS Services such as Glue, StepFunctions, Lambda, S3
Extract/Transform/Load data using tools such as Informatica IDMC
Skills Required
- Experience using AWS services such as Glue, StepFunctions, Lambda, S3
- Extract/Transform/Load using tools such as Informatica IDMC
- Programming and scripting: Python, SQL, Linux shell, PowerShell
- Data processing with pandas and PySpark
- Experience with Databricks and Delta Lake
- Experience with Azure Data Factory and/or AWS Glue (ETL/orchestration tools)
- Working knowledge of XLSX, CSV, JSON, relational databases and cloud storage
- Experience with cloud data warehouse solutions such as Amazon Redshift
- Data modeling and architecture (relational and dimensional modeling)
- Experience across full SDLC including testing, deployment and production support
- Experience with data processing orchestration/automation and change/incident management
- Experience working in OPS and/or public sector
- Communication, presentation, consulting, problem-solving and decision-making skills
What We Do
ITProFound Inc. is a global technology services and solutions provider specializing in software development, IT consulting, and staff augmentation. They deliver end-to-end business practice solutions, including ERP, cloud migrations, and data analytics, across various industries.
.png)
