- Data Pipeline Development: Builds understanding of the data needs of the clinet and designs, constructs, installs, tests, and maintains highly scalable data management systems using Microsoft Fabric suite (Azure Data Factory, Azure Synapse Analytics, etc.) and other relevant technologies that can efficiently and effectively meet those needs
- Perform data availability assessment and implement robust processes to ensure the timely identification, collection, and validation of relevant data sets required for the client
- ETL Processes: Develops ETL processes to extract data from various sources, transform the data according to business rules, and load it into a centralised data repository, ensuring data accuracy and availability.
- Data Lake: Implements and manages data storage solutions using Azure OneLake and ensures optimal data storage architecture for ease of access and analysis.
- Data Integration: Integrates data from various business systems into a unified data platform, enabling a consolidated view of information across the organization.
- Data Quality and Governance: Ensures data accuracy and quality by implementing data governance and quality control measures, including data validation and cleansing.
- Performance Optimisation: Monitors, tunes, and reports on the performance of data pipelines and databases to ensure they meet the functional and performance requirements.
- Security and Compliance: Implements security measures to protect data integrity and compliance with data protection regulations and company policies.
Requirements
- Azure Data Factory
- Azure Synapse Analytics
- Azure Data Lake or Apache Delta Lake
- Apache Spark or Databricks,
- Knowledge of implementing Apache Spark using pyspark
- Knowledge of working Azure BLOB Storage
- ETL Development
- SQL and Data Modeling
- Data Quality Management
- Security practices
- Performance tuning
- Testing and troubleshooting
- Clear notes and documentation
Skills Required
- Azure Data Factory
- Azure Synapse Analytics
- Azure Data Lake or Apache Delta Lake
- Apache Spark or Databricks
- Implementing Apache Spark using PySpark
- Azure Blob Storage
- ETL Development
- SQL and Data Modeling
- Data Quality Management
- Security practices for data
- Performance tuning of pipelines and databases
- Testing and troubleshooting data pipelines
- Clear notes and documentation
What We Do
AI driven technology solutions for modern enterprises. We help organizations transform, scale, and innovate through a combination of artificial intelligence and advanced software engineering. By applying intelligent automation and data driven systems, businesses can simplify operations, improve efficiency, and unlock new growth opportunities. Our expertise includes AI strategy and consulting to identify high impact opportunities, AI development to build machine learning models and predictive systems, and intelligent automation to streamline workflows and boost productivity. We also deliver cloud migration, application modernization, embedded systems, and digital product engineering to help businesses move beyond legacy limitations and build scalable, future ready solutions. With a strong focus on results, we enable faster time to market, reduced operational costs, and improved performance across critical business functions. Every solution is tailored to specific goals, ensuring measurable outcomes and long term value. Take the next step toward smarter growth. Contact us now for a consultation and start transforming your business today.
