Sr. Data Engineer AI

Scalable Systems is a USA-based Big Data, Analytics and Digital Transformation Company focused on vertical, innovative solutions. By providing next-generation technology solutions and services, we help organizations identify risks & opportunities, achieve operational excellence, and gain an innovative edge.
 

Responsibilities:

• Design and implement data strategies specifically tailored for AI/ML solutions developed by the AI Scrum team, ensuring optimal data flow between the AI Foundation and the applications.
• Collaborate with AI/ML Engineers to prepare datasets, implement data transformation, and streamline data input/output for machine learning models.
• Monitor and troubleshoot data-related issues within AI applications, offering timely solutions to maintain data integrity and application performance.
• Act as the primary interface between the AI Scrum team and the AI Foundation/Data Management team regarding data requirements, access, and quality.

Expertise:

• Proficiency in data manipulation and processing techniques, with experience using Azure data services (e.g., Azure Data Lake, Azure SQL Database, Azure Cosmos DB).
• Skilled in implementing data ingestion and transformation pipelines tailored for AI/ML workloads, with a focus on real-time data processing where necessary.
• Understanding of data schema design and data modeling specifically for AI/ML applications, ensuring datasets are optimized for machine learning algorithms.
• Knowledge of data privacy and security best practices within the Azure environment, ensuring compliance with data governance and regulations.
• Familiarity with the specific data requirements of generative AI applications, such as text, image, or sequence data, and the ability to efficiently process and prepare this data for use with Large Language Models (LLMs).

Experience:

• Proven experience working with large-scale data in a cloud environment, particularly Azure, with a strong understanding of the end-to-end data lifecycle in AI/ML projects.
• Experience in collaborating with cross-functional teams to deliver data solutions that support complex AI/ML applications.
• History of successful implementation of data processes that have directly contributed to the performance and scalability of AI/ML solutions.
• Background in working with diverse data types and formats, with experience in providing clean, well-structured data to AI/ML models for training and inference.