As a Big Data engineer you will be responsible for data-related implementation tasks that include provisioning of data storage services, ingesting, streaming and batch data, transforming data, implementing security requirements, implementing data retention policies, identifying performance bottlenecks, and accessing external data. You will develop and implement data Lake solutions for company-wide applications and manage large sets of structured, semi structured and unstructured data. You will work in a modern cloud-based data lakehousing environment alongside a team of diverse, intense, and interesting co-workers.
Job Responsibilities
• Performing analysis of large data stores and derive insights using Big Data querying
• Design and develop code, scripts, and data pipelines that leverage structured, semi-structured and unstructured data
• Should be able to manage Data ingestion pipelines and stream processing
• Perform computations using Azure functions
• Responsible for the documentation, design, and development of Hadoop applications
• Perform day-to-day tasks to accuracy with and minimal directions while meeting deadlines
Qualification & Skillset
• Bachelor/master’s degree in computer science
• At least 4-6 years of relevant working experience with Cloud Computation
• Hands-on experience on Azure/AWS
• Experience working with Apache Spark, Apache Hive, Apache HBase
• Hands-on experience on:
• Azure/AWS Architectures
• SQL, Scala, Node.js, JS, Java
• Python with Pandas, NumPy, TensorFlow
• ETL
Preferred Requirement
• Implementing Lambda Architecture with stream processing and batch processing
• Implement security standards and practices in the Architecture of Cloud Solution
• Training/Certification on Data Lake HDFS
• Training/Certification on Azure Data Lake Gen 2
• Training/Certification on Data Bricks
• Experience with Azure Data Explorer and Kusto query language would be a plus.
• Experience with Azure Data Factory would be a plus.
Top Skills
What We Do
Avanceon is the leading provider of automation, control systems integration, proprietary energy management solutions and support services. It has a strong market footprint through its offices in Dubai - UAE (covering Middle East), Lahore - Pakistan (covering South Asia) and Exton, Pennsylvania - United States of America (covering North America). Avanceon is also among the select group of ‘certified members’ of Control System Integrators Association (‘CSIA’) and is also listed on the Control Engineering Magazine’s System Integrator Hall of Fame.Avanceon has been in the automation business for the last 24 years and has transformed into a 360 degree solution provider for automation, energy management, service and maintenance. Servicing clients over a span of two decades, Avanceon has completed several projects for major blue chip companies enabling it to earn strong credentials, move up the learning curve and develop a diverse client base. Going forward, it intends to capitalize on its core strengths to expand its footprint in the Middle East by setting up offices in Qatar, Kingdom of Saudi Arabia (KSA) and Oman.
