Big Data Engineer (Python, Java, Analytics, Storage)
Role Overview:
We’re looking for a highly skilled Big Data Engineer to help architect and implement large-scale data storage and analytics platforms as part of a global client implementation. You’ll be responsible for managing data flow from ingestion through to storage and insight generation, using both open-source and enterprise technologies.
Key Responsibilities:
- Design and build end-to-end data pipelines from ingestion to analytics-ready format.
- Leverage Python and Java to process large datasets, build transformation logic, and optimize performance.
- Integrate real-time and batch data flows using Kafka, RabbitMQ, and NiFi.
- Model and manage scalable data storage solutions for analytics using HDFS, S3, or data lake tools.
- Support business and client teams with accessible, structured datasets and metadata tagging.
- Enable searchability and observability using ElasticSearch and visual tools.
Required Skills:
- 4+ years’ experience in big data engineering, with proven projects in implementation settings.
- Proficient in Python and Java for large-scale ETL and transformation.
- Experience with Kafka, RabbitMQ, NiFi, and streaming architecture design.
- Deep understanding of data lake / warehouse design, data partitioning, and indexing.
- Experience with ElasticSearch and Zookeeper in a production context.
- Familiarity with security and compliance practices in multi-tenant data environments.
Nice to Have:
- Experience integrating with BI tools or data visualization platforms.
- Knowledge of ML model deployment pipelines.
Other Requirements
- Eligibility for Top-Level Security Clearance:
- Candidates must be eligible to obtain and maintain security clearance at the highest level, in accordance with applicable national security regulations.
- On-Site Work Requirement:
- This role requires full-time, on-site presence at the client’s premises located in Pretoria. Remote or hybrid work arrangements are not applicable.
Similar Jobs
What We Do
Mi-C3 International Limited is a Malta based company that was established in 2014 by the CEO and founder, Glen Scott.
The history and the philosophy of Mi-C3 is in its name which is a combination of two acronyms. The genesis of the technology was in supporting Command and Control for organisations that needed technology to improve their situational awareness to manage and secure their facilities. However, our technology enabled Command and Control to be enhanced with Collaboration (C3) as organisations were empowered with multi-directional communications that took command and control to a new level as more decision-making could be automated and information flows organised in a way that reams of data are transformed into meaningful intelligence (Mi).
Generations of the Mi-C3 software started as a project in the early 2000’s as a solution to support a large oil and gas company operating across multiple countries to help manage its complex operation and considerable security risk spread over a wide and poorly accessible geographical area before the Internet of Things (IoT) was widely understood, and has subsequently grown to over 30,000 sites, and monitoring more than 1 million end points.
Over the years, this solution has evolved into our award winning* flagship product, AFFECTLI, and has delivered significant value to clients through the use of a combination of cutting-edge technologies in the fields of Digital Process Transformation, smart workspaces, AI, machine learning, data management and analytics.
AFFECTLI is an established, flagship product of Mi-C3 that won the Malta MCA eBusiness Award for Best B2B Application 2018, was shortlisted for a nomination in the World Summit Awards 2018 under the Business & Commerce category, is ISO 9001:2015; ISO/IEC 20000:2011 and SOC 2 Type II compliant.
