Data Engineer

Reposted 12 Days Ago
Be an Early Applicant
Johannesburg, City of Johannesburg, Gauteng
In-Office
Mid level
Software
The Role
Design and maintain real-time data integration pipelines using Apache NiFi, handle diverse data sources, and ensure data quality.
Summary Generated by Built In

About MI-C3

Established in 2014 by CEO and founder Glen Scott, MI-C3 International Limited is a Malta-based company specializing in delivering trusted software solutions tailored for mission-critical environments. Our flagship product, AFFECTLI, empowers organizations to make informed, data-driven decisions by providing a consolidated, real-time view of complex operations. We pride ourselves on fostering a collaborative, agile work environment that celebrates diversity, rewards 

Data Integrations Engineer (NiFi)

We are seeking a mid-level Data Integration Engineer with hands-on experience in Apache NiFi to join our dynamic team. In this role, you will design, implement, and maintain real-time data integration pipelines, handling data from diverse sources such as IoT/IIoT devices, third-party APIs, and raw files. Your primary focus will be on processing streaming data to provide valuable insights that drive informed decision-making within our organization.

As MI-C3 transitions towards Fluvio and Rust, experience with these technologies will be advantageous but is not mandatory. The ideal candidate will possess a deep understanding of data pipelines, real-time event streaming, and ETL workflows, coupled with a passion for exploring and implementing new technologies.

Key Responsibilities

  • Collaborate with cross-functional teams to design and implement scalable, real-time data streaming solutions using Apache NiFi.
  • Ingest and process data from various sources, including IoT/IIoT protocols (e.g., MQTT, SNMP, CoAP, TCP, WebSockets) and third-party APIs.
  • Develop and maintain robust ETL pipelines, ensuring data is transformed and loaded efficiently for analysis and storage.
  • Continuously monitor and optimize data workflows to maintain low-latency, high-throughput processing capabilities.
  • Configure and manage message brokers such as Kafka, RabbitMQ, and AMQP to facilitate efficient data exchange and support event-driven architectures.
  • Implement validation checks and quality measures to ensure the accuracy, reliability, and integrity of integrated data.
  • Proactively identify, diagnose, and resolve issues related to data ingestion, transformation, and streaming processes to ensure uninterrupted data flow.

Technical Requirements

  • Demonstrated experience in designing and implementing data integration solutions using Apache NiFi for real-time streaming data.
  • Strong skills in Java and Python for developing custom data processing components and applications.
  • Familiarity with tools such as Apache Spark and Kafka for building scalable data integration solutions.
  • Experience configuring and managing message brokers like RabbitMQ, AMQP, and Kafka to enable efficient data exchange.
  • Hands-on experience with protocols such as MQTT, SNMP, CoAP, TCP, and WebSockets for data capture from edge devices and industrial systems.
  • Knowledge of data validation techniques and quality assurance practices to ensure reliable data integration.
  • Strong analytical and problem-solving abilities, with a keen attention to detail.
  • Excellent communication and teamwork skills to effectively collaborate with cross-functional teams.
  • A proactive mindset with a willingness to learn and work with new tools and technologies, including Fluvio and Rust.

Preferred Qualifications

  • Bachelor's degree in Computer Science, Information Technology, or a related field.
  • Familiarity with Fluvio and Rust is a plusExperience with cloud-based platforms and distributed systems is advantageous.
  • Understanding of embedded systems, requirements engineering, and systems integration is beneficial.

What We Offer

  • Be part of a forward-thinking company that values innovation and continuous improvement.
  • Opportunities for professional development and career advancement within a growing organization.
  • A supportive and inclusive work environment that values diversity and collaboration.
  • A comprehensive compensation package commensurate with experience and qualifications.

Top Skills

Amqp
Apache Nifi
Spark
Coap
Fluvio
Java
Kafka
Mqtt
Python
RabbitMQ
Rust
Snmp
Tcp
Websockets
Am I A Good Fit?
beta
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.

The Company
HQ: Birkirkara
52 Employees
Year Founded: 2014

What We Do

Mi-C3 International Limited is a Malta based company that was established in 2014 by the CEO and founder, Glen Scott.

The history and the philosophy of Mi-C3 is in its name which is a combination of two acronyms. The genesis of the technology was in supporting Command and Control for organisations that needed technology to improve their situational awareness to manage and secure their facilities. However, our technology enabled Command and Control to be enhanced with Collaboration (C3) as organisations were empowered with multi-directional communications that took command and control to a new level as more decision-making could be automated and information flows organised in a way that reams of data are transformed into meaningful intelligence (Mi).

Generations of the Mi-C3 software started as a project in the early 2000’s as a solution to support a large oil and gas company operating across multiple countries to help manage its complex operation and considerable security risk spread over a wide and poorly accessible geographical area before the Internet of Things (IoT) was widely understood, and has subsequently grown to over 30,000 sites, and monitoring more than 1 million end points.

Over the years, this solution has evolved into our award winning* flagship product, AFFECTLI, and has delivered significant value to clients through the use of a combination of cutting-edge technologies in the fields of Digital Process Transformation, smart workspaces, AI, machine learning, data management and analytics.

AFFECTLI is an established, flagship product of Mi-C3 that won the Malta MCA eBusiness Award for Best B2B Application 2018, was shortlisted for a nomination in the World Summit Awards 2018 under the Business & Commerce category, is ISO 9001:2015; ISO/IEC 20000:2011 and SOC 2 Type II compliant.

Similar Jobs

LexisNexis Logo LexisNexis

Data Engineer

Information Technology • Legal Tech • Professional Services • Analytics • Business Intelligence
In-Office
4 Locations
10001 Employees

InfyStrat Software Services Logo InfyStrat Software Services

Data Engineer

Digital Media • Information Technology • Software
In-Office
Johannesburg, Gauteng, ZAF
177 Employees
In-Office
Johannesburg, City of Johannesburg, Gauteng, ZAF
373 Employees

Access Bank Logo Access Bank

Data Engineer

Fintech • Software • Financial Services
In-Office
Sandton, City of Johannesburg, Gauteng, ZAF
21374 Employees

Similar Companies Hiring

Standard Template Labs Thumbnail
Software • Information Technology • Artificial Intelligence
New York, NY
10 Employees
PRIMA Thumbnail
Travel • Software • Marketing Tech • Hospitality • eCommerce
US
15 Employees
Scotch Thumbnail
Software • Retail • Payments • Fintech • eCommerce • Artificial Intelligence • Analytics
US
25 Employees

Sign up now Access later

Create Free Account

Please log in or sign up to report this job.

Create Free Account