Data Platform Engineer (Kafka/ MQ/ Redis)

Job Brief
We are seeking a skilled Data Platform Engineer with hands-on experience in Kafka, Message Queues (MQ), and Redis to design, implement, and maintain high-performance messaging and caching systems. The engineer will support real-time data streaming, asynchronous communication, and in-memory data platforms that connect critical business applications and databases. This role focuses on ensuring reliability, scalability, and low latency across data pipelines and middleware infrastructure in Telecom and Digital Financial Services (DFS) environments.
VentureDive Overview
Founded in 2012 by veteran technology entrepreneurs from MIT and Stanford, VentureDive is the fastest-growing technology company in the region that develops and invests in products and solutions that simplify and improve the lives of people worldwide. We aspire to create a technology organization and an entrepreneurial ecosystem in the region that is recognized as second to none in the world.
Key Responsibilities:

• Deploy, configure, and manage Apache Kafka clusters, topics, partitions, and consumer groups for event-driven and streaming architectures.
• Install and maintain Message Queue (MQ) systems such as RabbitMQ or ActiveMQ for asynchronous data
• flow between microservices and databases.
• Manage and optimize Redis instances for caching, session storage, and high-speed data retrieval.
• Define and implement replication, retention, and partitioning strategies for Kafka topics.
• Design and manage streaming data integration between Kafka/MQ systems and RDBMS/NoSQL databases.
• Implement High Availability (HA) and Disaster Recovery (DR) configurations for Kafka, MQ, and Redis.
• Collaborate with development and database teams to create event-driven data pipelines and optimize message throughput.
• Monitor and tune performance using Prometheus, Grafana, or ELK Stack.
• Secure Kafka and MQ infrastructures using SASL, TLS, and role-based access controls.
• Automate operational workflows and deployments using Bash or Python scripting.
• Maintain detailed technical documentation, SOPs, and deployment diagrams.

Required Experience & Qualification:

• 2–4 years of experience in Kafka, MQ, and Redis administration and support.
• Strong understanding of Kafka architecture — brokers, topics, producers, consumers, and partitions.
• Proficiency with RabbitMQ or ActiveMQ configuration, clustering, and queue management.
• Experience with Redis clustering, Sentinel, persistence (RDB/AOF), and replication.
• Familiarity with Kafka Connect, MirrorMaker, and Schema Registry for data integration and replication.
• Knowledge of message delivery semantics (at-most-once, at-least-once, exactly-once).
• Strong Linux administration skills (Red Hat, Oracle Linux, CentOS, SUSE).
• Understanding of networking, load balancing, and distributed system principles.
• Hands-on experience with monitoring and alerting tools such as Prometheus or Grafana.
• Scripting and automation proficiency in Bash or Python.

Preferred Qualifications

• Experience with cloud-based messaging and caching services (AWS MSK, Azure Event Hubs, AWS ElastiCache, Google Pub/Sub).
• Familiarity with Kafka Streams, Flink, or Spark Streaming.
• Exposure to database integration for near real-time replication or event capture.
• Certifications in Kafka, Redis, or MQ technologies are a plus.

Soft Skills

• Strong analytical and problem-solving mindset.
• Excellent communication and teamwork abilities.
• Proactive, detail-oriented, and highly reliable under pressure.
• Ability to handle complex, high-throughput production systems independently.

Education

•  Bachelor’s degree in Computer Science, Information Technology, or related field.

What we look for beyond required skills
In order to thrive at VentureDive, you
…are intellectually smart and curious
…have the passion for and take pride in your work
…deeply believe in VentureDive’s mission, vision, and values
…have a no-frills attitude
…are a collaborative team player
…are ethical and honest
Are you ready to put your ideas into products and solutions that will be used by millions?
You will find VentureDive to be a quick pace, high standards, fun and a rewarding place to work at. Not only will your work reach millions of users world-wide, you will also be rewarded with competitive salaries and benefits. If you think you have what it takes to be a VenDian, come join us ... we're having a ball!    
#LI-Hybrid