Kafka Lead Engineer

About Our Team

Our employees thrive in a culture that's fast-paced and ego-free, where innovation and collaboration are encouraged at every turn. We are an organization that provides federal agencies and commercial clients instant access to experienced and talented professionals who understand their unique challenges and know the most efficient ways to address them. We are continually investing in resources and talent, so we stay prepared with specialized teams in the place who are experts in creating tailored technologies. Our solutions empower our clients to grow, modernize, and succeed in a rapidly evolving landscape.

We value all voices and want to attract talent from all backgrounds. We are on the lookout for individuals who are passionate about technology and thrive in environments where problem-solving is approached with creativity and enthusiasm. If you are someone who enjoys continuously expanding your skill set while tackling real-world business problems, you will feel right at home with us. Veterans and military spouses are especially encouraged to bring your unique and valuable experience to our team.

Overview: 

We are seeking a highly skilled Kafka Lead Engineer to join our team and work closely with the Army on mission-critical data stream engineering projects. In this role, you will play a key part in delivering essential data solutions in an artificial intelligence (AI) and low code no code (LCNC) architecture, helping the Army transform and optimize their data infrastructure. The ideal candidate will have a deep technical background in Kafka development and a strong focus on customer engagement, especially in adapting solutions to fit the Army's unique needs and working in an agile environment. 

As the Kafka Lead, you will design, develop, and implement data pipelines, streams, and connectors that organize and correlate disparate data, enabling real-time access for business consumers. Your ability to integrate Kafka technology with the Army’s mission and operational requirements is critical to the success of the project. 

Key Responsibilities: 

• Kafka Architecture and Engineering: Lead the design and development of Kafka-based data pipelines and streaming solutions, ensuring they meet the high availability, scalability, disconnected operations, and performance needs of the Army's mission-critical systems. 
• Data Integration: Develop connectors, transformations, and integration patterns that make disparate data accessible in real-time for business consumers, ensuring consistency, reliability, and speed. 
• Agile Collaboration: Work closely with cross-functional teams, including project managers, architects, data engineers, and Army stakeholders, to deliver solutions that align with their goals and are adaptable in an agile environment. 
• Customer-Focused Solutions: Engage with Army personnel to understand their workflows, operational needs, and mission goals. Tailor Kafka-based solutions to meet specific Army requirements, ensuring that the data infrastructure aligns with their unique operational context. 
• Mentorship and Leadership: Lead and mentor associate Kafka developers, guiding them through best practices, code reviews, and technical challenges. Provide direction to ensure all Kafka solutions are implemented efficiently and effectively. 
• Optimization and Troubleshooting: Monitor Kafka systems for performance, troubleshoot issues, and optimize pipeline efficiency, ensuring minimal downtime and continuous data flow. 
• Documentation and Best Practices: Create clear, detailed documentation of Kafka architectures, processes, and solutions. Ensure adherence to best practices, coding standards, and operational processes. 

TAG: #LI-I4DM

Required Skills and Qualifications: 

• Proven experience as a Kafka Engineer or Kafka Lead, with a strong background in architecting, designing and implementing Kafka-based data pipelines and real-time data streaming solutions. 
• Expertise in Kafka architecture, streaming technologies, and data pipeline development (including Kafka Streams, Kafka Connect, and related tools). 
• Expertise in Kafka Performance tuning, data tagging and data aggregation  
• Expertise in building DataOps practices, primarily as it relates to automating testing and deployments of Kafka based data products through integrated CI/CD pipelines. 
• Agile methodology experience, with the ability to adapt to fast-paced, changing environments and work closely with teams using agile frameworks (Scrum, Kanban, etc.). 
• Strong customer focus with the ability to engage directly with clients, understand their needs, and deliver technical solutions that align with their business goals. 
• Experience working in government or defense sectors is a plus, with a strong understanding of how to work in highly regulated and mission-critical environments. 
• Proficiency in Java, Scala, Python or other relevant programming languages for Kafka development. 
• Experience with Kafka-related tools (e.g., Kafka Streams, Kafka Connect, Schema Registry). 
• Strong understanding of data integration, ETL processes, and real-time analytics. 
• Excellent problem-solving, communication, and collaboration skills. 
• Security clearance or the ability to obtain one is preferred (depending on project requirements). 

Preferred Qualifications: 

• Familiarity with Army-specific systems, workflows, and organizational culture. 
• Experience integrating Kafka with Appian Data fabric 
• Experience using Kafka to automate data tagging 
• Experience with cloud-based solutions (e.g., AWS, Azure) and deploying Kafka on cloud platforms. 
• Strong experience in data modeling, distributed systems, and microservices architecture.