The Role
Own and operate a production Kafka estate supporting an event-driven integration platform. Responsibilities include Kafka architecture, topic and partition design, delivery guarantees, replay tooling, observability, capacity planning, resilience, CI/CD pipelines, containerized deployments, infrastructure automation, configuration management, and production support. The role partners with Python developers and global teams to maintain reliable, scalable, auditable platform operations.
Summary Generated by Built In
AgileEngine is an Inc. 5000 company that creates award-winning software for Fortune 500 brands and trailblazing startups across 17+ industries. We rank among the leaders in areas like application development and AI/ML, and our people-first culture has earned us multiple Best Place to Work awards.
WHY JOIN US
If you're looking for a place to grow, make an impact, and work with people who care, we'd love to meet you!
ABOUT THE ROLE
We are looking for a Infrastructure Engineer (Kafka) to own and operate the Kafka estate behind an event-driven integration platform, including its build, deployment, and monitoring infrastructure.
MUST HAVES
- Deep, hands-on production Kafka experience of at least 4+ years — this is the central requirement. Topic and partition design, consumer group mechanics, retention and compaction, replication, performance tuning, and troubleshooting lag, rebalances and delivery failures under load.
- Practical understanding of event-driven delivery semantics: at-least-once versus exactly-once, idempotency, ordering guarantees, dead-letter handling and replay.
- Strong enterprise Linux engineering and automation — shell plus Python scripting.
- CI/CD pipeline ownership, ideally Jenkins declarative pipelines, with real artifact and release management experience.
- Containers — Docker or Podman, OCI images — and container orchestration platforms.
- Infrastructure-as-code and configuration management.
- Observability tooling: log aggregation, metrics, alerting and SLO definition.
- GenAI-assisted development — you actively use AI coding agents in your daily workflow, or are demonstrably keen to adopt them. We automate and build with these tools across the squad and expect everyone to be fluent in them.
- Networking, TLS, secrets management and enterprise authentication (OAuth2).
- A production operations mindset: incident response, capacity planning and change control.
- Strong written and verbal communication, working with a global team.
- Upper-intermediate English level.
NICE TO HAVES
- Kafka ecosystem depth: Kafka Connect, Schema Registry, Kafka Streams, MirrorMaker, or managed Kafka services.
- Change-data-capture patterns and tooling.
- Infrastructure supporting large commercial SaaS platforms, including integration gateway or connector servers.
- Package and artifact distribution at enterprise scale.
- Experience in a regulated environment — audit retention, data classification, formal build and operate governance.
- Kubernetes; multi-region and high-availability design.
WHAT YOU WILL DO
- Own the Kafka estate — topic design and naming standards, partitioning and keying strategy, replication, retention and compaction policy, consumer group and offset management, quotas, and broker configuration and upgrades.
- Ensure platform resilience and reliability — keep the integration platform available when the upstream vendor SaaS platform is degraded or unavailable, absorb traffic spikes without overloading it, and ensure events remain replayable and auditable through a correctly sized, configured and operated Kafka infrastructure.
- Engineer the platform's delivery guarantees in practice: at-least-once and exactly-once semantics, idempotent producers, transactional writes where warranted, dead-letter topics, and self-service replay tooling for recovering from downstream failures.
- Build streaming observability: consumer lag, queue depth, throughput and rebalance monitoring; end-to-end correlation-ID tracing across topics; dashboards, service level objectives and alerting.
- Plan capacity, cost and retention for a durable log with regulatory audit retention requirements, and tune it as adoption scales across the firm.
- Evaluate and integrate vendor-native streaming and change-data-capture mechanisms as the platform's ingress and egress paths mature.
- Own and evolve the build and release pipeline — automated Jenkins pipelines producing packaged application installs for multiple enterprise Linux generations, signed container images, and generated documentation, promoted through internal artifact and image distribution services.
- Operate the runtime estate: containerised deployments, integration gateway hosts, and the ongoing operating-system modernisation of that estate.
- Engineer the infrastructure behind the platform's resilience and continuity — health detection, circuit breaking, and the reconciliation and monitoring that prove the platform recovers cleanly when the upstream vendor platform is degraded or unavailable.
- Automate environment provisioning and configuration, and keep the configuration and asset inventory a genuine single source of truth.
- Partner with the squad's Python developers on deployability, performance and cost; participate in change management and production support.
PERKS AND BENEFITS
- Professional growth: Mentorship, TechTalks, and personalized growth roadmaps.
- Competitive compensation: USD-based pay with education, fitness, and team activity budgets.
- Exciting projects: Modern solutions with Fortune 500 and top product companies.
- Flextime: Flexible schedule with remote and office options.
Meet Our Recruitment Process
Application → Coding Challenge → Video Interview → Technical Interview or Hiring Manager Interview
Each step helps us understand your skills and overall fit.
If it’s a match, you’ll receive an offer.
Skills Required
- At least 4 years of deep, hands-on production Kafka experience, including topic and partition design, consumer groups, retention, compaction, replication, performance tuning, and troubleshooting
- Understanding of at-least-once and exactly-once delivery, idempotency, ordering, dead-letter handling, and replay
- Enterprise Linux engineering and shell/Python scripting
- CI/CD pipeline ownership, preferably Jenkins declarative pipelines, with artifact and release management
- Container experience with Docker or Podman, OCI images, and orchestration platforms
- Infrastructure-as-code and configuration management experience
- Observability experience with log aggregation, metrics, alerting, and SLO definition
- Experience using or willingness to adopt GenAI-assisted development and AI coding agents
- Networking, TLS, secrets management, and enterprise authentication using OAuth2
- Production operations experience, including incident response, capacity planning, and change control
- Strong written and verbal communication skills
- Upper-intermediate English proficiency
- Kafka ecosystem experience with Kafka Connect, Schema Registry, Kafka Streams, MirrorMaker, or managed Kafka services
- Change-data-capture patterns and tooling
- Infrastructure supporting large commercial SaaS platforms, integration gateways, or connector servers
- Enterprise-scale package and artifact distribution
- Experience in regulated environments with audit retention, data classification, and formal governance
- Kubernetes and multi-region or high-availability design
Am I A Good Fit?
Get Personalized Job Insights.
Our AI-powered fit analysis compares your resume with a job listing so you know if your skills & experience align.
Success! Refresh the page to see how your skills align with this role.
The Company
What We Do
AgileEngine is a privately held company established in 2010 that builds dedicated teams of designers and developers. We turn good ideas into awesome software that people actually want to use. Some of the biggest names and the hottest startups around the world chose us to build their tech.
