Splunk Senior Staff Software Engineer - Performance Optimization & Innovation (PerfOpt)

The Performance Optimization and Innovation (PerfOpt) team improves the Splunk customer experience through deep performance work, and sets Splunk's long-term performance standards through guidelines and design.

We work where performance is won or lost: search and indexing hot paths, cache and bloomfilter efficiency, S3 transfer throughput, I/O workload management, and parallelism in the data retrieval path. It's a C++ core moving petabyte-scale data across a large distributed architecture, and the results land in what customers feel - search latency, ingest throughput, and infrastructure cost. We are also the organization's performance center of gravity: the benchmarking, design patterns, and engineering standards other teams adopt come from here.

Serve as the performance authority for critical issues, roadmap planning, architecture reviews, and customer conversations. Set technical direction across teams.

• Lead a performance domain end to end -hypothesis, instrumentation, design, measured customer-visible gain in production including cross-team problems nobody has framed yet.
• Architect for scale - design and re-architect subsystems in a distributed, petabyte-scale system: memory hierarchy, concurrency, I/O, tail-latency.
• Be responsible for the performance methodology - profiling and flame graph practice, lock contention analysis, benchmarking, CI regression gating.
• Introduce observability - design new telemetry sources and bring eBPF and continuous profiling into production use, so performance is measurable in the field, not just the lab.
• Enable leadership decisions - explain Directors and Managers on what the problem costs, what each option buys and costs, and the risk of inaction, with a recommendation you stand behind. A multi-week investigation becomes a one-page comparison they can act on in ten minutes.
• Deliver fast and make everyone faster - work agentically, and build reusable AI agentic workflows and tooling that cut the team's analysis loops from days to hours. Navigate AI optimization: point agents at changes that move the numbers, catch work that fails under a profile, redirect quickly.
• Lead and mentor senior and staff engineers.

Technical depth

• Expert-level C++ in production systems - memory management and allocators, move semantics, cache-friendly data structures, and a real command of the language's cost model.
• Deep performance engineering - depth across all of the below, production experience with several:
• CPU/memory profiling and flame-graph analysis (perf, eBPF, VTune), including off-CPU and continuous profiling.
• Lock contention resolution - hot mutexes, false sharing, atomics and memory ordering, lock-free techniques and when not to use them.
• Latency optimization in distributed architectures - tail latency, queuing, fan-out amplification, back-pressure, critical path analysis.
• Benchmarking - micro and workload benchmarks, sound test design and statistical analysis, sub-system validation CI regression detection.
• Performance modelling - analytical and capacity models that predict scaling limits and validate measured results
• Architecture and system design -design and re-architect existing solutions for performance, scalability and operability, and defend those designs to a critical audience.
• Docker and Kubernetes - performance characterization in containerized environments.
• Strong Python for tooling, benchmark harnesses, telemetry pipelines, and performance data analysis.
• Linux performance internals - scheduler, memory subsystem, page cache, filesystems, block I/O, network stack.
• Git and CI (e.g. GitLab CI) for automating builds, tests, benchmarks, and releases.

• Track record of informing leadership decisions - quantifying trade-offs, surfacing risk early.
• Distilling complex problems into comparable data points without losing the nuance or hiding uncertainty.
• Clear, concise, high-signal communication across audiences - mechanism-level with engineers, trade-off-level with leadership, impact-level with customers.
• AI savviness applied to delivery speed - daily use of AI coding agents on production work, building agentic workflows others adopt and setting the bar for verifying AI output against profiles, benchmarks, and telemetry.
• Lead, mentor and grow engineers at every level - coaching junior engineers into ownership, and influencing senior and staff peers through technical leadership rather than authority.

• Bachelor's + 12 years, Master's + 8 years, or PhD + 5 years of related experience, with specialized depth and breadth sufficient to advise management.

• Petabyte-scale data flow - ingest, storage, retrieval and search at scale, object storage (S3) access patterns, caching and eviction, I/O workload management.
• Splunk internals, or a comparable search, analytics, or large-scale data platform.
• Storage / query engine, or distributed search optimization.
• Hardware-level work - SIMD, NUMA, cache/TLB behavior, PGO/LTO, compiler optimization.
• Go, Rust, or another systems language alongside C++.
• AWS, Azure, or GCP -storage, networking, and instance performance characteristics.
• Terraform, Puppet, or Ansible for reproducible performance test environments.

• Performance problems at a scale very few companies have - petabytes of data, real customers, and measurable impact from every millisecond you remove!
• A constant stream of new things to learn. We're always expanding into new areas, bringing in open source projects and giving back, and exploring new technologies.
• Exceptionally versatile and dedicated peers, from engineering to product management to customer support.
• Career growth through technical ownership, leadership opportunities, and coaching.
• A stable work environment with clear ownership and measurable goals.
• Flexible hybrid work model (balanced work-from-home and in-office collaboration)
• Splunk is an equal opportunity employer. We welcome applicants of all backgrounds and provide reasonable accommodations throughout the hiring process. Benefits include flexible hybrid work, growth and mentorship opportunities, and a collaborative, supportive team.

At Cisco, we’re revolutionizing how data and infrastructure connect and protect organizations in the AI era – and beyond. We’ve been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds. These solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint.

Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions. Add to that our worldwide network of doers and experts, and you’ll see that the opportunities to grow and build are limitless. We work as a team, collaborating with empathy to make really big things happen on a global scale. Because our solutions are everywhere, our impact is everywhere. 

We are Cisco, and our power starts with you.