Splunk Staff Software Engineer - Performance Optimization & Innovation (PerfOpt)

The Performance Optimization and Innovation (PerfOpt) team improves the Splunk customer experience through deep performance work, and sets Splunk's long-term performance standards through guidelines and design.

We work where performance is won or lost: search and indexing hot paths, cache and bloomfilter efficiency, S3 transfer throughput, I/O workload management, and parallelism in the data retrieval path. It's a C++ core moving petabyte-scale data across a large distributed architecture, and the results land in what customers feel - search latency, ingest throughput, and infrastructure cost. We are also the organization's performance center of gravity: the benchmarking, design patterns, and engineering standards other teams adopt come from here.

Senior performance engineers advise teams on roadmap and architecture, resolve customer performance issues, and set technical direction across partner teams.

• Lead performance work from hypothesis through instrumentation and design to measured customer impact, coordinating across teams as needed.
• Design petabyte-scale components for memory, concurrency, I/O, and tail latency, guiding architecture for scalability and reliability.
• We use profiling, flame graphs, lock contention analysis, benchmarks, and CI regression gates to identify and prevent performance issues.
• We instrument production with telemetry, eBPF, and continuous profiling to measure performance in real workloads.
• Compare performance options, quantify costs and risks, and present evidence-based recommendations to partners.
• Reusable AI workflows shorten performance analysis from days to hours. Engineers set targets, verify results with profiles and benchmarks, and adjust when evidence falls short.
• Plan performance work, estimate effort, and lead postmortems to identify root causes and prevent regressions.
• Mentor peers and less experienced engineers on the team.

Technical depth

• Advanced C++ in production, including memory management, allocators, move semantics, cache-aware structures, and performance trade-offs.
• Deep performance engineering - depth across the below, production experience with several: CPU/memory profiling and flamegraph analysis (perf, eBPF, VTune), including off-CPU and continuous profiling.Lock contention resolution - hot mutexes, false sharing, atomics and memory ordering, lock-free techniques and when not to use them.Latency optimization in distributed architectures - tail latency, queuing, fan-out amplification, backpressure, critical path analysis.Benchmarking - micro and workload benchmarks, sound test design and statistical analysis, sub-system validation CI regression detection.Performance modelling - analytical or capacity models that help predict scaling limits and validate measured results
• Architecture and system design - design and improve existing solutions for performance, scalability and operability, and explain those designs to a critical audience.
• Docker and Kubernetes - performance characterization in containerized environments.
• Strong Python for tooling, benchmark harnesses, telemetry pipelines, and performance data analysis.
• Linux performance internals - scheduler, memory subsystem, page cache, filesystems, block I/O, network stack.
• Git and CI (e.g. GitLab CI) for automating builds, tests, benchmarks, and releases.

• Track record of informing technical decisions - quantifying trade-offs, surfacing risk early.
• Distilling sophisticated problems into comparable data points without losing the nuance or hiding uncertainty.
• Clear, concise, high-signal communication across audiences - mechanism-level with engineers, trade-off-level with leadership, impact-level with customers.
• AI savviness applied to delivery speed - daily use of AI coding agents on production work, building agentic workflows the team adopts and applying a high bar for verifying AI output against profiles, benchmarks, and telemetry.
• Mentor and grow engineers - coaching junior engineers into ownership, and influencing peers through technical leadership rather than authority.

Bachelor's plus 8 years of related experience, master's plus 6 years, or PhD plus 3 years.

• Experience with search or storage engines and large-scale data platforms.
• Splunk internals.
• Hardware-level work - SIMD, NUMA, cache/TLB behavior, PGO/LTO, compiler optimization.
• Go, Rust, or another systems language alongside C++.
• AWS, Azure, or GCP -storage, networking, and instance performance characteristics.
• Terraform, Puppet, or Ansible for reproducible performance test environments.

• Performance problems at a scale very few companies have - petabytes of data, real customers, and measurable impact from every millisecond you remove!
• A constant stream of new things to learn. We're always expanding into new areas, bringing in open source projects and giving back, and exploring new technologies.
• Exceptionally versatile and dedicated peers, from engineering to product management to customer support.
• Career growth through technical ownership, leadership opportunities, and coaching.
• A stable work environment with clear ownership and measurable goals.
• Flexible hybrid work model (balanced work-from-home and in-office collaboration)
• Splunk is an equal opportunity employer. We welcome applicants of all backgrounds and provide reasonable accommodations throughout the hiring process. Benefits include flexible hybrid work, growth and mentorship opportunities, and a collaborative, supportive team.

At Cisco, we’re revolutionizing how data and infrastructure connect and protect organizations in the AI era – and beyond. We’ve been innovating fearlessly for 40 years to create solutions that power how humans and technology work together across the physical and digital worlds. These solutions provide customers with unparalleled security, visibility, and insights across the entire digital footprint.

Fueled by the depth and breadth of our technology, we experiment and create meaningful solutions. Add to that our worldwide network of doers and experts, and you’ll see that the opportunities to grow and build are limitless. We work as a team, collaborating with empathy to make really big things happen on a global scale. Because our solutions are everywhere, our impact is everywhere. 

We are Cisco, and our power starts with you.