Senior Site Reliability Engineer AI Security Federal

BizFirst is assisting our client with recruiting a Senior Site Reliability Engineer for AI security in a federal civilian agency environment. As the resident reliability engineer, you will keep a self-managed, distributed AI runtime security platform on AWS EKS healthy, fast, and compliant. The platform handles the agency's AI requests, making its availability and latency important to mission systems.

You will work alongside the agency's DevOps and security teams to operate, scale, and upgrade the platform, and partner with our client's engineering team to improve reliability over time.

Our client is a technology integrator that supports government customers with IT modernization, performance engineering, and cybersecurity. Its engineering teams collaborate on complex technical programs in a culture of innovation, continuous improvement, and professional development.

Operate, monitor, and troubleshoot a distributed, high-throughput platform on Amazon EKS, addressing pod and node issues, networking, storage, and performance bottlenecks.

Plan and execute upgrades to the platform, Kubernetes versions, and add-ons with minimal downtime and within agency change management.

Manage infrastructure as code with Terraform and Helm, keeping environments consistent from development through production.

Design and tune autoscaling, capacity, and load testing to meet latency and throughput targets as demand grows.

Build and maintain metrics, logs, traces, and alerting; define service level objectives (SLOs); and lead incident response and post-incident reviews.

Implement customer security requirements, including secrets management with AWS Secrets Manager and KMS, least-privilege IAM, network controls, image hardening, patching, and evidence for authorization to operate (ATO) and continuous monitoring.

Bachelor's degree in Computer Science, Engineering, or a related field, or equivalent practical experience.

Five years of experience in site reliability, DevOps, or cloud infrastructure engineering.

Experience operating production Kubernetes clusters, including monitoring, scaling, and troubleshooting containerized workloads.

Experience with core AWS services, including IAM, VPC, EC2, load balancing, KMS, Secrets Manager, and CloudWatch.

Experience managing infrastructure as code with Terraform.

Experience scripting or programming in Python, Go, or Bash.

Ability to obtain and maintain a DHS Public Trust determination.

The following qualifications are preferred, not required:

Experience running Amazon EKS, including managed node groups, Karpenter or Cluster Autoscaler, IRSA or EKS Pod Identity, and the AWS Load Balancer Controller.

Experience operating latency-sensitive, high-throughput services, including load testing and performance tuning.

Experience with Prometheus, Grafana, OpenTelemetry, and forwarding logs to a SIEM such as Splunk.

Experience with Helm and GitOps or CI/CD for Kubernetes, including Argo CD, Flux, GitHub Actions, or GitLab CI.

Experience in federal environments, including ATO, NIST SP 800-53, AWS GovCloud, FIPS endpoints, and STIG-hardened container images.

Experience with GPU node pools or serving machine learning models on Kubernetes.

Familiarity with AI and LLM applications and traffic patterns, including streaming responses and large payloads.

Certifications such as CKA, CKS, AWS Solutions Architect or SysOps Administrator, or HashiCorp Terraform Associate.

Medical, dental, and vision coverage.

Employer matching contributions to a 401(k) retirement plan.

Competitive compensation and bonus opportunities.

Support for continuing education and professional growth, including tuition reimbursement.

Paid leave and holidays.

Our client is an Equal Opportunity Employer and a U.S. Federal Government contractor. All qualified applicants receive consideration without regard to race, color, religion, sex (including pregnancy, childbirth, or related medical conditions), sexual orientation, gender identity or expression, national origin, age, disability, genetic information, marital status, veteran status, or any other protected characteristic under applicable federal, state, or local laws.

Our client provides reasonable accommodations to individuals with disabilities throughout the application and employment process. Applicants who need assistance or an accommodation due to a disability may contact our client.

U.S. citizenship is required for all positions with a government clearance and certain other restricted positions.

Location: McLean, Virginia | Onsite

Employment: Full-time

Public Trust: Ability to obtain and maintain a DHS Public Trust determination