Lead Engineer - L3 Observability

Presidio, Where Teamwork and Innovation Shape the Future
At Presidio, we're at the forefront of a global technology revolution, transforming industries through cutting-edge digital solutions and next-generation AI. We empower businesses - and their internal customers - to achieve more through innovation, automation, and intelligent insights.

The Role

Level 3 Managed Service Engineer serves as the senior escalation point and technical authority for end-to-end observability across the environment. This role is responsible for resolving or driving resolution of highly complex incidents spanning application, infrastructure, network, and third-party dependencies, while owning observability strategy, Datadog architecture, monitoring standards, and continuous improvement. The engineer is expected to guide L1 and L2 engineers, lead root cause analysis and major incident reviews, and partner with cross-functional teams to improve detection, reduce noise, and strengthen operational resilience for both 3-tier and cloud-native applications. 

Key Responsibilities

• Serve as the final escalation and technical authority for complex, cross-domain incidents involving application, infrastructure, network, and third-party dependencies.
• Own the end-to-end observability architecture and strategy in Datadog, including instrumentation standards, tagging strategy, dashboard governance, and monitor design principles.
• Lead root cause analysis for major/critical incidents and drive post-incident reviews, ensuring permanent fixes and preventive monitoring are implemented.
• Define and continuously improve monitoring coverage for new and existing services, including 3-tier applications, microservices, and cloud-native workloads.
• Design and govern SLOs/SLIs aligned to business-critical services, and advise application and platform teams on instrumentation best practices.
• Drive automation initiatives: Datadog-as-code (Terraform/API-based configuration management), auto-remediation workflows, and integration with CI/CD pipelines.
• Partner with application, infrastructure, network, and security teams to close monitoring gaps and reduce mean time to detect (MTTD) and mean time to resolve (MTTR).
• Own capacity planning and performance engineering insights derived from long-term observability data trends.
• Manage Datadog platform health: cost optimization, license/usage governance, data retention, and integration management.
• Mentor L1 and L2 engineers, review their RCA quality, and build enablement material (runbooks, training, best practices).
• Act as a key stakeholder in client governance calls, presenting trends, risks, and improvement roadmaps.
• Evaluate and pilot new Datadog features or complementary observability tools, and lead their rollout.
• Define escalation matrices and coordinate cross-team response during major outages.

Required Skills and Experience

• 6+ years of experience in Managed Services, SRE, or platform engineering roles, with at least 3 years of hands-on ownership of a Datadog (or equivalent) observability platform at scale.
• Expert-level knowledge of Datadog, including:
• Platform architecture: organization/team structuring, RBAC, tagging strategy, and multi-environment setups.
• APM: distributed tracing at scale, service dependency mapping, and instrumentation strategy across polyglot environments.
• Synthetics and RUM: designing comprehensive digital experience monitoring strategies tied to business KPIs.
• Log Management: pipeline architecture, log-based metrics, and cost/volume governance.
• SLOs, error budgets, and anomaly/forecast-based alerting at an advanced level.
• Deep understanding of infrastructure and cloud architecture: compute, containers, orchestration (Kubernetes), auto-scaling, and multi-cloud/hybrid environments.
• Deep understanding of networking: routing, load balancing, CDN, DNS, and complex connectivity troubleshooting across hybrid environments.
• Strong incident and problem management leadership experience, including major incident command and post-incident review facilitation.
• Experience defining and governing observability standards across multiple teams or business units.

Application & Architecture Understanding (technology-agnostic)

• Expert-level understanding of 3-tier and N-tier application architectures, including how failures cascade across presentation, application, and data layers.
• Expert-level understanding of cloud-native and distributed architectures: microservices, event-driven systems, service meshes, API gateways, and asynchronous messaging.
• Ability to advise application teams on observability best practices (instrumentation, tracing context propagation, structured logging) without needing to write code in a specific language.
• Strong grasp of resiliency and failure patterns (retries, circuit breakers, backpressure, caching, graceful degradation) and how to detect their failure modes through observability data.

Good to Have

• Datadog advanced/expert certifications, or equivalent recognized observability certifications.
• Strong scripting/automation background (Python, Terraform, or similar) for Datadog-as-code and custom tooling.
• Experience with FinOps or cost governance for observability tooling at scale.
• Exposure to multiple observability ecosystems (New Relic, Dynatrace, Splunk, Grafana/Prometheus, OpenTelemetry).
• Experience presenting to senior stakeholders or clients on observability maturity and roadmaps.

Soft Skills

• Strategic thinking with the ability to balance immediate incident needs against long-term platform health.
• Strong leadership and mentoring capability across L1/L2 engineers.
• Excellent stakeholder management and executive communication skills.
• Calm, decisive leadership during major incidents.
• Strong documentation and process-governance discipline.

Education

• Bachelor's or Master's degree in Computer Science, Information Technology, Electronics, or a related field (or equivalent practical experience).

Work Details

• Level: L3 (Senior/Lead, Managed Services)
• Shift: Primarily general shift, with on-call responsibility for major incident escalations
• Reporting To: Observability Practice Lead / Delivery Manager

Your future at Presidio
Joining Presidio means stepping into a culture of trailblazers - thinkers, builders, and collaborators - who push the boundaries of what's possible. With our expertise AI-driven analytics, cloud solutions, cybersecurity, and next-gen infrastructure, we enable businesses to stay ahead in an ever-evolving digital world.

Here, your impact is real. Whether you're harnessing the power of Generative AI, architecting resilient digital ecosystems, or driving data-driven transformation, you'll be part of a team that is shaping the future.

Ready to innovate? Let's redefine what's next-together.

About Presidio
Presidio is committed to hiring the most qualified candidates to join our amazing culture. We aim to attract and hire top talent from all backgrounds, including underrepresented and marginalized communities. We encourage women, people of color, people with disabilities, and veterans to apply for open roles at Presidio. Diversity of skills and thought is a key component to our business success.

At Presidio, speed and quality meet technology and innovation. Presidio is a trusted ally for organizations across industries with a decades-long history of building traditional IT foundations and deep expertise in AI and automation, security, networking, digital transformation, and cloud computing. Presidio fills gaps, removes hurdles, optimizes costs, and reduces risk. Presidio's expert technical team develops custom applications, provides managed services, and enables actionable data insights and builds forward-thinking solutions that drive strategic outcomes for clients globally. For more information visit https://presidio.com

 Applications will be accepted on a rolling basis.

Presidio has a strong commitment to the community we serve and our employees. As an Equal Opportunity Employer, we strive to have a workforce that includes the community we serve.

 Recruitment Agencies, Please Note: Presidio does not accept unsolicited agency resumes/CVs. Do not forward resumes/CVs to our career's email address, Presidio employees or any other means. Presidio is not responsible for any feeds related to unsolicited resumes/CVs.