Consulting/Principal Site Reliability Engineer

About the Role

We're looking for a Consulting/Principal Site Reliability Engineer to guide our observability and reliability initiative for Government Systems, running on Azure Government. This is an advanced individual-contributor role for someone with a proven track record leading an organization through a transition to SLO-based reliability engineering — you've done this before, and you know what it takes to get development teams bought in. This is a hands-on SRE role, not a software development role: you'll work alongside other SREs on the initiative, guiding development teams on telemetry and instrumentation strategy rather than writing that code yourself. You'll operate as the go-to expert on Datadog and observability practice, helping define meaningful SLOs and error budgets and making data-driven reliability decisions. You'll also help train and mentor junior SREs, working within priorities set by the SRE Manager.

Eligibility: This role requires U.S. citizenship and the ability to obtain and maintain a Public Trust clearance.

What You'll Do

• Guide the observability and reliability initiative, working alongside other SREs to set the technical direction for how the organization measures and improves reliability

• Lead the transition to SLO-based reliability engineering, building on prior experience doing this elsewhere

• Guide development teams on telemetry and instrumentation strategy for their applications, without doing the application development work yourself

• Serve as the primary technical authority on Datadog, from dashboarding to alerting to SLO tracking

• Recommend service level objectives in partnership with Product and Development teams

• Act as a senior escalation point during incidents

• Help improve and evolve the broader SRE framework

• Build disaster recovery plans, including advanced fault injection scenarios

• Advise on SRE training curriculum and content, and help set and follow priorities established by the SRE Manager

Tech Stack You'll Work With

• Cloud: Microsoft Azure Government

• Server Environment: Linux and Windows

• CI/CD: Azure DevOps Pipelines, GitHub Actions

• Infrastructure as Code: Terraform

• Observability: Datadog (primary)

• AI-assisted development: GitHub Copilot

What We're Looking For

Observability (core focus)

• Significant, hands-on experience with Datadog — dashboards, alerting, SLO/error-budget tracking, and telemetry pipelines

• Proven experience leading an organization's transition to SLO-based reliability engineering, not just supporting one

• Track record guiding development teams on instrumentation and telemetry strategy (advisory, not hands-on application coding)

• Deep technical understanding of observability techniques across the full stack; able to bring clarity to complex incidents or performance issues

• Able to build templated observability dashboards and configuration as code so others can implement quickly

• Able to set and influence appropriate SLOs and Error Budgets org-wide

• Comfortable operating across both Linux and Windows server environments

Incident Management

• Experience preparing and assessing SREs for on-call readiness, including mentoring and identifying training needs

• Skilled at running post-mortems where participants feel safe to contribute; fosters a culture of learning from failure

• Comfortable acting as a senior on-call escalation point and guiding restoration efforts

• Experience partnering with SRE and Development leads to identify and fix root causes of excessive alerting and pager load

Design for Reliability

• Advanced understanding of systems design, including high availability, deployment, and recovery techniques, plus failure modes and system behaviors

• Specialized depth in one or more areas such as denial-of-service protection or containerization, with the ability to consult others on it

• Able to make resilience and recoverability decisions backed by load testing data and fault injection results

• Experience documenting SRE design practices and contributing to a broader SRE framework

Disaster Recovery

• Experience contributing to DR plans, including recovery priorities, procedures, and guiding others through runbooks

• Able to design and lead DR practice scenarios and incident response exercises, identifying knowledge gaps and improvements

• Experience planning chaos engineering scenarios to simulate faults across memory, network, I/O, or distributed systems

Platforms and Automation

• Able to build engineering and architecture consensus for new standard components and services on the Paved Road, including platforms and CI/CD

• Experience promoting adoption and standardization, with contributions back via inner sourcing

• A champion for Paved Road adoption, driving efficiency by removing rework and silos

Reliability Culture

• Creates and shares SRE best practices and training material

• Comfortable coaching other SREs and identifying development gaps

• Able to spot and eliminate large-scale toil

AI-Driven Development (desired)

• Experience using GitHub Copilot to accelerate troubleshooting, automation, or reliability tooling

• Interest in helping define best practices for AI-assisted development within an SRE context

Behavioral Competencies

• Collaboration and Teamwork

• Customer & External Focus

• Solves Problems and Analyzes Issues

• Learning Agility

• Builds Relationships

• Develops Others

U.S. National Base Pay Range: $118,300 - $219,800. Geographic differentials may apply in some locations to better reflect local market rates.

This job is eligible for an annual incentive bonus.

We know your well-being and happiness are key to a long and successful career. We are delighted to offer country specific benefits. Click here to access benefits specific to your location.

We are committed to providing a fair and accessible hiring process. If you have a disability or other need that requires accommodation or adjustment, please let us know by completing our Applicant Request Support Form or please contact 1-855-833-5120.

Criminals may pose as recruiters asking for money or personal information. We never request money or banking details from job applicants. Learn more about spotting and avoiding scams here.

Please read our Candidate Privacy Policy.

We are an equal opportunity employer: qualified applicants are considered for and treated during employment without regard to race, color, creed, religion, sex, national origin, citizenship status, disability status, protected veteran status, age, marital status, sexual orientation, gender identity, genetic information, or any other characteristic protected by law.

USA Job Seekers:

EEO Know Your Rights.