Principal Site Reliability Engineer (Cloud, Observability & Automation)

Are you ready to make an impact at DTCC?

Do you want to work on innovative projects, collaborate with a dynamic and supportive team, and receive investment in your professional development? At DTCC, we are at the forefront of innovation in the financial markets. We are committed to helping our employees grow and succeed. We believe that you have the skills and drive to make a real impact. We foster a thriving internal community and are committed to creating a workplace that looks like the world that we serve.

The Information Technology group delivers secure, reliable technology solutions that enable DTCC to be the trusted infrastructure of the global capital markets. The team delivers high-quality information through activities that include development of essential, building infrastructure capabilities to meet client needs and implementing data standards and governance.

Pay and Benefits:

• Competitive compensation, including base pay and annual incentive

• Comprehensive health and life insurance and well-being benefits, based on location

• Pension / Retirement benefits

• Paid Time Off and Personal/Family Care, and other leaves of absence when needed to support your physical, financial, and emotional well-being.

• DTCC offers a flexible/hybrid model of 3 days onsite and 2 days remote (onsite Tuesdays, Wednesdays and a third day unique to each team or employee).

The Impact You Will Have in This Role

The Enterprise Application Support (EAS) team supports critical applications across the ITP and ECS business lines, ensuring the reliability, scalability, and performance of enterprise platforms.

As a Principal Site Reliability Engineer (SRE), you will lead reliability engineering efforts across mission-critical enterprise platforms and applications. You will be a partner with engineering, product, infrastructure, cloud, and operations teams to improve service reliability, resiliency, observability, and operational excellence.

This is a senior technical leadership role focused on establishing reliability standards, driving service health improvements, advancing observability practices, reducing operational risk, and enabling highly available, scalable systems through automation, modern engineering practices, and AI-enabled technologies.

Your Primary Responsibilities

• Drive reliability, scalability, performance, and resiliency improvements across critical enterprise applications and services.

• Define and implement reliability engineering practices, including service level indicators (SLIs), service level objectives (SLOs), error budgets, and operational KPIs.

• Design and implement observability solutions using platforms such as Grafana, Splunk, Dynatrace, OpenTelemetry, and related monitoring technologies.

• Establish effective monitoring, alerting, telemetry, and service health frameworks that enable proactive issue detection and rapid resolution.

• Partner with development teams to incorporate reliability, observability, resiliency, and operational readiness into application design and delivery processes.

• Perform architecture reviews and identify opportunities to improve system availability, performance, fault tolerance, and recoverability.

• Lead major incident response efforts, root cause analysis activities, and continuous service improvement initiatives.

• Drive alert optimization, incident reduction, and MTTR improvement programs through data-driven operational analysis.

• Champion resiliency engineering practices, including failure testing, disaster recovery preparedness, dependency analysis, and recovery validation.

• Develop automation, self-healing capabilities, and AI-assisted operational solutions using Python, Java, AWS services, Amazon Q, Kiro, or related technologies.

• Collaborate with technology and business stakeholders to identify operational risks and prioritize reliability investments.

• Mentor engineers and promote adoption of Site Reliability Engineering best practices across the organization.

Qualifications

• Bachelor’s degree in computer science, Engineering, Information Technology, or equivalent experience.

• 8+ years of experience in Site Reliability Engineering, Production Engineering, DevOps, Application Support Engineering, Software Engineering, or related disciplines.

• Experience supporting large-scale, highly available production environments.

Talent Needed for Success

• Deep understanding of Site Reliability Engineering principles and operational excellence practices.

• Strong experience defining and managing SLIs, SLOs, error budgets, and reliability metrics.

• Expertise with observability and monitoring platforms such as Grafana, Splunk, Dynatrace, ITSI, OpenTelemetry, and related technologies.

• Experience with distributed systems, microservices architectures, cloud-native applications, and enterprise platforms.

• Strong programming and automation skills using Python, Java, Go, or similar languages.

• Experience leading major incident management, root cause investigations, and service improvement programs.

• Strong knowledge of Linux/Unix systems, networking fundamentals, performance tuning, and troubleshooting methodologies.

• Experience with AWS and modern cloud technologies.

• Excellent communication, collaboration, and stakeholder management skills.

• Ability to influence engineering teams and drive reliability-focused outcomes across complex organizations.

Preferred Qualifications

• Experience implementing observability strategies and enterprise monitoring standards.

• Experience with Kubernetes, containers, CI/CD pipelines, Infrastructure as Code, and modern software delivery practices.

• Experience with AI-assisted engineering platforms such as Amazon Q, Kiro, GitHub Copilot, or similar technologies.

• Experience in financial services or other highly regulated, mission-critical environments.

• Proven track record driving reliability transformation initiatives and measurable operational improvements.

Success Measures

• Improved service availability and reliability.

• Increased SLO adoption and reliability maturity.

• Reduced incident volumes and operational noise.

• Improved MTTR and recovery effectiveness.

• Strong observability coverage and actionable telemetry.

• Increased automation and operational efficiency.

• Enhanced resiliency, recoverability, and production readiness.

Preferred Qualifications

• Experience with AI-assisted engineering tools such as Amazon Q, Kiro, or similar technologies.

• Experience designing and measuring SLOs, SLIs, and operational KPIs.

• Experience supporting large-scale enterprise applications in financial services or other highly regulated environments.

The salary range is indicative for roles at the same level within DTCC across all US locations. Actual salary is determined based on the role, location, individual experience, skills, and other considerations. We are an equal opportunity employer and value diversity at our company. We do not discriminate on the basis of race, religion, color, national origin, sex, gender, gender expression, sexual orientation, age, marital status, veteran status, or disability status. We will ensure that individuals with disabilities are provided reasonable accommodation to participate in the job application or interview process, to perform essential job functions, and to receive other benefits and privileges of employment. Please contact us to request accommodation.

With over 50 years of experience, DTCC is the premier post-trade market infrastructure for the global financial services industry. From 20 locations around the world, DTCC, through its subsidiaries, automates, centralizes, and standardizes the processing of financial transactions, mitigating risk, increasing transparency, enhancing performance and driving efficiency for thousands of broker/dealers, custodian banks and asset managers. Industry owned and governed, the firm innovates purposefully, simplifying the complexities of clearing, settlement, asset servicing, transaction processing, trade reporting and data services across asset classes, bringing enhanced resilience and soundness to existing financial markets while advancing the digital asset ecosystem. In 2024, DTCC’s subsidiaries processed securities transactions valued at U.S. $3.7 quadrillion and its depository subsidiary provided custody and asset servicing for securities issues from over 150 countries and territories valued at U.S. $99 trillion. DTCC’s Global Trade Repository service, through locally registered, licensed, or approved trade repositories, processes more than 25 billion messages annually. To learn more, please visit us at www.dtcc.com or connect with us on LinkedIn, X, YouTube, Facebook and Instagram.

DTCC proudly supports Flexible Work Arrangements favoring openness and gives people freedom to do their jobs well, by encouraging diverse opinions and emphasizing teamwork. When you join our team, you’ll have an opportunity to make meaningful contributions at a company that is recognized as a thought leader in both the financial services and technology industries. A DTCC career is more than a good way to earn a living. It’s the chance to make a difference at a company that’s truly one of a kind.

Learn more about Clearance and Settlement by clicking here.

Serves as a dedicated technology resource for advancing DTCC’s business opportunities and providing industry thought leadership for leveraging new technology. The goal of this new department is to partner internally with IT, our business and regulatory divisions and externally with clients, regulators, and fintech vendors, to help build new platforms and business models to advance DTCC’s mission to support the financial markets.