JobCubby JobCubby
Sign up
Perplexity

Member of Technical Staff (Site Reliability Engineer, Crawler)

Perplexity London, England, United Kingdom

Software Development · 201-500 employees

7 h ago
sre Senior (5-10 yrs) Full-time United Kingdom
Create a free account to apply — email only, no card. You can also save this posting or score it against your profile with AI.

Applying here? Try the free cover letter tool — paste this posting and your résumé, no account needed.

About the role

You will operate and optimize Kubernetes-based crawler infrastructure while building observability and CI/CD pipelines to ensure system reliability. Additionally, you will lead incident response, perform capacity planning, and automate routine remediation tasks.

What they look for

Kubernetes Python C++ Rust Go Terraform Linux Networking TCP/IP DNS TLS HTTP Load Balancing Datadog Prometheus Grafana

Requirements

Candidates must have 5+ years of experience in SRE or production engineering with strong proficiency in Python, C++, Rust, or Go. You are also required to have hands-on experience with cloud infrastructure, Kubernetes, and Linux networking fundamentals.

Full description

Keeping a crawler running is not the same as keeping it productive. Queues can grow while servers look healthy, and a small change can become a fleet-wide bottleneck. On Perplexity's Crawler team, you'll improve throughput, plan capacity, and build operational reliability through safer deployments, better observability, and durable fixes.

Responsibilities

  • Operate and optimize Kubernetes workloads and supporting infrastructure for crawler services, including fetching and rendering fleets.
  • Build and maintain infrastructure as code and CI/CD pipelines, with safe rollouts, validation, and fast rollback.
  • Define SLOs and build monitoring, alerting, and observability for availability, latency, throughput, and resource saturation.
  • Own capacity planning and resource efficiency; diagnose queue backlogs, network failures, and resource bottlenecks with service owners.
  • Participate in the team's on-call rotation, lead incident response, and turn blameless postmortems into durable reliability improvements.
  • Write production software to automate diagnosis and routine remediation, with validation and approval gates for high-impact actions.

Qualifications

  • 5+ years of experience in SRE, production engineering, or infrastructure-heavy backend roles, with strong programming skills in Python, C++, Rust, or Go.
  • Hands-on experience with cloud infrastructure, production Kubernetes, infrastructure as code such as Terraform, and automated deployment workflows.
  • Strong Linux and networking fundamentals, including TCP/IP, DNS, TLS, HTTP, and load balancing.
  • Experience building observability with tools such as Datadog, Prometheus, or Grafana, and using SLOs to guide reliability improvements.
  • Proven experience diagnosing distributed-system failures, leading production incidents, and improving system performance or resource efficiency.

Nice to have

  • Experience operating web crawlers, search infrastructure, or browser fleets at scale.
  • Experience with distributed queues, backpressure, workload isolation, or multi-region recovery.

Similar roles