Inference QA Engineer

Applied Computing was founded in 2024 to build Orbital, a physics-informed foundation model for energy operations. We’re live across oil and gas, refineries, and petrochemicals, working towards our mission: sustainable abundance for a growing planet.

The hydrocarbon industry keeps the world running. But its complexity has left operators tied to legacy systems, making critical decisions on less than 10% of available data. We built Orbital to change that. It’s a foundation model built specifically for energy that lets companies use AI at scale, harnessing all of their operational data and optimising in real time for any metric. Decisions get faster, operations get safer, and carbon intensity falls.

We’ve raised over $32 million, including one of the largest seed rounds for an AI company in the UK. We’re just getting started

About the role

We build Orbital, an agentic LLM platform that answers hard operational and analytical questions for enterprise clients in energy and heavy industry. It routes questions across a graph of tools (data retrieval, SQL-generating coders, statistical analysis, anomaly detection, forecasting, RAG) and synthesises answers that engineers act on. When it is right, it compresses hours of analyst work into seconds.

When it is subtly wrong, quiet, or slow, that costs us trust with technical customers who check our numbers.

We are hiring a senior engineer to own inference quality end to end: the person who watches every deployment, decides what “working” means, builds the systems that prove it, and catches regressions before a client does. This is a founding-level QA function. You will define the discipline, not just execute a checklist.

What you’ll own

Evaluation frameworks. We test on a tiered model: T1 data retrieval, T2 statistical analysis, T3 open-ended inference and root-cause, each with its own pass threshold. You will own and extend this framework, design test sets with real ground-truth rubrics (expected answer, pass criteria, known failure modes, source tables), and set the bar for what ships.

The test harness and its runs. We run prompts against live model endpoints at scale (tens of thousands of runs), on schedules and on demand, measuring consistency across repeated iter-ations, ground-truth match, structural correctness of generated queries, and paraphrase robust-ness. You will run these executions, keep the harness healthy, and turn raw runs into a verdict.

Model and agent monitoring systems. You will build the observability layer for a multi-step agentic system: not just “did the endpoint return 200” but did the planner route to the right tool,did the tool actually execute, did the agent loop terminate, and is the final answer grounded.

Concrete failure modes we already fight and want caught automatically:

• Silent empty responses — the reasoning trace renders but the answer stays empty, while

every layer reports success.

• False refusals — a tool crashes or times out, returns nothing, and the model fabricates “I

don’t have that data” instead of erroring loudly.

• Tool-routing misses — the planner should have fired a tool and didn’t, or double-counts

raw identifiers instead of canonical ones.

• Latency and non-termination — multi-tool agent loops that blow past timeout budgets.

• Paraphrase and run-to-run instability — the same question three ways, or the same

prompt three times, giving materially different answers.

Ground truth and the feedback loop. You will work with subject-matter experts to extract vetted ground truths from deployment feedback, feed them back into the eval sets, and close the loop so every confirmed defect becomes a permanent regression guard.

Regression discipline. After every fix ships, you devise tier-appropriate tests biased at the failure mode plus regression guards, run them against the deployment, and report pass-rate against threshold and whether the original failure recurred. You are the gate.

What we’re looking for

• 5+ years in software, ML, data, or QA engineering, with real ownership of a quality-critical

system.

• Strong Python. Comfortable in a FastAPI + Postgres + Docker world, reading logs across

services and tracing a request through a distributed pipeline.

• Fluency with LLM behaviour: prompting, tool/function calling, agentic loops, RAG, and the

ways they fail (hallucination, refusal, silent truncation, non-determinism).

• Experience designing evaluation: LLM-as-judge, deterministic checks, ground-truth scor-

ing, statistical consistency measures (e.g. coefficient of variation across repeated runs).

• SQL literacy — you can read a generated query and judge whether it answers the question

and hits the right tables.

• A monitoring and observability instinct: you reach for dashboards, alerts, and trace inspec-

tion by default, and you build them when they don’t exist.

• Rigour about uncertainty. You report calibrated ranges, not overclaimed point estimates,

and you say plainly when something is unverified.

Bonus

• Experience evaluating or red-teaming agentic / multi-tool LLM systems specifically.

• MLflow or similar trace and experiment tooling.

• A background talking to technical end users (engineers, analysts) and translating their “it

feels off” into a reproducible test.

• Time-series, forecasting, or industrial and operational data domains.

Why it matters

Our customers are engineers who verify our output. Inference quality is the product. This role decides whether we can look a client in the eye and say the system works, and back it with numbers. You will have the mandate to build that assurance layer from the ground up.