Score agent turns with a remote scorer

A remote scorer signal scores each completed agent turn with an HTTP endpoint that you host, instead of with an LLM judge. When a turn ends, the W&B Weave agent scoring worker sends the turn to your endpoint in an HTTP POST and records the response as feedback on the turn. The result appears as a tag or rating on the Signals tab of the Agents view. This page covers remote scorers for agent turns. To score Calls traced with @weave.op, see Score Calls with remote scorers. You configure remote scorers with the Python SDK or the Weave UI. The TypeScript SDK doesn’t include RemoteScorer.

How agent turn scoring works

An agent turn is scored in the following sequence:

A turn ends. When a root span (a span with no parent) ends, Weave treats it as a completed turn and emits a weave.genai.turn_ended event.
The agent scoring worker loads the project’s active signals that target weave.genai.turn_ended, then applies each signal’s filters and sample rate.
For each RemoteScorer on a matching signal, the worker builds a schema_version: 2 request from the turn’s span, including its messages, resolves the scorer’s credentials, checks the endpoint URL against the allowed hosts, and sends the POST.
The worker validates the response and writes the result as feedback on the turn. Tags and ratings appear on the Signals tab.

Weave scores only completed turns. Individual LLM and tool spans, and whole conversations, aren’t remote scoring targets. A remote scorer signal is a Monitor whose op_names is ["weave.genai.turn_ended"], whether you create it in the UI or with the SDK. The worker retries a failed attempt with the same Idempotency-Key, up to three attempts within 30 seconds of the first attempt. A 5xx, 408, or 429 response is retried. A timeout uses the whole 30 seconds, so a timed-out request isn’t retried. Any other 4xx response isn’t retried. If your endpoint can’t score a turn in time, return 503 quickly rather than letting the request time out, so that Weave retries it. Agent turn scoring requires the structured result format, because Weave stores the tags and ratings as typed feedback columns.

Enable remote scoring

Remote scoring is off until it’s enabled for your organization or deployment, and the scoring worker calls a scorer endpoint only if its host is on an allowlist. How you enable it depends on your deployment type. Multi-tenant Cloud To enable remote scorers for an organization, an organization or billing admin must:

Open https://wandb.ai/account-settings/[ORG]/settings, replacing [ORG] with the organization that owns your project.
Select the Remote scoring tab.
Turn on Enable remote scoring.
Under Allowed hosts, click Add host and enter each host that remote scorers may call. Saving with remote scoring enabled requires at least one host. Leave the port blank to allow any port on that host.
Click Save settings.

Dedicated Cloud Ask W&B to enable remote scoring for your deployment and configure its allowed hosts. Self-Managed If you run W&B Weave in a W&B Self-Managed deployment, set these environment variables through extraEnv on each scoring worker: the online evaluation worker, the call scoring worker, and the agent scoring worker.

Environment variable	Default	Effect
`WF_SCORING_WORKER_REMOTE_SCORING_ENABLED`	`false`	Turns on outbound requests to remote scorers. When `false`, no remote scorer runs, regardless of other settings.
`WF_SCORING_WORKER_REMOTE_HTTP_TIMEOUT_SECONDS`	`30`	Timeout for each request to a scorer endpoint.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_HOSTS`	empty	Comma-separated operator allowlist of `host` or `host:port` entries.
`WF_SCORING_WORKER_REMOTE_SCORER_VALIDATE_HOSTS`	`true`	Enforces the allowed hosts lists. Private, loopback, and cloud metadata addresses are rejected regardless of this setting.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP`	`false`	Permits `http://` endpoint URLs.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS`	empty	Comma-separated CIDR networks, such as `10.0.0.0/8`, whose private addresses remote scorers may call.

To show the remote scoring settings and scorer options in the Weave UI, also set GORILLA_GATE_WEAVE_REMOTE_SCORING=true on the W&B server. Allowed host rules The scoring worker checks every scorer endpoint URL, and separately the OAuth token endpoint URL when a scorer uses OAuth, against these rules:

An entry matches an exact host, with an optional port. An entry without a port allows any port on that host.
An entry that starts with *. matches subdomains at any depth, but not the domain itself. *.corp.example.com matches a.corp.example.com and a.b.corp.example.com, not corp.example.com. The suffix after *. must contain at least two labels, so *.com is rejected. A wildcard can’t be combined with an IP address.
When both an operator allowlist and an organization allowlist exist, the URL must satisfy both. An empty operator allowlist adds no restriction. When no allowlist exists at all, the worker rejects every host.
Loopback, private, internal, and cloud metadata addresses are rejected. On Self-Managed, private addresses in the networks listed in WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS are allowed.
HTTPS is required unless the deployment permits insecure HTTP.
Redirects aren’t followed.

Build the scorer endpoint

Your endpoint accepts a JSON POST from Weave and returns a JSON score. For a reference implementation, see Sample code.

Request

Weave sends one HTTP POST per scored target to the scorer’s endpoint URL, with these headers:

Header	Value
`Content-Type`	`application/json`
`Authorization`	`Bearer [TOKEN]`
`Idempotency-Key`	A key derived from the scored target, the monitor version, and the scorer version.
`X-Correlation-ID`	A correlation ID for this request.
`X-Weave-Schema-Version`	`1` or `2`, equal to `schema_version` in the body.

Weave might deliver the same scoring attempt more than once. Use Idempotency-Key to deduplicate if needed for your endpoint. The key is stable for one request version, so a V1 and a V2 request for the same Call carry different keys. Every request body has these top-level fields:

Field	Description
`schema_version`	Integer, `1` or `2`.
`scoring_call_id`, `scoring_trace_id`	Identifiers for this scoring attempt.
`monitor`	`name` and `version_digest` of the monitor that selected the target.
`scorer`	`name`, `ref`, and optional `config`. `config` is the mapping set on the `RemoteScorer`, passed through unchanged.
`triggered_at`	Optional ISO 8601 timestamp.

Weave omits an optional field with no value rather than sending it as null. Weave might add optional fields to a version without changing its number, so ignore fields you don’t recognize. Request and response bodies are limited to 1 MiB each and contain JSON text only, never images, audio, or video. A target that exceeds these limits isn’t sent, so it isn’t scored. Each request carries one target. An agent turn request carries two version numbers. The top-level schema_version is the envelope version, which is 2 for agent turns. The scored data is under scoring_target, a tagged union with three fields:

type: The kind of target. agent_turn for a turn. The contract also defines call, which agent turn scoring never sends.
schema_version: The payload version for that type. It counts independently of the envelope version. The agent_turn payload is at payload version 1.
payload: The data for that type.

Check the envelope version first, then check scoring_target.type together with scoring_target.schema_version to choose how to score the payload. Return 4xx for a combination your endpoint doesn’t handle, such as call if you score only agent turns. The agent_turn payload at payload version 1 has these fields:

Field	Description
`event_type`	`weave.genai.turn_ended`.
`project_id`	The project, as `[YOUR-TEAM]/[YOUR-PROJECT]`.
`trace_id`, `span_id`	Identifiers of the turn’s root span.
`span_name`	The span name, for example `invoke_agent sample-support-agent`.
`operation_name`	The GenAI operation name, for example `invoke_agent`. `null` if the trace didn’t record it.
`started_at`, `ended_at`	ISO 8601 timestamps.
`conversation`	`id` and `name`. Each is `null` if the trace didn’t record it.
`agent`	`name`, `version`, and `description`. Each is `null` if the trace didn’t record it.
`status`	`code`, `message`, and `error_type`. `code` is `UNSET`, `OK`, or `ERROR`. `message` and `error_type` are `null` if the trace didn’t record them.
`messages`	`system_instructions`, `input`, and `output`. Each is a list, `[]` when the turn has no messages of that kind.

A turn that ends without an explicit status arrives with status.code set to UNSET. Treat UNSET as a normal completed turn and ERROR as the failure signal. Each message in input and output has role, content, and finish_reason. content is plain text, or a JSON-encoded array of parts when the message carries structured content such as tool calls.

{
  "schema_version": 2,
  "scoring_target": {
    "type": "agent_turn",
    "schema_version": 1,
    "payload": {
      "event_type": "weave.genai.turn_ended",
      "project_id": "[YOUR-TEAM]/[YOUR-PROJECT]",
      "trace_id": "0af7651916cd43dd8448eb211c80319c",
      "span_id": "b7ad6b7169203331",
      "span_name": "invoke_agent sample-support-agent",
      "operation_name": "invoke_agent",
      "started_at": "2026-06-08T12:00:00+00:00",
      "ended_at": "2026-06-08T12:00:01+00:00",
      "conversation": {
        "id": "conversation-0001",
        "name": "Support session"
      },
      "agent": {
        "name": "sample-support-agent",
        "version": "2026-06-08",
        "description": "Answers support questions"
      },
      "status": {
        "code": "UNSET",
        "message": null,
        "error_type": null
      },
      "messages": {
        "system_instructions": [
          "Answer the user accurately and concisely."
        ],
        "input": [
          {
            "role": "user",
            "content": "What are your support hours?",
            "finish_reason": ""
          }
        ],
        "output": [
          {
            "role": "assistant",
            "content": "Our support team is available Monday through Friday, 9am to 5pm Eastern.",
            "finish_reason": "stop"
          }
        ]
      }
    }
  },
  "scoring_call_id": "018f8d6c-8d5f-7000-8000-000000000005",
  "scoring_trace_id": "018f8d6c-8d5f-7000-8000-000000000006",
  "monitor": {
    "name": "example_remote_scorer_agent_monitor",
    "version_digest": "monitor-version-digest"
  },
  "scorer": {
    "name": "example_remote_scorer",
    "ref": "weave:///[YOUR-TEAM]/[YOUR-PROJECT]/object/example_remote_scorer:scorer-version-digest",
    "config": {
      "example_threshold": 0.8
    }
  },
  "triggered_at": "2026-06-08T12:00:02+00:00"
}

Weave adds optional fields to a payload without changing its payload version. Weave removes, renames, or changes the meaning of a field only with a new payload version for that type, or a new envelope version for changes to the envelope. A new target type joins the V2 union at payload version 1.

Response

Return HTTP 200 with a JSON object that has two fields:

schema_version: Integer equal to the request’s schema_version.
result: One score object, a list of score objects, or an object of the form {"scores": [...]}.

A score object has these fields:

Field	Required	Description
`value`	Yes	A tag, as a string of at most 36 characters, or a rating, as a number from `0.0` to `1.0`.
`reason`	No	A string that explains the score.
`confidence`	No	A number from `0.0` to `1.0`.

Weave treats any non-200 response as a scorer failure and records no feedback for that attempt. Weave doesn’t follow redirects and treats them as failures. Weave doesn’t parse the body of an error response. Return 4xx for requests your endpoint never accepts and 5xx for temporary problems. For an agent turn request, the response’s schema_version is 2. The following example returns one score object:

{
  "schema_version": 2,
  "result": {
    "value": "concise",
    "reason": "72 characters.",
    "confidence": 0.9
  }
}

Weave normalizes the tags and reasons in the result before storing them. See How Weave normalizes scores.

Authenticate requests from Weave

Weave authenticates to your endpoint with a bearer token. The request carries no W&B credential. The token proves to your endpoint that the request came from Weave, not the reverse. Each RemoteScorer uses one of two modes:

Mode	What Weave does	What you configure
`oauth_client_credentials`	Requests a token from your OAuth token endpoint with the client credentials grant, sending the client ID and secret with HTTP Basic authentication (`client_secret_basic`), then sends that token to your scorer endpoint.	The token endpoint URL, the client ID, the name of the secret that holds the client secret, and an optional scope.
`static_bearer`	Sends a fixed token to your scorer endpoint.	The name of the secret that holds the token.

Store the client secret or bearer token in the secret store of the team that owns the project before you register the scorer. The RemoteScorer configuration holds only the secret name. The scoring worker resolves the value at scoring time.

Create a remote scorer signal

Create the signal in the Weave UI or with the Python SDK. Both create a RemoteScorer attached to a monitor that targets weave.genai.turn_ended.

Weave UI

Create the signal from the Agents view:

In the Weave project sidebar, click Agents.
In the tab bar, click Signals.
Click New signal, then click Remote scorer.
In the Remote scorer drawer, Scored by is set to Remote scorer. Configure the following fields:
- Scorer name: The name shown in the Scorer column of the Signals table. Up to 128 characters.
- Scoring endpoint URL: The URL Weave sends the POST to.
- Authentication: Static bearer or OAuth client credentials. For static bearer, select or enter the Bearer token secret name. For OAuth, enter the Token endpoint URL, Client ID, Client secret name, and optional Scope. Secret fields take team secret names, not values.
- Config (JSON, optional): A JSON object passed to your endpoint as scorer.config.
- Only score turns matching (Optional): Expand Advanced, then add filters to restrict which turns the signal scores, for example by agent name, agent version, operation name, tool name, or status code. To score every turn, leave it empty. Weave combines multiple filters with AND logic.
- Sample rate (Optional): Expand Advanced, then set the fraction of matching turns the signal scores.
Click Create signal.

The remote scorer form has no tag or rating fields. Your endpoint determines what it returns, and the Signals table shows the tags and ratings it receives. In the Scorer column, a remote scorer signal shows a webhook icon.

Python SDK

Publish a RemoteScorer, then activate a Monitor that lists it in scorers and targets weave.genai.turn_ended in op_names.

import weave
from weave.flow.monitor import Monitor
from weave.scorers.remote_scorer import RemoteScorer, StaticBearerAuthConfig

weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

scorer = RemoteScorer(
    name="policy_remote_scorer",
    endpoint_url="https://scoring.example.com/weave/score",
    config={"threshold": 0.9},  # Sent to your endpoint as scorer.config
    auth_config=StaticBearerAuthConfig(
        mode="static_bearer",
        bearer_secret_name="WEAVE_REMOTE_SCORER_BEARER_TOKEN",
    ),
)
weave.publish(scorer, name="policy_remote_scorer")

monitor = Monitor(
    name="policy_remote_signal",
    scorers=[scorer],
    op_names=["weave.genai.turn_ended"],  # Score completed agent turns
    sampling_rate=1.0,
)
monitor.activate()

For OAuth client credentials, pass an OAuthClientCredentialsConfig as auth_config instead.

Sample code

The examples/remote_scorer directory in the weave repository is the reference implementation of this contract and the source of truth for the sample code. One endpoint in that sample accepts V1 Call requests, V2 Call requests, and V2 agent turn requests. For agent turns, these files apply:

remote_scorer_app.py: A FastAPI app with GET /health and POST /score.
auth.py: A development-only bearer token check against the REMOTE_SCORER_DEV_BEARER_TOKEN environment variable.
scoring_logic.py: Unwraps either envelope with extract_scoring_target, then scores the turn’s last output message.
sample_request_v2_agent_turn.json: A complete V2 agent turn request.
register_remote_scorer.py --agent-turn: Publishes a RemoteScorer and activates a monitor for completed agent turns.
trigger_test_agent_turn.py: Logs one turn with weave.conversation.log_turn.

To run the endpoint locally and send it a V2 agent turn request without Weave, start the app, then send the sample request:

python3 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
export REMOTE_SCORER_DEV_BEARER_TOKEN="dev-token"
uvicorn remote_scorer_app:app --host 127.0.0.1 --port 8000

curl -sS http://127.0.0.1:8000/score \
  -H "Authorization: Bearer dev-token" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: local-contract-check" \
  -H "X-Correlation-ID: local-contract-check" \
  -H "X-Weave-Schema-Version: 2" \
  --data @sample_request_v2_agent_turn.json

The sample requires Weave 0.53.0 or later. A local run verifies only the contract.

Test the signal

Before you test, deploy the endpoint at an HTTPS URL that’s on the allowed hosts, and register it. Log one completed turn, then check the Signals tab. Scoring is asynchronous, so the result appears after a delay.

import uuid

from opentelemetry import trace

import weave
from weave.conversation import Message, log_turn

weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

result = log_turn(
    conversation_id=f"sample-{uuid.uuid4().hex}",
    agent_name="sample-support-agent",
    system_instructions=["Answer the user accurately and concisely."],
    messages=[Message.user("What are your support hours?")],
    output_messages=[
        Message.assistant(
            "Our support team is available Monday through Friday, 9am to 5pm Eastern."
        )
    ],
)

# weave.init exports spans from a background thread. Flush before a short script exits.
trace.get_tracer_provider().force_flush(30_000)
print(result.conversation_id, result.trace_ids)

Your endpoint receives a V2 request with scoring_target.type set to agent_turn, and Weave records the result as feedback on that turn.

Troubleshooting

Symptom	What to check
Remote scorer doesn’t appear in the New signal drawer	Remote scoring is enabled in the owning organization’s Remote scoring settings, or by your deployment administrator.
The endpoint URL is rejected	The scorer host, and the token endpoint host for OAuth, are in the allowed hosts. Check for a port mismatch, a non-HTTPS URL, or a private or internal address.
The endpoint returns `401` or `403`	The secret name, the OAuth client credentials, audience, and scope, and your endpoint’s token validation.
The endpoint returns `400` for agent turns	The endpoint checks the envelope version and accepts `scoring_target.type` `agent_turn` at payload version `1`.
No result appears on the Signals tab	Scoring is asynchronous. Confirm the turn matched the signal’s filters and sample rate, and that the turn’s root span ended.
The turn matched the signal but no tag or rating appears	Weave records nothing for a failed scoring attempt. Check your endpoint’s request logs for the `X-Correlation-ID` header. If the endpoint received no request, the URL failed the allowed host rules or the secret didn’t resolve. If it received a request, the response failed validation.
No result after a `200` response	The response’s `schema_version` is `2`, and `result` uses one of the three structured shapes.

Get Started

Guides

Cookbooks

Reference

Details & Support

How agent turn scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Create a remote scorer signal

Weave UI

Python SDK

Sample code

Test the signal

Troubleshooting

​How agent turn scoring works

​Enable remote scoring

​Build the scorer endpoint

​Request

​Response

​Authenticate requests from Weave

​Create a remote scorer signal

​Weave UI

​Python SDK

​Sample code

​Test the signal

​Troubleshooting

How agent turn scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Create a remote scorer signal

Weave UI

Python SDK

Sample code

Test the signal

Troubleshooting