Score Calls with remote scorers

A remote scorer is a scorer that runs on your infrastructure instead of inside W&B Weave. When a monitor selects a Call, the Weave scoring worker sends the Call to your HTTPS endpoint in an HTTP POST request and records the response as feedback on that Call. Use a remote scorer when your scoring logic can’t run in Weave, such as a policy check against internal data or a model you host yourself. This page covers remote scorers for Calls traced with @weave.op. To score agent turns in the Agents view, see Score agent turns with a remote scorer. You configure remote scorers for Calls with the Python SDK. The TypeScript SDK doesn’t include RemoteScorer.

How remote scoring works

A Call is scored in the following sequence:

A Call to a monitored Op ends.
The scoring worker finds the active monitors whose operations include that Op, then applies each monitor’s filter and sampling rate.
For each RemoteScorer on a matching monitor, the worker builds a schema_version: 1 request that contains the Call, resolves the scorer’s credentials, checks the endpoint URL against the allowed hosts, and then sends the POST request.
The worker validates the response and writes the result as feedback on the Call. It also records the scoring attempt as a Call, whether or not the attempt succeeded.

A remote scorer runs only through a monitor. You can’t use it in weave.Evaluation or with call.apply_scorer(). Both call the scorer’s score() method, which for RemoteScorer raises NotImplementedError, because only the scoring worker sends the request. The Score calls action in the Weave UI also rejects a RemoteScorer, with the message RemoteScorer requires a monitor. Each selected Call produces one request, sent once. If the request times out or gets no response, Weave doesn’t retry it. The default timeout is 30 seconds.

Enable remote scoring

Remote scoring is off until it’s enabled for your organization or deployment, and the scoring worker calls a scorer endpoint only if its host is on an allowlist. How you enable it depends on your deployment type. Multi-tenant Cloud To enable remote scorers for an organization, an organization or billing admin must:

Open https://wandb.ai/account-settings/[ORG]/settings, replacing [ORG] with the organization that owns your project.
Select the Remote scoring tab.
Turn on Enable remote scoring.
Under Allowed hosts, click Add host and enter each host that remote scorers may call. Saving with remote scoring enabled requires at least one host. Leave the port blank to allow any port on that host.
Click Save settings.

Dedicated Cloud Ask W&B to enable remote scoring for your deployment and configure its allowed hosts. Self-Managed If you run W&B Weave in a W&B Self-Managed deployment, set these environment variables through extraEnv on each scoring worker: the online evaluation worker, the call scoring worker, and the agent scoring worker.

Environment variable	Default	Effect
`WF_SCORING_WORKER_REMOTE_SCORING_ENABLED`	`false`	Turns on outbound requests to remote scorers. When `false`, no remote scorer runs, regardless of other settings.
`WF_SCORING_WORKER_REMOTE_HTTP_TIMEOUT_SECONDS`	`30`	Timeout for each request to a scorer endpoint.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_HOSTS`	empty	Comma-separated operator allowlist of `host` or `host:port` entries.
`WF_SCORING_WORKER_REMOTE_SCORER_VALIDATE_HOSTS`	`true`	Enforces the allowed hosts lists. Private, loopback, and cloud metadata addresses are rejected regardless of this setting.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOW_INSECURE_HTTP`	`false`	Permits `http://` endpoint URLs.
`WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS`	empty	Comma-separated CIDR networks, such as `10.0.0.0/8`, whose private addresses remote scorers may call.
`WF_SCORING_WORKER_REMOTE_SCORER_REQUIRE_STRUCTURED_RESULT_SCHEMA`	`true`	Rejects responses whose `result` isn’t in the structured score format.

To show the remote scoring settings and scorer options in the Weave UI, also set GORILLA_GATE_WEAVE_REMOTE_SCORING=true on the W&B server. Allowed host rules The scoring worker checks every scorer endpoint URL, and separately the OAuth token endpoint URL when a scorer uses OAuth, against these rules:

An entry matches an exact host, with an optional port. An entry without a port allows any port on that host.
An entry that starts with *. matches subdomains at any depth, but not the domain itself. *.corp.example.com matches a.corp.example.com and a.b.corp.example.com, not corp.example.com. The suffix after *. must contain at least two labels, so *.com is rejected. A wildcard can’t be combined with an IP address.
When both an operator allowlist and an organization allowlist exist, the URL must satisfy both. An empty operator allowlist adds no restriction. When no allowlist exists at all, the worker rejects every host.
Loopback, private, internal, and cloud metadata addresses are rejected. On Self-Managed, private addresses in the networks listed in WF_SCORING_WORKER_REMOTE_SCORER_ALLOWED_PRIVATE_CIDRS are allowed.
HTTPS is required unless the deployment permits insecure HTTP.
Redirects aren’t followed.

Build the scorer endpoint

Your endpoint accepts a JSON POST from Weave and returns a JSON score. For a reference implementation, see Sample code.

Request

Weave sends one HTTP POST per scored target to the scorer’s endpoint URL, with these headers:

Header	Value
`Content-Type`	`application/json`
`Authorization`	`Bearer [TOKEN]`
`Idempotency-Key`	A key derived from the scored target, the monitor version, and the scorer version.
`X-Correlation-ID`	A correlation ID for this request.
`X-Weave-Schema-Version`	`1` or `2`, equal to `schema_version` in the body.

Weave might deliver the same scoring attempt more than once. Use Idempotency-Key to deduplicate if needed for your endpoint. The key is stable for one request version, so a V1 and a V2 request for the same Call carry different keys. Every request body has these top-level fields:

Field	Description
`schema_version`	Integer, `1` or `2`.
`scoring_call_id`, `scoring_trace_id`	Identifiers for this scoring attempt.
`monitor`	`name` and `version_digest` of the monitor that selected the target.
`scorer`	`name`, `ref`, and optional `config`. `config` is the mapping set on the `RemoteScorer`, passed through unchanged.
`triggered_at`	Optional ISO 8601 timestamp.

Weave omits an optional field with no value rather than sending it as null. Weave might add optional fields to a version without changing its number, so ignore fields you don’t recognize. Request and response bodies are limited to 1 MiB each and contain JSON text only, never images, audio, or video. A target that exceeds these limits isn’t sent, so it isn’t scored. Each request carries one target. For a Call, schema_version is 1 and the scored Call is at the top level under original_call:

Field	Description
`project_id`	The project, as `[YOUR-TEAM]/[YOUR-PROJECT]`.
`call_id`, `trace_id`	Identifiers of the scored Call.
`op_name`	The Op ref of the scored Call.
`inputs`	The Call’s inputs.
`output`	The Call’s output.
`started_at`, `ended_at`	ISO 8601 timestamps.

{
  "schema_version": 1,
  "original_call": {
    "project_id": "[YOUR-TEAM]/[YOUR-PROJECT]",
    "call_id": "018f8d6c-8d5f-7000-8000-000000000001",
    "trace_id": "018f8d6c-8d5f-7000-8000-000000000002",
    "op_name": "weave:///[YOUR-TEAM]/[YOUR-PROJECT]/op/sample_remote_scorer_target:*",
    "inputs": {
      "message": "test message for scoring"
    },
    "started_at": "2026-06-08T12:00:00+00:00",
    "ended_at": "2026-06-08T12:00:01+00:00",
    "output": {
      "reply": "received: test message for scoring",
      "status": "ok"
    }
  },
  "scoring_call_id": "018f8d6c-8d5f-7000-8000-000000000003",
  "scoring_trace_id": "018f8d6c-8d5f-7000-8000-000000000004",
  "monitor": {
    "name": "example_remote_scorer_monitor",
    "version_digest": "monitor-version-digest"
  },
  "scorer": {
    "name": "example_remote_scorer",
    "ref": "weave:///[YOUR-TEAM]/[YOUR-PROJECT]/object/example_remote_scorer:scorer-version-digest",
    "config": {
      "example_threshold": 0.8
    }
  },
  "triggered_at": "2026-06-08T12:00:02+00:00"
}

Response

Return HTTP 200 with a JSON object that has two fields:

schema_version: Integer equal to the request’s schema_version.
result: One score object, a list of score objects, or an object of the form {"scores": [...]}.

A score object has these fields:

Field	Required	Description
`value`	Yes	A tag, as a string of at most 36 characters, or a rating, as a number from `0.0` to `1.0`.
`reason`	No	A string that explains the score.
`confidence`	No	A number from `0.0` to `1.0`.

Weave treats any non-200 response as a scorer failure and records no feedback for that attempt. Weave doesn’t follow redirects and treats them as failures. Weave doesn’t parse the body of an error response. Return 4xx for requests your endpoint never accepts and 5xx for temporary problems. For a Call request, the response’s schema_version is 1. This response returns one rating and one tag:

{
  "schema_version": 1,
  "result": [
    {
      "value": 1.0,
      "reason": "Message is 32 characters; concise messages score best.",
      "confidence": 1.0
    },
    {
      "value": "concise",
      "reason": "Message length category is concise.",
      "confidence": 0.9
    }
  ]
}

Authenticate requests from Weave

Weave authenticates to your endpoint with a bearer token. The request carries no W&B credential. The token proves to your endpoint that the request came from Weave, not the reverse. Each RemoteScorer uses one of two modes:

Mode	What Weave does	What you configure
`oauth_client_credentials`	Requests a token from your OAuth token endpoint with the client credentials grant, sending the client ID and secret with HTTP Basic authentication (`client_secret_basic`), then sends that token to your scorer endpoint.	The token endpoint URL, the client ID, the name of the secret that holds the client secret, and an optional scope.
`static_bearer`	Sends a fixed token to your scorer endpoint.	The name of the secret that holds the token.

Store the client secret or bearer token in the secret store of the team that owns the project before you register the scorer. The RemoteScorer configuration holds only the secret name. The scoring worker resolves the value at scoring time.

Register a remote scorer

A remote scorer is a RemoteScorer object attached to a monitor. Create it with the Python SDK. Publish a RemoteScorer, then activate a Monitor that lists it in scorers and names the Ops to score in op_names. endpoint_url is required. config and auth_config are optional.

import weave
from weave.flow.monitor import Monitor
from weave.scorers.remote_scorer import (
    OAuthClientCredentialsConfig,
    RemoteScorer,
    StaticBearerAuthConfig,
)

weave.init("[YOUR-TEAM]/[YOUR-PROJECT]")

# Static bearer token, read from the team secret named WEAVE_REMOTE_SCORER_BEARER_TOKEN
auth = StaticBearerAuthConfig(
    mode="static_bearer",
    bearer_secret_name="WEAVE_REMOTE_SCORER_BEARER_TOKEN",
)

# OAuth client credentials instead:
# auth = OAuthClientCredentialsConfig(
#     mode="oauth_client_credentials",
#     token_endpoint_url="https://idp.example.com/oauth2/token",
#     client_id="weave-remote-scorer",
#     client_secret_name="WEAVE_REMOTE_SCORER_CLIENT_SECRET",
#     scope="score:remote",
# )

scorer = RemoteScorer(
    name="policy_remote_scorer",
    endpoint_url="https://scoring.example.com/weave/score",
    config={"threshold": 0.9},  # Sent to your endpoint as scorer.config
    auth_config=auth,
)
weave.publish(scorer, name="policy_remote_scorer")

monitor = Monitor(
    name="policy_remote_monitor",
    scorers=[scorer],
    op_names=["generate_response"],  # Op names in this project, or full weave:/// Op refs
    sampling_rate=1.0,
)
monitor.activate()

When you run the code, monitor.activate() publishes the monitor as active and expands bare Op names into full Op refs for the current project.

Sample code

The examples/remote_scorer directory in the weave repository is the reference implementation of the request and response format described on this page. It’s written in Python with FastAPI, but your endpoint can use any language, framework, or host. For Calls, these files apply:

remote_scorer_app.py: A FastAPI app with GET /health and POST /score.
scoring_logic.py: Framework-independent request parsing and scoring, written to be copied into your own service. For a Call, it scores the inputs.message value.
auth.py: A development-only bearer token check against the REMOTE_SCORER_DEV_BEARER_TOKEN environment variable.
register_remote_scorer.py --op-name: Publishes a RemoteScorer and activates a Monitor for an Op.
trigger_test_trace.py: Creates a traced Call that the monitor can select.
sample_request.json: A complete V1 request for a Call.

To test the endpoint locally without Weave, start the app, then send it the sample request:

python3 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt
export REMOTE_SCORER_DEV_BEARER_TOKEN="dev-token"
uvicorn remote_scorer_app:app --host 127.0.0.1 --port 8000

curl -sS http://127.0.0.1:8000/score \
  -H "Authorization: Bearer dev-token" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: local-contract-check" \
  -H "X-Correlation-ID: local-contract-check" \
  -H "X-Weave-Schema-Version: 1" \
  --data @sample_request.json

The sample requires Weave 0.53.0 or later. A local run verifies only the request-response behavior you defined for the endpoint. The Weave scoring worker rejects loopback addresses, and hosted deployments don’t permit insecure HTTP.

Test the scorer

Before you test, deploy the endpoint at an HTTPS URL that’s on the allowed hosts, and register it. Trigger a scored Call and confirm the result:

Call the monitored Op at least once.
Confirm that your endpoint received a request. Scoring is asynchronous, so the request arrives after the Call ends.
In the Traces tab, open the Call and check its feedback.

Your endpoint receives a V1 request with the Call under original_call, and Weave records the result as feedback on that Call.

Troubleshooting

Symptom	What to check
Remote scorer options don’t appear in the UI	Remote scoring is enabled in the owning organization’s Remote scoring settings, or by your deployment administrator.
The endpoint URL is rejected	The scorer host, and the token endpoint host for OAuth, are in the allowed hosts. Check for a port mismatch, a non-HTTPS URL, or a private or internal address.
The endpoint returns `401` or `403`	The secret name, the OAuth client credentials, audience, and scope, and your endpoint’s token validation.
The token request succeeds but scoring fails, or the reverse	Each URL separately. The token endpoint and scorer endpoint are validated independently and can use different hosts.
No feedback appears	Scoring is asynchronous. Confirm the Call matched the monitor’s operation, filter, and sampling rate. Weave records every attempt as a scorer Call in the Traces tab. A failed attempt appears as an errored scorer Call that carries the failure reason.
No feedback after a `200` response	The response’s `schema_version` equals the request’s, and `result` uses one of the three structured shapes.

Get Started

Guides

Cookbooks

Reference

Details & Support

How remote scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Register a remote scorer

Sample code

Test the scorer

Troubleshooting

​How remote scoring works

​Enable remote scoring

​Build the scorer endpoint

​Request

​Response

​Authenticate requests from Weave

​Register a remote scorer

​Sample code

​Test the scorer

​Troubleshooting

How remote scoring works

Enable remote scoring

Build the scorer endpoint

Request

Response

Authenticate requests from Weave

Register a remote scorer

Sample code

Test the scorer

Troubleshooting