I tested 34 grade-A x402 requests. Eight answers weren't useful.

A payment can succeed. An API can return valid JSON. The answer can still be wrong. This is what happened when I paid for responses from some of the best-graded endpoints in the x402 ecosystem.

Gideon Ibemere Jr. · Run executed September 15, 2026 · Report written September 16, 2026

Requests attempted
34

18 endpoints · 11 providers

Responses graded
31

Three requests never settled

Not useful
8

25.8% of graded responses

What happened across the 34 valid-run attempts

  • 23Useful
  • 8Not useful
  • 3Unsettled / ungraded

The question behind the experiment

Most endpoint assessments start outside the response: is the service reachable? Does it return a compliant 402? Is the advertised price stable? Can the payment settle?

Those questions matter. But none establishes whether the buyer received an answer that was useful for the task.

I built x402-spend to record what an agent actually paid for, then paired it with x402-spend-reviews to attach an outcome to an on-chain-verifiable payment.

Run 001 asked whether being reachable, compliant and highly graded was enough to predict useful delivery. My prediction, documented before paying, was that some responses would still fail.

What I tested

On September 14, I froze a sample of 34 requests, 18 distinct endpoints and 11 providers, all graded A by x402 Trust at selection time. I chose questions whose answers I could check independently: forward and reverse geocoding, known-answer and relevance search, BTC/USD pricing, and weather.

I committed the questions and expected answers before sending payments. The pre-registered method dates back to September 6, and the full technical report documents its amendments.

Of the 34 requests in the valid run, 31 settled and returned responses. The other three were calls to the same search endpoint, which returned 502 without taking payment. Those three are ungraded, not counted as failures of response quality.

Of the 31 responses graded against the prepared expectations:

  • 23 were useful.
  • Eight were not useful (25.8% of graded responses).
  • Six were marked wrong; two were insufficient.
  • The valid run spent $0.308, including $0.085 on not-useful responses.

The striking result was the type of failure. Every not-useful response was a semantic failure. None was empty or malformed. Payment succeeded; the response had the expected structure; the content didn't accomplish the task.

The endpoint that answered a question nobody asked

One result makes the difference tangible.

The grade-A endpoint agent402.tools/api/geocode charged $0.001 per lookup on Avalanche. I sent it two different Austin venue queries. Both responses were byte-identical and described the White House in Washington, D.C.

The response contained real coordinates, a plausible source identifier and valid attribution. Its own query field said "1600 Pennsylvania Ave NW, Washington, DC" — the example query in the provider's documentation, not the input I'd sent.

What I observed: two different requests produced the same, unrelated response, and the returned query didn't match either input.

What I infer: that particular route appeared to ignore the caller's input and return a canned example. I cannot claim from two requests that it behaves this way for every possible query.

There was also a useful control: three other routes on the same provider answered correctly. This wasn't evidence that the whole provider was unusable. It was evidence that a route can pass external checks while failing the buyer's actual task.

An agent could pass that response downstream without noticing: everything inside the wrong answer looked coherent.

Not all incorrect results look obviously broken

The remaining failures were less spectacular but still consequential.

Some geocoding results returned the wrong address or location while keeping a valid JSON shape. Two search responses surfaced plausible material without enough evidence to answer the precise question asked.

One search result contained the correct person's name, but attached it to the wrong year. Merely mentioning the answer wasn't enough: would a buyer who didn't already know the answer have learned it from that response?

That was the useful/not-useful test. A well-formed response that requires the buyer to already know the answer isn't doing much work for the buyer.

Where the experiment itself went wrong

The report includes an entire first attempt that I excluded from the quality analysis.

I had committed the questions, but the harness initially sent bare endpoint URLs without the query parameters. Its results could not fairly measure whether the providers answered properly formed questions. All 34 first-attempt rows were voided as quality evidence, including eight settled calls costing $0.041.

I fixed the request construction, added parameter validation and response-body capture, then ran the prepared sample again. The first attempt remains in the report because quietly omitting it would make the experimental history harder to audit.

There is another accounting boundary: 31 calls were graded, but only 30 have distinct transaction hashes suitable for per-call verification. This is a publication eligibility count, not proof that all 30 have been submitted to or accepted by the registry. One provider used prepaid credit, allowing a legitimate paid call without a fresh on-chain transfer. The first call transferred 10,000 atomic units; 5,000 was consumed immediately and the remaining 5,000 by the next call. The first call's chain-verifiable review must therefore name the full transfer and explain the credit allocation. Inventing a second transaction would be false provenance, so the second call remains in the research findings but outside the per-transaction registry dataset.

What the result does — and doesn't — say

A field report, not an ecosystem-wide failure rate

25.8% is eight of 31 graded responses in this specific sample. It is not an estimate that 25.8% of all x402 endpoints or requests fail. These endpoints were selected because they were highly graded, and the tasks were selected because the answers could be checked.

There are important limitations:

  • One reviewer made every usefulness judgment; there was no independent second rater.
  • The number of calls per task and provider was small, particularly for weather.
  • The forward-geocoding test allowed a 100-metre tolerance, while reverse geocoding required an exact house number. That asymmetry makes some near-boundary failures uncertain.
  • The experiment measured a deliberately checkable slice of API work, not the most complex or ambiguous agent tasks.

I also noted price differences, freshness differences and apparent shared upstream data in separate exploratory observations. Those were not pre-registered tests, did not change outcome labels and are hypotheses for future work, not established comparative rankings.

Why I think this matters

The point isn't that external monitoring is useless; it's that liveness, payment correctness and delivered utility measure different things.

For an agent choosing between two paid endpoints, the cheapest successful call may not be the cheapest useful result. The buyer needs a way to record what happened after payment, compare outcomes for similar tasks and check the evidence behind those judgments.

This run is one small example of why that missing signal matters. It isn't a finished marketplace, a provider blacklist or a claim that on-chain proof can establish whether a reviewer is honest. A transaction proves payment. Usefulness still requires an assessment of the response.

That's the boundary I'm exploring with x402-spend.

Read the evidence