EndpointEvaluator › API Documentation

POST /api/v1/evaluate

Evaluate LLM output against a reference text. Returns a verdict indicating whether the output is consistent with the reference. See Scoring Details for method comparisons and examples.

Requires: X-API-Key header

curl -X POST https://endpointevaluator.com/api/v1/evaluate \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "output_text": "The order ships tomorrow.",
    "reference_text": "Your order will ship tomorrow morning.",
    "scoring_method": "combined",
    "client_id": "test-case-42",
    "group_id": "ci-run-2026-05-16"
  }'

{
  "evaluation_id": "eval_abc123xyz789",
  "client_id": "test-case-42",
  "group_id": "ci-run-2026-05-16",
  "scoring_method": "combined",
  "verdict": "consistent",
  "method_verdicts": {
    "lexical": "consistent",
    "semantic": "consistent",
    "inferential": "consistent"
  },
  "requested_methods": ["lexical", "semantic", "inferential"],
  "completed_methods": ["lexical", "semantic", "inferential"],
  "missing_methods": [],
  "flags": [],
  "method_errors": {},
  "credits_charged": 100,
  "credits_refunded": 0,
  "credits_consumed": 100,
  "credits_remaining": 400
}

Single-method responses omit method_verdicts, requested_methods, completed_methods, missing_methods, flags, and method_errors; those fields describe cross-method outcomes that only apply to combined.

{
  "evaluation_id": "eval_AbC123dEf456",
  "client_id": "test-case-42",
  "group_id": "ci-run-2026-05-16",
  "scoring_method": "combined",
  "verdict": "consistent",
  "method_verdicts": {...},
  "raw_score": 0.8421,
  "raw_scores": {
    "lexical": 0.91,
    "semantic": 0.84,
    "inferential": 0.86
  },
  "credits_charged": 100,
  "credits_consumed": 100,
  "credits_remaining": 799900
}

Accounts with bonus features unlocked additionally receive raw_score on every method and raw_scores on combined evaluations. See pricing to unlock.

GET /api/v1/balance

Check your current credit balance. Shows free credits, paid credits, and total available. This call is free.

Requires: X-API-Key header

curl -X GET https://endpointevaluator.com/api/v1/balance \
  -H "X-API-Key: YOUR_API_KEY"

{
  "free": 450,
  "paid": 0,
  "total": 450,
  "tier": "free",
  "bonus_unlocked": false
}

GET /api/v1/evaluations/:evaluation_id

Look up the details of a prior evaluation. Useful for auditing or debugging your evaluation history. This call costs one credit.

Requires: X-API-Key header

curl -X GET https://endpointevaluator.com/api/v1/evaluations/eval_abc123xyz789 \
  -H "X-API-Key: YOUR_API_KEY"

{
  "evaluation_id": "eval_abc123xyz789",
  "scoring_method": "combined",
  "evaluated_at": "2026-03-24T18:30:00Z",
  "client_id": "test-case-42",
  "group_id": "ci-run-2026-05-16",
  "credits_consumed": 100,
  "result": {
    "verdict": "consistent",
    "method_verdicts": {
      "lexical": "consistent",
      "semantic": "consistent",
      "inferential": "consistent"
    }
  }
}

The credits_consumed field shows credits used for the original evaluation, not this lookup.

Rate Limits

API requests are rate-limited per account based on your tier.

Any paid balance puts the account on the paid tier, regardless of which credit pack (Small, Medium, or Large) was purchased. Rate limit headers are included in API responses: X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset

Error Responses

Every error response carries a top-level error_type discriminator. Clients should branch on error_type, not on HTTP status alone, for forward compatibility. The closed catalog at v1.0 has 16 values listed below; new values will arrive only in MINOR or MAJOR releases per the changelog. For server-side error logging guidance (severity by error_type), see the Production Checklist .

Idempotency-related errors are emitted only when the idempotency feature is enabled per-account; off by default at v1.0.

All-methods-failed body (422 and 503)

Both 422 and 503 responses share the same body shape; only the error_type field and the HTTP status differ. 503 additionally sets a Retry-After header. The body is the same whether the request was a single-method call or a combined call.

{
  "error_type": "all_methods_failed_transient",
  "error": "Evaluation failed",
  "evaluation_id": "eval_abc123xyz789",
  "scoring_method": "combined",
  "requested_methods": ["lexical", "semantic", "inferential"],
  "completed_methods": [],
  "missing_methods": ["lexical", "semantic", "inferential"],
  "method_errors": {
    "lexical": { "reason": "no_key_tokens", "transient": false },
    "semantic": { "reason": "pool_timeout", "transient": true },
    "inferential": { "reason": "pool_timeout", "transient": true }
  },
  "credits_charged": 0,
  "credits_refunded": 100
}

For 422 the same body is returned with error_type: "all_methods_failed" and every per-method transient flag set to false. The transient flag on each entry of method_errors tells the client whether that specific sub-method's failure is retry-worthy in isolation; the top-level error_type tells the client whether the whole call is.

Partial-combined body (503)

Returned when scoring_method: "combined" and at least one sub-method failed to produce a score. Combined fails closed: rather than return a 200 with a partial combined result (which CI pipelines cannot reliably distinguish from a real combined score), the API returns 503 with a full refund. Retry after the Retry-After header.

{
  "error_type": "partial_combined_unavailable",
  "error": "Partial combined evaluation unavailable",
  "evaluation_id": "eval_abc123xyz789",
  "scoring_method": "combined",
  "requested_methods": ["lexical", "semantic", "inferential"],
  "completed_methods": ["lexical"],
  "missing_methods": ["semantic", "inferential"],
  "method_errors": {
    "semantic": { "reason": "pool_timeout", "transient": true },
    "inferential": { "reason": "nli_unparseable", "transient": false }
  },
  "credits_charged": 0,
  "credits_refunded": 100
}

Per-sub-method transient flags are preserved for clients that want to log the underlying failure shape; the top-level call is always retry-worthy regardless of sub-method classification, so no HTTP-status disambiguation like the 422 / 503 split on all_methods_failed applies here.

FAQ

Common questions about combined scoring and its fails-closed semantics. See Response Fields and Error Responses above for the underlying field and status-code definitions.