Endpoint Evaluator

Catch LLM drift before your users.

LLMs are powerful but unpredictable.
A model update, a prompt change, a provider swap, and suddenly
your system is giving customers wrong answers with complete confidence.

Use our API to get visibility into this risk.

Sign Up for Free Evaluations Returning User? Log In Here

Know when your AI goes off-script

Detect drift

Score LLM outputs against reference responses. Make drops in quality more noticeable.

Zero supply chain risk

Pure API. No SDK. No packages to install. Nothing touches your codebase.

Safeguard your data

Zero evaluation text retention. Your data stays private.

Prepay for credits, no subscription

500 free credits daily. Scale when you're ready.

How Does It Work?

Our API evaluates the consistency of your data.

You provide a reference text — the response you expect — and an output text — the response your system actually produced. We measure the consistency between them and return a score.

Try It Now — Run a Free Evaluation

If you have an AI chatbot that tells your customers about your hotel,
you might want to make sure it's providing the correct information.

Reference Text — the response you expect 61 / 1,000

Output Text — the response your system actually produced 41 / 1,000

Our system will evaluate whether your system Output Text is consistent with
your Reference Text. Complete the captcha below to Evaluate the example above.

Sign Up for Free Evaluations Returning User? Log In Here

What Can You Evaluate?

If your task produces text that should match a reference, this works.

Review our Frequently Asked Questions (FAQ) page for additional information.

Chatbot Quality Assurance

Your chatbot says it's quoting your docs. Is it? Send the chatbot response and your source document — get a consistency verdict in milliseconds.

Model Comparison

Switching from GPT-4 to Claude to Llama? Run the same prompts, score each output against your reference. Pick the model that's actually most faithful, not just most fluent.

Content Review Automation

Your team uses LLMs to draft reports, summaries, or translations. Add an automated accuracy check before anything goes out the door.

RAG Pipeline Verification

Your retrieval-augmented generation looks right. But "looks right" isn't a test. Verify that generated answers actually match retrieved context.

It's Simple: Buy Credits, Use Credits

No subscriptions. No recurring charges. No new supply chain risks.

	Free	Small	Medium	Large
Price	$0	$10	$30	$100
Credits	500/day	50,000	200,000	800,000
Lexical (20 credits each)	—	$0.0040	$0.0030	$0.0025
Semantic (40 credits each)	—	$0.0080	$0.0060	$0.0050
Inferential (80 credits each)	—	$0.0160	$0.0120	$0.0100
Combined (100 credits each)	—	$0.0200	$0.0150	$0.0125
Bonus	—	—	—	Bonus Features →
		Buy	Buy	Buy

Sign Up for Free Evaluations Returning User? Log In Here

Three Ways to Measure, Plus All-in-One

There are different ways to measure consistency. Each offers a balance of quality, cost, and speed.

Deep dive with examples →

Lexical: Words

Consistent

Most of the reference wording appears in the output

Reference: "Refund issued within 5 business days."

Output: "Your refund will be issued within 5 business days."

Inconsistent

Little of the reference wording appears in the output

Reference: "Refund issued within 5 business days."

Output: "We've credited your account."

20 credits <200ms

Semantic: Meaning

Consistent

Output uses similar vocabulary and phrasing

Reference: "Refund issued within 5 business days."

Output: "We've credited your account."

Inconsistent

Meaning doesn't match at first glance

Reference: "The store opens at 9am."

Output: "The store opens at 9pm."

40 credits <500ms

Inferential: Logic

Consistent

Logically entailed from the reference

Reference: "All orders ship within 24 hours."

Output: "Your order ships within a day."

Inconsistent

Contradictory meaning to the reference

Reference: "The order ships tomorrow."

Output: "The order won't ship tomorrow."

80 credits <2s

Combined

All Methods

Evaluate all three methods in one API call.

Reference: "All orders ship within 24 hours."

Output: "Your order ships within a day."

Simple verdicts for CI pipelines. Unanimous agreement required for overall verdict. If the methods disagree, the verdict is ambiguous.

Lexical: consistent

Semantic: consistent

Inferential: consistent

Verdict: consistent

100 credits <3s

See It in Action

A hotel chatbot tells guests breakfast is included. After a model update, it starts saying breakfast is not included. Lexical and semantic scoring see mostly overlapping words and similar meaning — the surface looks fine. Inferential catches the logical contradiction.

Read our Quick Start Guide for more details and CI/CD Integration examples.

Request

curl -X POST https://endpointevaluator.com/api/v1/evaluate \
  -H "X-API-Key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "reference_text": "Breakfast is included with your stay",
    "output_text": "Breakfast is not included with your stay",
    "scoring_method": "combined"
  }'

Response

{
  "evaluation_id": "eval_abc123xyz789",
  "scoring_method": "combined",
  "verdict": "ambiguous",
  "method_verdicts": {
    "lexical": "consistent",
    "semantic": "consistent",
    "inferential": "inconsistent"
  },
  "flags": [
    "nli_contradicted"
  ],
  "credits_consumed": 100,
  "credits_remaining": 400
}

Sign Up for Free Evaluations Returning User? Log In Here

Bonus Features for Large Customers

Purchase a Large credit pack and permanently unlock these features on your account.

Raw Scores

Get the raw numerical score (0.0–1.0) alongside the categorical verdict for every evaluation. Build custom thresholds, track trends, and fine-tune your quality gates.

Batch Evaluation

Submit up to 10 evaluations in a single API call. Ideal for CI/CD pipelines running test suites — evaluate multiple outputs against references in one request.

Extended History

Access your evaluation results (reference and output text are not retained) for 30 days instead of 7. More time for trend analysis, auditing, and debugging regressions across model updates.