RAG Document Relevance Scoring with Jev

The scenario

A retrieval system can find text about an account without finding the passage that answers a password-reset question. Add an explicit relevance judgment between retrieval and answer generation. This exercise evaluates one query–passage pair at a time so you can see exactly what changes the score.

From input to next step

1. Pair the user’s query with a retrieved passage.
2. Score against ordered criteria: unrelated, partially useful, directly useful.
3. Compare candidate scores and retain the passages that meet your evaluated relevance policy.

Try it. Then change one thing.

Compare the original scenario with a deliberate variation. You can edit either input, run Jev, inspect the result, and share your decision card. Loading an example does not send a model request.

What to look for: The password-reset instructions should land near the highly relevant end of the 0–2 rubric. Decimal scores are possible; this range is an expectation rather than a saved model output.

Experiment: keep the topic, remove the answer

This passage mentions an account but does not explain password reset. Compare its score with the original instructions instead of rewarding topical vocabulary alone.

Use the second scenario tab above to load this variation. Compare the returned decision as well as any probabilities: a changed answer can reveal an unclear policy, and an unchanged answer can reveal that the relevant detail was not captured by your question.

Use this pattern in your application

This playground request shape uses the original scenario. It is a reference for the input, not a public integration endpoint. For application integration, follow the official TypeSafe documentation linked in our Jev API guide. Keep API keys on the server and validate the returned type before taking action.

{
  "playground": "https://aijev.net",
  "language": "JavaScript",
  "request": {
    "model": "typesafe/jev-1.13",
    "state": "Query: How do I reset my password?\nDocument: Open account settings, choose Security, then select Reset password. A confirmation link will be sent to your email.",
    "questions": {
      "decision": {
        "type": "score",
        "instructions": "How relevant is the document to the query?",
        "criteria": [
          "Not relevant: does not address the question",
          "Partially relevant: offers some useful context",
          "Highly relevant: directly answers the question"
        ]
      }
    },
    "provider": {
      "allow_fallbacks": false
    }
  },
  "note": "AIJev Playground request shape only. For application integration, visit https://docs.typesafe.ai."
}

How to evaluate this workflow

Label query–passage pairs with human judgments. Inspect ranking quality, the fraction of answer-bearing passages retained, and downstream answer correctness. Run separate checks for queries whose answer is absent from every retrieved passage.

Save a held-out set before changing your instructions. Compare new versions against the same examples, and review disagreements instead of assuming a valid response is a correct one. For escalation design, see confidence-based routing.

Frequently asked questions

References & further reading

Background on retrieval controls and relevance. The query, passages, and Jev scoring rubric here are original examples. See Amazon Bedrock: Configure retrieval and response generation.

For the underlying decision primitives, read TypeSafe’s System One documentation and our guide to Choice, Score, and Noul.