Jev AI Examples

/ 01

Same Question, Three Real Inputs

Recorded live evaluations on October 1, 2026, through OpenRouter. We kept the same Choice question and option descriptions, then changed only the fictional state. Three calls are a demonstration, not an accuracy benchmark.

The ambiguous input split browser 53% and search_docs 47%. Stating that internal pricing was stale changed this run to browser 96%. Known arithmetic inputs selected calculator; Jev chose a tool, it did not calculate the answer. A previous September 30 run of the ambiguous input selected search_docs 51% instead: repeat calls can change a close winner.

Model: typesafe/jev-1.13-20260917. Provider: TypeSafe. Each record includes the UTC timestamp, complete request and response, token usage, and client-measured duration including network. These observations do not establish a production threshold.

Ambiguous source freshness

{
  "key": "tool",
  "label": "Ambiguous source freshness",
  "state": "A user asks: \"What is the current input-token price for Jev? Cite the official documentation.\" Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
  "request": {
    "state": "A user asks: \"What is the current input-token price for Jev? Cite the official documentation.\" Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
    "questions": {
      "tool": {
        "type": "choice",
        "instructions": "Which tool should the agent use first?",
        "criteria": {
          "search_docs": "Search the internal knowledge base",
          "browser": "Open current public documentation",
          "calculator": "Perform arithmetic on known values"
        }
      }
    },
    "model": "typesafe/jev-1.13"
  },
  "response": {
    "model": "typesafe/jev-1.13-20260917",
    "provider": "TypeSafe",
    "answers": {
      "tool": {
        "type": "choice",
        "choice": "browser",
        "probabilities": {
          "calculator": 0,
          "search_docs": 0.47,
          "browser": 0.53
        },
        "confidence": 0.29
      }
    },
    "usage": {
      "input_tokens": 375,
      "output_tokens": 39,
      "cost": 0.00001575
    }
  },
  "recordedAt": "2026-10-01T08:51:16.011Z",
  "durationMs": 3374
}

Explicit fresh public source

{
  "key": "tool-current",
  "label": "Explicit fresh public source",
  "state": "Find the current Jev input-token price and cite the official public documentation. The internal knowledge base is six months old and contains no current pricing. Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
  "request": {
    "state": "Find the current Jev input-token price and cite the official public documentation. The internal knowledge base is six months old and contains no current pricing. Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
    "questions": {
      "tool": {
        "type": "choice",
        "instructions": "Which tool should the agent use first?",
        "criteria": {
          "search_docs": "Search the internal knowledge base",
          "browser": "Open current public documentation",
          "calculator": "Perform arithmetic on known values"
        }
      }
    },
    "model": "typesafe/jev-1.13"
  },
  "response": {
    "model": "typesafe/jev-1.13-20260917",
    "provider": "TypeSafe",
    "answers": {
      "tool": {
        "type": "choice",
        "choice": "browser",
        "probabilities": {
          "search_docs": 0.04,
          "browser": 0.96,
          "calculator": 0
        },
        "confidence": 0.94
      }
    },
    "usage": {
      "input_tokens": 383,
      "output_tokens": 39,
      "cost": 0.000016086
    }
  },
  "recordedAt": "2026-10-01T08:51:16.558Z",
  "durationMs": 546
}

Known arithmetic inputs

{
  "key": "tool-math",
  "label": "Known arithmetic inputs",
  "state": "Calculate the total for 12 items priced at $4 each. All numbers are supplied; no facts need to be looked up. Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
  "request": {
    "state": "Calculate the total for 12 items priced at $4 each. All numbers are supplied; no facts need to be looked up. Available tools: search_docs (internal knowledge base), browser (current public docs), calculator (arithmetic).",
    "questions": {
      "tool": {
        "type": "choice",
        "instructions": "Which tool should the agent use first?",
        "criteria": {
          "search_docs": "Search the internal knowledge base",
          "browser": "Open current public documentation",
          "calculator": "Perform arithmetic on known values"
        }
      }
    },
    "model": "typesafe/jev-1.13"
  },
  "response": {
    "model": "typesafe/jev-1.13-20260917",
    "provider": "TypeSafe",
    "answers": {
      "tool": {
        "type": "choice",
        "choice": "calculator",
        "probabilities": {
          "calculator": 1,
          "browser": 0,
          "search_docs": 0
        },
        "confidence": 1
      }
    },
    "usage": {
      "input_tokens": 381,
      "output_tokens": 39,
      "cost": 0.000016002
    }
  },
  "recordedAt": "2026-10-01T08:51:17.386Z",
  "durationMs": 828
}

/ 02

Agent Tool Selection

State: a user needs the current Jev API price from official public documentation and asks for a citation. Available tools are search_docs (internal knowledge base), browser (current public docs) and calculator (arithmetic). The illustrative response selects the browser. A real model can weigh the sources differently; compare the dated experiment above. The values below are preset to illustrate the response shape.

{ "type": "choice", "choice": "browser", "probabilities": { "search_docs": 0.13, "browser": 0.82, "calculator": 0.05 }, "confidence": 0.77 }

/ 03

Agent Action Risk Scoring

State: an agent plans to send a customer’s conversation history, email and account ID to an unverified third-party endpoint without recorded consent or an approved data flow. A Score rates the action across low, moderate and high risk. These preset values illustrate a high-risk result; they are not from a live evaluation.

{ "type": "score", "score": 1.71, "legend": { "0": "Low — approved destination and minimal data", "1": "Moderate — some uncertainty or limited exposure", "2": "High — sensitive data sent to an unverified destination" }, "probabilities": { "0": 0.04, "1": 0.21, "2": 0.75 }, "confidence": 0.89 }

/ 04

Support Ticket Priority

State: a production outage returns HTTP 500 errors to every workspace member and blocks all deployments. A Choice question assigns a response priority from p0 (critical broad outage), p1 (major impairment) or p2 (routine). The sample selects p0 to show how a ticket triage result can be represented.

{ "type": "choice", "choice": "p0", "probabilities": { "p0": 0.91, "p1": 0.08, "p2": 0.01 }, "confidence": 0.94 }

/ 05

Run Your Own

Prefill a request in the Jev playground and inspect its generated JSON. Read the field reference for exact parsing rules.

/ FAQ

Frequently asked questions