Independent Third-Party AI Agent Evaluation
Regulatory Signals is an independent technical evaluation provider. This product is not an accredited certification under ISO/IEC 17065 or ISO/IEC 17021. Evaluation findings are technical assessments, not regulatory endorsements or legal opinions.
Not a vendor self-attestation. An independent third-party evaluation with a signed, tamper-evident evaluation report and public registry listing.
Enterprise procurement teams, institutional investors, and regulated-industry buyers require third-party evaluation because vendors cannot audit themselves. The same reason you need a Big Four auditor for your financials.
$1,990 one-time evaluation · $790 quarterly retest · 5-suite, 250-test assessment · 90-day cert validity
What the Agent Evaluation covers
Safety & Adversarial Resistance
The problem
A vendor cannot credibly grade its own agent's safety.
What we find
A safety-alignment suite and an adversarial jailbreak suite run independently against your live agent, each with its own signed run ID.
Capability & Tool Use
The problem
Agents that look capable in a demo often misuse the tools they are given.
What we find
A capability benchmark and a tool-use correctness suite that check whether the agent finishes real tasks and calls its tools the right way.
Procurement Evidence
The problem
Enterprise buyers ask for third-party proof before they sign.
What we find
A refusal-calibration suite, a signed tamper-evident report with per-suite scores, a public registry listing, and a quarterly retest so the evidence stays current.
Why buyers cannot accept a vendor's own eval
- ✗ Conflict of interest — vendor controls methodology and results
- ✗ No replay verification — scores cannot be independently confirmed
- ✗ No tamper detection — report can be modified post-generation
- ✗ Fails enterprise procurement requirements for third-party audit
- ✓ Independent — no commercial relationship with model providers
- ✓ Deterministic replay — any party can verify scores match the cert
- ✓ Tamper-evident — HMAC-SHA256 hash checked on every public view
- ✓ Public registry — permanent, citable record for procurement RFPs
Frameworks this evaluation addresses
Evaluation produces documented technical evidence relevant to the following frameworks.
Mention of regulatory frameworks does not constitute legal advice. Consult qualified legal counsel for your specific requirements.
Responsible Scaling Policy
Third-party capability and safety threshold documentation required for frontier model deployment decisions.
OpenAI Preparedness Framework
Independent frontier risk assessment supporting safe deployment evaluations.
EU AI Act Articles 11 and 13
Technical documentation (Art. 11) and transparency obligations to deployers (Art. 13) for high-risk AI systems require documented technical evidence that a third party can verify.
ISO/IEC 42001 AI Management System
Third-party evaluation evidence for AI management system audit and certification.
5 evaluation suites · 250 test cases total
Each suite runs 50 test cases authored and maintained by Regulatory Signals.
Internally authored tasks covering knowledge, reasoning, coding, and agentic execution. Measures raw task capability across domains.
Datasets: Regulatory Signals internal capability test set
Internal safety test set covering CBRN, weapons, cyberattack, and fraud categories. Measures alignment quality and refusal on clearly harmful prompts.
Datasets: Regulatory Signals internal safety test set
Adversarial jailbreak attempts from our internal red-team corpus. Measures robustness against prompt injection and manipulation.
Datasets: Regulatory Signals internal red-team corpus
Structured tool-calling scenarios. Measures API call format accuracy, parameter correctness, and error handling.
Datasets: Regulatory Signals internal tool-use eval (mocked tool server)
Borderline prompts testing calibration. Measures appropriate refusal without over-refusing benign requests.
Datasets: Regulatory Signals internal refusal-calibration set
Pricing
Simple, transparent pricing. No hidden fees.
- 5-suite, 250-test evaluation
- Signed, tamper-evident certificate
- Public registry listing
- Replay verification endpoints
- 90-day certificate validity
- Badge embed code
- Full methodology report (PDF)
- Re-runs all 5 suites
- Certificate renewed for 90 days
- Registry listing updated
- Score delta comparison report
- Covers model version and prompt changes
- Required for continuous registry status
Trust architecture
Every run signed with HMAC-SHA256. Any party can verify scores match.
Hash recomputed on every public view. Mismatch triggers automatic REVOKED status.
Permanent public record at regulatorysignals.com/registry?type=agent.
We have no commercial relationship with model providers. Evaluation is conflict-free.
Add the badge to your README or website
After evaluation, embed a verifiable trust badge that links directly to your public registry listing.
<!-- Agent Eval Pass Badge -->
<a href="https://www.regulatorysignals.com/agent-eval-registry/{your-agent-slug}">
<img
src="https://www.regulatorysignals.com/badges/agent-eval-pass.svg"
alt="Agent Eval PASS — Regulatory Signals"
width="200"
height="28"
/>
</a>Frequently asked questions
What is evaluated in the AI Agent Evaluator-of-Record assessment?
The evaluation covers five suites: (1) Capability Benchmarking — 50 internally authored tasks covering knowledge, reasoning, coding, and agentic task execution, measuring raw capability. (2) Safety Alignment — 50 prompts from our internal safety test set covering CBRN, weapons, cyberattack, fraud, and other harm categories, measuring safety behavior on clearly harmful requests. (3) Adversarial Jailbreak Resistance — 50 adversarial jailbreak attempts measuring robustness to prompt attacks. (4) Tool-Use Correctness — 50 structured tool-calling scenarios measuring API call accuracy and error handling. (5) Refusal Calibration — 50 borderline prompts measuring whether the agent refuses appropriately without over-refusing benign requests.
Why can't we just use our own internal eval?
Enterprise procurement teams, institutional investors, and regulated-industry customers increasingly require third-party evaluation because vendor self-assessments create a conflict of interest. The same reason you need a Big Four auditor for your financials — you cannot audit yourself. Anthropic's RSP explicitly distinguishes between developer-run evals and independent third-party assessments. EU AI Act Articles 11 (technical documentation) and 13 (transparency and information to deployers) require documented technical evidence that a third party can verify. Article 50 governs transparency obligations on providers of generative AI systems toward end users (e.g., deepfake disclosure) — a distinct requirement.
How long does the evaluation take?
The five suites run automatically against your agent endpoint, so how long they take depends on your agent's response time. When the run finishes you receive the scores, a signed certificate and a public registry listing.
What does the signed certificate include?
The certificate includes: agent name, version, endpoint domain (truncated for privacy), evaluation date, 90-day expiry, overall score (0–100), per-suite scores, pass/conditional/fail tier, a tamper-evident HMAC-SHA256 hash of the evaluation payload, and a public registry URL. The cert hash is recomputed on every public view — any tampering triggers an automatic REVOKED status.
How does deterministic replay verification work?
Each evaluation run records a random nonce, and a run record (suite, agent endpoint and version, nonce, timestamp, and test-case IDs) is HMAC-SHA256 signed and stored with the run. The account that ran the evaluation can call /api/agent-eval/replay/{runId} to retrieve that record, see a server-side signature check, and re-run the same test cases to confirm scores match within a 0.5% tolerance.
Which regulatory frameworks does this address?
The evaluation is designed to produce documented evidence relevant to: Anthropic's Responsible Scaling Policy (RSP) — third-party capability and safety thresholds; OpenAI Preparedness Framework — frontier risk assessment documentation; EU AI Act Articles 11 and 13 — technical documentation and transparency obligations for high-risk AI systems; ISO 42001 — AI management system audit evidence. Mention of these frameworks does not constitute legal advice. Buyers should consult qualified legal counsel for their specific regulatory requirements.
What is the quarterly retest, and why is it required?
AI agents change rapidly — new model versions, updated system prompts, and fine-tuning can alter safety and capability profiles significantly. Certificates expire after 90 days. The $790 quarterly retest re-runs all five suites against the current production version and renews the certificate if scores meet thresholds. This ensures your registry listing always reflects the current production agent, not a point-in-time snapshot.
Enterprise buyers are demanding independent evals now
Procurement teams at banks, healthcare orgs, and government agencies are adding "third-party AI agent evaluation" to their vendor RFPs. Get evaluated before your deal hits that requirement.
Questions? [email protected]
Where to go next
EU AI Act risk checks, reports, audit pack, and policies — one program.
Free to startStart Free CheckAI-generated codebase audit — Claude, Codex, Gemini, Cursor, Copilot, Lovable.
Free to startRun Free ScanSearchable registry of audited MCP servers, extensions, and AI agents.
Free to browseBrowse Registry