First-answer accuracy
Does the initial answer meet the task’s reference answer and evaluation criteria?
AI security & reasoning
Pyrodit Scan Lab
AI security & reasoning evaluation workspace · Discuss your deployment
Evaluate how AI systems respond to attacks, solve tasks and handle feedback—with traceable evidence.
Follow the work
Follow an assessment from execution to review.
Three recorded screens show where the operator checks the run, examines its evidence and returns to the wider project posture.
Recorded product screen
Watch the assessment
The run summary places target and policy beside the live event stream. Review progress, timeouts and candidate hits together; a completed run is not a declaration that the target is safe.
- Follow individual test executions
- See which probe is running, which detector produced a signal and how long the execution took. The event stream connects test activity to candidate findings instead of leaving the operator with a progress bar alone.
- Keep progress and risk separate
- Evaluated requests, candidate hits, timeouts and severity answer different questions. Read them together to distinguish an attack finding from an execution problem, and identify what still needs investigation.
- Move from a signal to its context
- A candidate hit carries its probe, detector, confidence and execution reference into review. The next step is to inspect the evidence and decide whether the issue is confirmed or a false positive.
Example output
- Assessment record
- A run identifier tying target, policy, execution events and candidate findings together.
Is this the right product?
- Built for
- Security and engineering teams evaluating AI applications and models.
- What it helps you do
- Evaluate how AI systems respond to attacks, solve tasks and handle feedback—with traceable evidence.
- Its role in your stack
- The assessment workspace, used before release and during regression testing. Guard applies controls in the live request and action path.
Scope & availability
A clean run is limited to its scope and completed coverage. Incomplete or failed checks must not be counted as passing results. Read reasoning measurements alongside their rubric and answer evidence. A reasoning failure is not a security finding; a correct answer is not proof of safety.
- Starts with
- An authorised target, a test policy and defined attack scenarios or reasoning tasks.
- Produces
- Traceable security assessments and reasoning measurements with their evaluation context.
- Deployment
- Assessment API and workers separated from the target application; evaluation begins in a controlled environment.
Reasoning evaluation
Does the answer hold up under scrutiny?
A fluent answer can still miss the goal or abandon a correct conclusion under pressure. Scan Lab’s reasoning engine evaluates the answer you can observe, the task it must solve and its response to feedback.
Recovery after valid feedback
Among initially wrong answers, does the model reach the correct result after a valid correction?
Resistance to false feedback
Among initially correct answers, does the model preserve the correct result after a misleading challenge?
Criteria, evidence and clear outcomes.
The evaluation records the rubric version, reference answer, relevant answer excerpts and pass, fail or inconclusive outcomes. Supported tasks use either criterion-based model evaluation or deterministic comparison of a structured answer. Missing evidence and execution failures remain visible; they do not become passing scores.
Before changing a banking assistant’s model.
Use a reviewed set of eligibility rules and policy exceptions. Check whether the assistant reaches the right conclusion, recovers when a policy detail is corrected and resists an incorrect challenge. Repeat the same task scope after a model or prompt change; compare eligible counts as well as the rates.
Security findings and reasoning measurements answer different questions and retain separate interpretation. Results apply to the selected tasks and criteria. Define the reasoning assessment scope with our team; the security screens shown below illustrate security assessments.
Discuss a reasoning assessment →A security assessment your team can work with.
Scan Lab connects test preparation, bounded execution and evidence review in one workspace. The useful result is not just a score: it is a finding the team can trace, assess and act on.
01
Define the target and the operating boundary
Start with a project and a connection to the endpoint under test. Select the environment, probe language and applicable execution sources. Launch review checks the exact request against connection readiness and compatible worker capabilities, making missing requirements visible before the run is dispatched.
02
Build a deliberate attack scope
Choose risk families and individual probes, or use a supported framework edition and its requirements to resolve the test scope. Probe-selection review records the exact selected revisions and request budget. Prompt injection, data leakage, unsafe output and tool misuse become explicit test choices where the enabled catalogue provides coverage.
03
Control the cost and intensity of testing
Policies bound request count, duration, request rate and concurrency. Multi-step execution and AI-generated variations have their own policy limits and depend on the selected execution profile. Save scan configurations to reuse the setup; review the effective limits again when the target, policy or scope changes.
04
Separate detection from interpretation
Probe execution, detector verdicts and evaluator results carry distinct roles. Inspect the selected detector revisions and coverage alongside the finding’s severity, score and confidence. Worker readiness is checked separately: a detector being listed in the catalogue does not mean it was available or executed in a particular run.
05
Turn candidate hits into reviewed findings
Filter findings by assessment, severity and triage status. Open the underlying occurrences and execution context, then record a triage decision and its rationale. The decision history stays attached to the finding, giving subsequent reviewers a way to understand what was confirmed, dismissed or reconsidered.
06
Share the assessment with its evidence
Open the report for a specific run and export it as HTML or PDF. Keep scope, coverage, finding instances and their provenance together for engineering review or an internal assessment handoff. The report represents that recorded assessment; framework attribution supplies context, not a certification of compliance.
See the finding. Assess its evidence.
Actual product screens from the presentation · Recorded test data · English interface
From candidate hit to review
Severity, detector, test scenario and masked evidence meet in the same finding. Move into the details without losing the source of the signal during review.
A shareable assessment summary
The report connects severity distribution and execution evidence status to the same assessment. Scores and counters on this screen belong to the sample run.
From a candidate hit to a defensible next step.
Consider an assistant that exposes information during an adversarial test. Scan Lab helps the reviewer connect the observed behaviour to a specific test and its evidence before deciding what to change.
Inspect
Read the probe, detector signal and available execution trace. Check masking and coverage so the conclusion reflects the evidence actually retained.
Decide
Confirm the finding or record why it is a false positive. Keep the rationale with the triage history so engineering and security teams share the same context.
Reassess
After changing the application, run the relevant scenarios again and review the new evidence. A completed scan or the absence of a hit alone is not proof that the issue is resolved.
Architecture & control
An assessment pipeline you can inspect.
Pyrodit Scan Lab brings security assessment and practical reasoning evaluation into one product. Define attack scenarios or reasoning tasks, inspect the recorded answers and keep security findings separate from answer-quality measurements.
Target to run
Check that the target and test workers are ready. Set request volume, rate, duration and concurrency limits, and choose how sensitive test data is masked.
Traceable detection
Combine rules, entity recognition and model-based analysis where available. Track which detector version produced a signal, what it covered and the evidence behind it.
Finding to report
Review findings alongside the recorded test conversation and execution trace. Share the scope, severity, evidence and recommended next steps in an assessment report.
Use cases
Enterprise challenges. Coordinated decisions.
Keep goods moving, production running and banking operations accountable. Explore workflows across business units, systems and approval boundaries.
Coordinate corporate onboarding through one review file.
Bring relationship managers, customer due diligence and operations into a shared case. Track legal entities, authorised representatives, missing evidence and review decisions together.
Explore the workflow →In practice
What happens in day-to-day use?
Explore the working steps, operational controls and evaluation boundaries behind the product.
How do we know the assessment can actually run?
Worker readiness distinguishes available, degraded and unavailable capabilities. The capability view records healthy and stale workers, missing requirements and compatible execution workers. Review this readiness alongside the scan scope before interpreting results: a missing detector or unavailable worker is an execution limitation, not evidence that a target is safe.
How are detector changes reviewed?
Qualification records attach to a detector revision rather than an unversioned name. Authorised users can inspect reports and attestations, plan requalification, issue qualifications and revoke them. Evaluation, publication and revocation have separate permissions, supporting a review process when detectors, model assets or their execution environment change.
What can the team take into a remediation review?
A run report provides a common reference for the assessment, with report retrieval and HTML or PDF download surfaces. Findings retain the risk and evidence context needed to discuss remediation. Agree on the target configuration, detector revisions, coverage and retest scope so a later run can be compared meaningfully with the original assessment.
Works alongside
Give each product a clear responsibility.
Combine the capabilities your process needs. Define how products share context, permissions and results across your workflow.
Pyrodit Guard
Use confirmed risks to shape a runtime protection policy. A scan finding informs the design; it does not automatically enable Guard enforcement.
Pyrodit Agentic Runtime
Assess an agentic application after changes to its models, prompts or tool paths, with an authorised test target and explicit coverage.
Build with Mindtro