Vendor/Model Evaluation Scorecard
Purpose
Structured evaluation rubric for assessing AI vendors and models across security, privacy, performance, and risk dimensions.
Related Controls
1. Evaluation Overview
Capture basic information about the vendor and the evaluation context.
Vendor/Model Name: [VENDOR NAME]
Evaluation Date: [DATE]
Evaluator: [NAME], [ROLE TITLE]
Use Case: [DESCRIPTION OF INTENDED USE]
Data Classification: Public / Internal / Confidential / Restricted
Risk Tier: Low / Medium / High
Decision: Approved / Approved with Conditions / Rejected
Next Review Date: [DATE]
2. Evaluation Criteria
Score each criterion 1-5 (1=Poor, 5=Excellent). Minimum passing score is 3.0 average with no category below 2.
| Category | Criterion | Score (1-5) | Evidence/Notes |
|---|---|---|---|
| Security | Data encryption (transit and rest) | ||
| Access controls and authentication | |||
| Vulnerability management and patching | |||
| Privacy | Data processing agreement (DPA) available | ||
| Data retention and deletion policies | |||
| GDPR/CCPA compliance documentation | |||
| Performance | Accuracy/quality benchmarks provided | ||
| Latency and throughput SLAs | |||
| Uptime guarantees and incident history | |||
| Transparency | Model card or documentation available | ||
| Training data provenance disclosed | |||
| Bias testing and fairness reporting | |||
| Operational | API stability and versioning policy | ||
| Support responsiveness and SLA | |||
| Exit strategy and data portability |
3. Scoring Summary
Aggregate scores and determine the final recommendation.
| Category | Average Score | Pass/Fail |
|---|---|---|
| Security | ||
| Privacy | ||
| Performance | ||
| Transparency | ||
| Operational | ||
| Overall Average |
Pass Criteria
- Overall average must be 3.0 or higher
- No individual category average below 2.0
- Security category must be 3.0 or higher for any system processing Internal or higher data
- Privacy category must be 3.0 or higher for any system processing PII
4. Conditions & Risk Acceptance
Document any conditions that must be met before deployment or risks that are being accepted.
Conditions for Approval
- [CONDITION — e.g., "Vendor must execute DPA before data processing begins"]
- [CONDITION — e.g., "API keys must be stored in secrets management, not application code"]
- [CONDITION — e.g., "Output filtering must be implemented before customer-facing deployment"]
Accepted Risks
- [RISK — e.g., "Model may produce inaccurate outputs; mitigated by mandatory human review"]
- [RISK — e.g., "Vendor does not provide training data provenance; mitigated by output monitoring"]
Approvals
- Security Review: [NAME] — [DATE] — Approved / Rejected
- Privacy Review: [NAME] — [DATE] — Approved / Rejected
- Business Sponsor: [NAME] — [DATE] — Approved / Rejected