The LLM Evaluation Framework
evaluation-metrics evaluation-framework llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
Jun 29, 2025 - Python
#
llm-evaluation-framework
Here are 16 public repositories matching this topic...
Agentic LLM Vulnerability Scanner / AI red teaming kit 🧪
agent-framework ai-red-team prompt-testing llm-security llm-vulnerabilities llm-evaluation llm-fuzzing llm-evaluation-framework llm-guardrails llm-scanner llm-jailbreaks llm-fuzzer llm-fuzzer-aggregator agent-security
-
Updated
Jun 20, 2025 - Python
The official evaluation suite and dynamic data release for MixEval.
benchmark evaluation benchmarking-suite evaluation-framework benchmarking-framework foundation-models large-language-models large-language-model llm-inference llm-evaluation large-multimodal-models llm-evaluation-framework benchmark-mixture mixeval
-
Updated
Nov 10, 2024 - Python
LangFair is a Python library for conducting use-case level LLM bias and fairness assessments
python ai artificial-intelligence bias fairness ai-safety fairness-testing bias-detection fairness-ai fairness-ml responsible-ai ethical-ai large-language-models llm llm-evaluation llm-evaluation-framework llm-evaluation-metrics
-
Updated
Jun 18, 2025 - Python
MIT-licensed Framework for LLMs, RAGs, Chatbots testing. Configurable via YAML and integrable into CI pipelines for automated testing.
open-source ci testing-tools chatbot-framework testing-framework chatbot-testing rag ai-chat large-language-models llm ai-testing llm-evaluation llm-evaluation-framework prompt-test llm-testing ai-testing-tool generative-ai-testing rag-testing summarization-testing
-
Updated
Dec 11, 2024 - Python
Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)
metrics good-first-issue llm prompt-engineering generative-ai llmops llm-eval llm-tools llm-evaluation llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework
-
Updated
Feb 13, 2025 - Python
An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.
qa-automation-test rl-training llm exact-matching llm-evaluation llm-evaluation-toolkit llm-evaluation-framework reward-modeling
-
Updated
Jun 21, 2025 - Python
Develop reliable AI apps
-
Updated
Jun 19, 2025 - Python
[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models
machine-learning explainable-ai llm llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics acl2024
-
Updated
Jul 19, 2024 - Python
FM-Leaderboard-er allows you to create leaderboard to find the best LLM/prompt for your own business use case based on your data, task, prompts
-
Updated
Oct 31, 2024 - Python
Realign is a testing and simulation framework for AI applications.
ai simulation evaluation alignment red-teaming rag prompt-engineering llms llmops llm-eval llm-evaluation aiengineering llm-evaluation-framework
-
Updated
Dec 4, 2024 - Python
Multilingual Evaluation Toolkits
-
Updated
Nov 7, 2024 - Python
Shin Rakuda is a comprehensive framework for evaluating and benchmarking Japanese large language models, offering researchers and developers a flexible toolkit for assessing LLM performance across diverse datasets.
-
Updated
Sep 17, 2024 - Python
Hackable, simple, llm evals on preference datasets
-
Updated
Jun 16, 2025 - Python
A set of auxiliary systems designed to provide a measure of estimated confidence for the outputs generated by Large Language Models.
dataset datasets llm llms llm-training llm-evaluation llms-reasoning llm-evaluation-toolkit llms-benchmarking llm-evaluation-framework llm-evaluation-metrics llms-efficency llms-evalution
-
Updated
May 25, 2025 - Python
Tools for systematic large language model evaluations
evaluation-metrics evaluation-framework llm llms llm-evaluation llm-evaluation-toolkit llm-evaluation-framework llm-evaluation-metrics llm-benchmarking
-
Updated
May 23, 2025 - Python
Improve this page
Add a description, image, and links to the llm-evaluation-framework topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the llm-evaluation-framework topic, visit your repo's landing page and select "manage topics."