evaluate#

langsmith.evaluation._runner.evaluate(target: TARGET_T | Runnable | EXPERIMENT_T | Tuple[EXPERIMENT_T, EXPERIMENT_T], /, data: DATA_T | None = None, evaluators: Sequence[EVALUATOR_T] | Sequence[COMPARATIVE_EVALUATOR_T] | None = None, summary_evaluators: Sequence[SUMMARY_EVALUATOR_T] | None = None, metadata: dict | None = None, experiment_prefix: str | None = None, description: str | None = None, max_concurrency: int | None = 0, num_repetitions: int = 1, client: langsmith.Client | None = None, blocking: bool = True, experiment: EXPERIMENT_T | None = None, upload_results: bool = True, **kwargs: Any) → ExperimentResults | ComparativeExperimentResults[source]#

在给定数据集上评估目标系统。

参数:

  • target (TARGET_T | Runnable | EXPERIMENT_T | Tuple[EXPERIMENT_T, EXPERIMENT_T]) – 要评估的目标系统或实验。可以是接受字典并返回字典的函数，langchain Runnable，现有实验 ID 或实验 ID 的二元组。

  • data (DATA_T) – 要评估的数据集。可以是数据集名称、示例列表或示例生成器。

  • evaluators (Sequence[EVALUATOR_T] | Sequence[COMPARATIVE_EVALUATOR_T] | None) – 要在每个示例上运行的评估器列表。评估器签名取决于目标类型。默认为 None。

  • summary_evaluators (Sequence[SUMMARY_EVALUATOR_T] | None) – 要在整个数据集上运行的摘要评估器列表。如果比较两个现有实验，则不应指定。默认为 None。

  • metadata (dict | None) – 要附加到实验的元数据。默认为 None。

  • experiment_prefix (str | None) – 为实验名称提供的前缀。默认为 None。

  • description (str | None) – 实验的自由文本描述。

  • max_concurrency (int | None) – 要运行的最大并发评估数。如果为 None，则不设置限制。如果为 0，则不并发。默认为 0。

  • client (langsmith.Client | None) – 要使用的 LangSmith 客户端。默认为 None。

  • blocking (bool) – 是否阻塞直到评估完成。默认为 True。

  • num_repetitions (int) – 运行评估的次数。数据集中的每个项目将运行和评估这么多次。默认为 1。

  • experiment (schemas.TracerSession | None) – 要扩展的现有实验。如果提供，则忽略 experiment_prefix。仅供高级用法。如果 target 是现有实验或实验的二元组，则不应指定。

  • load_nested (bool) – 是否加载实验的所有子运行。默认值是仅加载顶级根运行。仅当 target 是现有实验或实验的二元组时才应指定。

  • randomize_order (bool) – 是否随机化每次评估的输出顺序。默认值为 False。仅当 target 是现有实验的二元组时才应指定。

  • upload_results (bool)

  • kwargs (Any)

返回:

如果 target 是函数、Runnable 或现有实验。ComparativeExperimentResults：如果 target 是现有实验的二元组。

返回类型:

ExperimentResults

示例

准备数据集

>>> from typing import Sequence
>>> from langsmith import Client
>>> from langsmith.evaluation import evaluate
>>> from langsmith.schemas import Example, Run
>>> client = Client()
>>> dataset = client.clone_public_dataset(
...     "https://smith.langchain.com/public/419dcab2-1d66-4b94-8901-0357ead390df/d"
... )
>>> dataset_name = "Evaluate Examples"

基本用法

>>> def accuracy(run: Run, example: Example):
...     # Row-level evaluator for accuracy.
...     pred = run.outputs["output"]
...     expected = example.outputs["answer"]
...     return {"score": expected.lower() == pred.lower()}
>>> def precision(runs: Sequence[Run], examples: Sequence[Example]):
...     # Experiment-level evaluator for precision.
...     # TP / (TP + FP)
...     predictions = [run.outputs["output"].lower() for run in runs]
...     expected = [example.outputs["answer"].lower() for example in examples]
...     # yes and no are the only possible answers
...     tp = sum([p == e for p, e in zip(predictions, expected) if p == "yes"])
...     fp = sum([p == "yes" and e == "no" for p, e in zip(predictions, expected)])
...     return {"score": tp / (tp + fp)}
>>> def predict(inputs: dict) -> dict:
...     # This can be any function or just an API call to your app.
...     return {"output": "Yes"}
>>> results = evaluate(
...     predict,
...     data=dataset_name,
...     evaluators=[accuracy],
...     summary_evaluators=[precision],
...     experiment_prefix="My Experiment",
...     description="Evaluating the accuracy of a simple prediction model.",
...     metadata={
...         "my-prompt-version": "abcd-1234",
...     },
... )
View the evaluation results for experiment:...

仅在示例子集上评估

>>> experiment_name = results.experiment_name
>>> examples = client.list_examples(dataset_name=dataset_name, limit=5)
>>> results = evaluate(
...     predict,
...     data=examples,
...     evaluators=[accuracy],
...     summary_evaluators=[precision],
...     experiment_prefix="My Experiment",
...     description="Just testing a subset synchronously.",
... )
View the evaluation results for experiment:...

流式传输每个预测以更轻松+更积极地调试。

>>> results = evaluate(
...     predict,
...     data=dataset_name,
...     evaluators=[accuracy],
...     summary_evaluators=[precision],
...     description="I don't even have to block!",
...     blocking=False,
... )
View the evaluation results for experiment:...
>>> for i, result in enumerate(results):
...     pass

将 evaluate API 与现成的 LangChain 评估器一起使用

>>> from langsmith.evaluation import LangChainStringEvaluator
>>> from langchain_openai import ChatOpenAI
>>> def prepare_criteria_data(run: Run, example: Example):
...     return {
...         "prediction": run.outputs["output"],
...         "reference": example.outputs["answer"],
...         "input": str(example.inputs),
...     }
>>> results = evaluate(
...     predict,
...     data=dataset_name,
...     evaluators=[
...         accuracy,
...         LangChainStringEvaluator("embedding_distance"),
...         LangChainStringEvaluator(
...             "labeled_criteria",
...             config={
...                 "criteria": {
...                     "usefulness": "The prediction is useful if it is correct"
...                     " and/or asks a useful followup question."
...                 },
...                 "llm": ChatOpenAI(model="gpt-4o"),
...             },
...             prepare_data=prepare_criteria_data,
...         ),
...     ],
...     description="Evaluating with off-the-shelf LangChain evaluators.",
...     summary_evaluators=[precision],
... )
View the evaluation results for experiment:...

评估 LangChain 对象

>>> from langchain_core.runnables import chain as as_runnable
>>> @as_runnable
... def nested_predict(inputs):
...     return {"output": "Yes"}
>>> @as_runnable
... def lc_predict(inputs):
...     return nested_predict.invoke(inputs)
>>> results = evaluate(
...     lc_predict.invoke,
...     data=dataset_name,
...     evaluators=[accuracy],
...     description="This time we're evaluating a LangChain object.",
...     summary_evaluators=[precision],
... )
View the evaluation results for experiment:...

在版本 0.2.0 中更改: ‘max_concurrency’ 默认值从 None（对并发没有限制）更新为 0（完全没有并发）。

在本页中