beta_evaluators

Creates, updates, deletes, gets or lists a beta_evaluators resource.

Overview

Name	beta_evaluators
Type	Resource
Id	azure.ai_projects.beta_evaluators

Fields

The following fields are returned by SELECT queries:

get_credentials

Name	Datatype	Description
blobReference	object	Credential info to access the storage account. Required.

get_version

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
display_name	string	Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.
categories	array	The categories of the evaluator. Required.
created_at	string (date-time)	Creation date/time of the evaluator. Required.
created_by	string	Creator of the evaluator. Required.
definition	object	Definition of the evaluator. Required.
description	string	The asset description text.
evaluator_type	string	The type of the evaluator. Required. Known values are: "builtin" and "custom". (builtin, custom)
generation_artifacts	object	Provenance artifacts from the generation pipeline. Read-only; present only on evaluator versions created via an EvaluatorGenerationJob. Each artifact resolves to a versioned Foundry Dataset.
metadata	object	Metadata about the evaluator.
modified_at	string (date-time)	Last modified date/time of the evaluator. Required.
supported_evaluation_levels	array	Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).
tags	object	Tag dictionary. Tags can be added, removed, and updated.
version	string	The version of the resource. Required.

list_versions

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
display_name	string	Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.
categories	array	The categories of the evaluator. Required.
created_at	string (date-time)	Creation date/time of the evaluator. Required.
created_by	string	Creator of the evaluator. Required.
definition	object	Definition of the evaluator. Required.
description	string	The asset description text.
evaluator_type	string	The type of the evaluator. Required. Known values are: "builtin" and "custom". (builtin, custom)
generation_artifacts	object	Provenance artifacts from the generation pipeline. Read-only; present only on evaluator versions created via an EvaluatorGenerationJob. Each artifact resolves to a versioned Foundry Dataset.
metadata	object	Metadata about the evaluator.
modified_at	string (date-time)	Last modified date/time of the evaluator. Required.
supported_evaluation_levels	array	Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).
tags	object	Tag dictionary. Tags can be added, removed, and updated.
version	string	The version of the resource. Required.

get_generation_job

Name	Datatype	Description
id	string	Server-assigned unique identifier. Required.
created_at	string (date-time)	The timestamp when the job was created, represented in Unix time (seconds since January 1, 1970). Required.
error	object	Error details — populated only on failure.
finished_at	string (date-time)	The timestamp when the job finished, represented in Unix time (seconds since January 1, 1970).
inputs	object	Caller-supplied inputs.
result	object	Evaluator Definition.
status	string	Current lifecycle status. Required. Known values are: "queued", "in_progress", "succeeded", "failed", and "cancelled". (queued, in_progress, succeeded, failed, cancelled)
usage	object	Token consumption summary. Populated when the job reaches a terminal state.

list

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
display_name	string	Display Name for evaluator. It helps to find the evaluator easily in AI Foundry. It does not need to be unique.
categories	array	The categories of the evaluator. Required.
created_at	string (date-time)	Creation date/time of the evaluator. Required.
created_by	string	Creator of the evaluator. Required.
definition	object	Definition of the evaluator. Required.
description	string	The asset description text.
evaluator_type	string	The type of the evaluator. Required. Known values are: "builtin" and "custom". (builtin, custom)
generation_artifacts	object	Provenance artifacts from the generation pipeline. Read-only; present only on evaluator versions created via an EvaluatorGenerationJob. Each artifact resolves to a versioned Foundry Dataset.
metadata	object	Metadata about the evaluator.
modified_at	string (date-time)	Last modified date/time of the evaluator. Required.
supported_evaluation_levels	array	Evaluation levels this evaluator supports (e.g., turn, conversation). When omitted on create, the service defaults to ["turn"]. On update, omitting this field leaves it unchanged; an empty list is rejected. Custom code-based evaluators support only turn; custom prompt-based evaluators support exactly one level (turn or conversation).
tags	object	Tag dictionary. Tags can be added, removed, and updated.
version	string	The version of the resource. Required.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_credentials	select	name, version, endpoint		Get evaluator credentials. Retrieves SAS credentials for accessing the storage account associated with the specified evaluator version.
get_version	select	name, version, endpoint		Get an evaluator version. Retrieves the specified evaluator version, returning 404 if it does not exist.
list_versions	select	name, endpoint	type, limit	List evaluator versions. Returns the available versions for the specified evaluator.
get_generation_job	select	job_id, endpoint		Get an evaluator generation job. Gets the details of an evaluator generation job by its ID.
list	select	endpoint	type, limit	List latest evaluator versions. Lists the latest version of each evaluator.
create_version	exec	name, endpoint, evaluator_type, categories, definition		Create an evaluator version. Creates a new evaluator version with an auto-incremented version identifier.
list_generation_jobs	exec	endpoint	limit, order, after, before	List evaluator generation jobs. Returns a list of evaluator generation jobs. The List API has up to a few seconds of propagation delay, so a recently created job may not appear immediately; use the Get evaluator generation job API with the job ID to retrieve a specific job without delay.
create_generation_job	exec	endpoint	Operation-Id	Create an evaluator generation job. Creates an evaluator generation job. The service generates rubric-based evaluator definitions from the provided source materials asynchronously.
delete_version	exec	name, version, endpoint		Delete an evaluator version. Removes the specified evaluator version. Returns 204 whether the version existed or not.
update_version	exec	name, version, endpoint, evaluator_type, categories, definition		Update an evaluator version. Updates the specified evaluator version in place.
delete_generation_job	exec	job_id, endpoint		Delete an evaluator generation job. Deletes an evaluator generation job by its ID. Deletes the job record only; the generated evaluator (if any) is preserved.
pending_upload	exec	name, version, endpoint, pendingUploadType		Start a pending upload. Initiates a new pending upload or retrieves an existing one for the specified evaluator version.
cancel_generation_job	exec	job_id, endpoint		Cancel an evaluator generation job. Cancels an evaluator generation job by its ID.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
endpoint	string	The service endpoint host (no scheme), e.g. myaccount.table.cosmos.azure.com:443 - value of the client endpoint parameter. (default: )
job_id	string	The ID of the job to cancel. Required.
name	string	Required.
version	string	The specific version id of the EvaluatorVersion to operate on. Required.
Operation-Id	string	Client-generated unique ID for idempotent retries. When absent, the server creates the job unconditionally. Default value is None.
after	string
before	string	A cursor for use in pagination. before is an object ID that defines your place in the list. For instance, if you make a list request and receive 100 objects, ending with obj_foo, your subsequent call can include before=obj_foo in order to fetch the previous page of the list. Default value is None.
limit	integer	A limit on the number of objects to be returned. Limit can range between 1 and 100, and the default is 20. Default value is None.
order	string	Sort order by the created_at timestamp of the objects. asc for ascending order anddesc for descending order. Known values are: "asc" and "desc". Default value is None.
type	string	Filter evaluators by type. Possible values: 'all', 'custom', 'builtin'. Is one of the following types: Literal["builtin"], Literal["custom"], Literal["all"], str Default value is None.

SELECT examples

get_credentials

Get evaluator credentials. Retrieves SAS credentials for accessing the storage account associated with the specified evaluator version.

SELECT
blobReference
FROM azure.ai_projects.beta_evaluators
WHERE name = '{{ name }}' -- required
AND version = '{{ version }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

get_version

Get an evaluator version. Retrieves the specified evaluator version, returning 404 if it does not exist.

SELECT
id,
name,
display_name,
categories,
created_at,
created_by,
definition,
description,
evaluator_type,
generation_artifacts,
metadata,
modified_at,
supported_evaluation_levels,
tags,
version
FROM azure.ai_projects.beta_evaluators
WHERE name = '{{ name }}' -- required
AND version = '{{ version }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list_versions

List evaluator versions. Returns the available versions for the specified evaluator.

SELECT
id,
name,
display_name,
categories,
created_at,
created_by,
definition,
description,
evaluator_type,
generation_artifacts,
metadata,
modified_at,
supported_evaluation_levels,
tags,
version
FROM azure.ai_projects.beta_evaluators
WHERE name = '{{ name }}' -- required
AND endpoint = '{{ endpoint }}' -- required
AND type = '{{ type }}'
AND limit = '{{ limit }}'
;

get_generation_job

Get an evaluator generation job. Gets the details of an evaluator generation job by its ID.

SELECT
id,
created_at,
error,
finished_at,
inputs,
result,
status,
usage
FROM azure.ai_projects.beta_evaluators
WHERE job_id = '{{ job_id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list

List latest evaluator versions. Lists the latest version of each evaluator.

SELECT
id,
name,
display_name,
categories,
created_at,
created_by,
definition,
description,
evaluator_type,
generation_artifacts,
metadata,
modified_at,
supported_evaluation_levels,
tags,
version
FROM azure.ai_projects.beta_evaluators
WHERE endpoint = '{{ endpoint }}' -- required
AND type = '{{ type }}'
AND limit = '{{ limit }}'
;

Lifecycle Methods

create_version

Create an evaluator version. Creates a new evaluator version with an auto-incremented version identifier.

EXEC azure.ai_projects.beta_evaluators.create_version 
@name='{{ name }}' --required, 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"display_name": "{{ display_name }}", 
"metadata": "{{ metadata }}", 
"evaluator_type": "{{ evaluator_type }}", 
"categories": "{{ categories }}", 
"supported_evaluation_levels": "{{ supported_evaluation_levels }}", 
"definition": "{{ definition }}", 
"description": "{{ description }}", 
"tags": "{{ tags }}"
}'
;

list_generation_jobs

List evaluator generation jobs. Returns a list of evaluator generation jobs. The List API has up to a few seconds of propagation delay, so a recently created job may not appear immediately; use the Get evaluator generation job API with the job ID to retrieve a specific job without delay.

EXEC azure.ai_projects.beta_evaluators.list_generation_jobs 
@endpoint='{{ endpoint }}' --required, 
@limit='{{ limit }}', 
@order='{{ order }}', 
@after='{{ after }}', 
@before='{{ before }}'
;

create_generation_job

Create an evaluator generation job. Creates an evaluator generation job. The service generates rubric-based evaluator definitions from the provided source materials asynchronously.

EXEC azure.ai_projects.beta_evaluators.create_generation_job 
@endpoint='{{ endpoint }}' --required, 
@Operation-Id='{{ Operation-Id }}' 
@@json=
'{
"inputs": "{{ inputs }}"
}'
;

delete_version

Delete an evaluator version. Removes the specified evaluator version. Returns 204 whether the version existed or not.

EXEC azure.ai_projects.beta_evaluators.delete_version 
@name='{{ name }}' --required, 
@version='{{ version }}' --required, 
@endpoint='{{ endpoint }}' --required
;

update_version

Update an evaluator version. Updates the specified evaluator version in place.

EXEC azure.ai_projects.beta_evaluators.update_version 
@name='{{ name }}' --required, 
@version='{{ version }}' --required, 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"display_name": "{{ display_name }}", 
"metadata": "{{ metadata }}", 
"evaluator_type": "{{ evaluator_type }}", 
"categories": "{{ categories }}", 
"supported_evaluation_levels": "{{ supported_evaluation_levels }}", 
"definition": "{{ definition }}", 
"description": "{{ description }}", 
"tags": "{{ tags }}"
}'
;

delete_generation_job

Delete an evaluator generation job. Deletes an evaluator generation job by its ID. Deletes the job record only; the generated evaluator (if any) is preserved.

EXEC azure.ai_projects.beta_evaluators.delete_generation_job 
@job_id='{{ job_id }}' --required, 
@endpoint='{{ endpoint }}' --required
;

pending_upload

Start a pending upload. Initiates a new pending upload or retrieves an existing one for the specified evaluator version.

EXEC azure.ai_projects.beta_evaluators.pending_upload 
@name='{{ name }}' --required, 
@version='{{ version }}' --required, 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"pendingUploadId": "{{ pendingUploadId }}", 
"connectionName": "{{ connectionName }}", 
"pendingUploadType": "{{ pendingUploadType }}"
}'
;

cancel_generation_job

Cancel an evaluator generation job. Cancels an evaluator generation job by its ID.

EXEC azure.ai_projects.beta_evaluators.cancel_generation_job 
@job_id='{{ job_id }}' --required, 
@endpoint='{{ endpoint }}' --required
;