datasets

Creates, updates, deletes, gets or lists a datasets resource.

Overview

Name	datasets
Type	Resource
Id	azure.ai_projects.datasets

Fields

The following fields are returned by SELECT queries:

get

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
connectionName	string	The Azure Storage Account connection name. Required if startPendingUploadVersion was not called before creating the Dataset.
dataUri	string	URI of the data (example _). Required.
description	string	The asset description text.
isReference	boolean	Indicates if the dataset holds a reference to the storage, or the dataset manages storage itself. If true, the underlying data will not be deleted when the dataset version is deleted.
tags	object	Tag dictionary. Tags can be added, removed, and updated.
type	string	Dataset type. Required. Known values are: "uri_file" and "uri_folder".
version	string	The version of the resource. Required.

list_versions

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
connectionName	string	The Azure Storage Account connection name. Required if startPendingUploadVersion was not called before creating the Dataset.
dataUri	string	URI of the data (example _). Required.
description	string	The asset description text.
isReference	boolean	Indicates if the dataset holds a reference to the storage, or the dataset manages storage itself. If true, the underlying data will not be deleted when the dataset version is deleted.
tags	object	Tag dictionary. Tags can be added, removed, and updated.
type	string	Dataset type. Required. Known values are: "uri_file" and "uri_folder".
version	string	The version of the resource. Required.

list

Name	Datatype	Description
id	string	Asset ID, a unique identifier for the asset.
name	string	The name of the resource. Required.
connectionName	string	The Azure Storage Account connection name. Required if startPendingUploadVersion was not called before creating the Dataset.
dataUri	string	URI of the data (example _). Required.
description	string	The asset description text.
isReference	boolean	Indicates if the dataset holds a reference to the storage, or the dataset manages storage itself. If true, the underlying data will not be deleted when the dataset version is deleted.
tags	object	Tag dictionary. Tags can be added, removed, and updated.
type	string	Dataset type. Required. Known values are: "uri_file" and "uri_folder".
version	string	The version of the resource. Required.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get	select	name, version, endpoint		Get a version. Get the specific version of the DatasetVersion. The service returns 404 Not Found error if the DatasetVersion does not exist.
list_versions	select	name, endpoint		List versions. List all versions of the given DatasetVersion.
list	select	endpoint		List latest versions. List the latest version of each DatasetVersion.
create_or_update	insert	name, version, endpoint, dataUri, type		Create or update a version. Create a new or update an existing DatasetVersion with the given version id.
create_or_update	replace	name, version, endpoint, dataUri, type		Create or update a version. Create a new or update an existing DatasetVersion with the given version id.
delete	delete	name, version, endpoint		Delete a version. Delete the specific version of the DatasetVersion. The service returns 204 No Content if the DatasetVersion was deleted successfully or if the DatasetVersion does not exist.
get_credentials	exec	name, version, endpoint		Get dataset credentials. Gets the SAS credential to access the storage account associated with a Dataset version.
pending_upload	exec	name, version, endpoint, pendingUploadType		Start a pending upload. Initiates a new pending upload or retrieves an existing one for the specified dataset version.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
endpoint	string	The service endpoint host (no scheme), e.g. myaccount.table.cosmos.azure.com:443 - value of the client endpoint parameter. (default: )
name	string	The name of the resource. Required.
version	string	The specific version id of the DatasetVersion to operate on. Required.

SELECT examples

get

Get a version. Get the specific version of the DatasetVersion. The service returns 404 Not Found error if the DatasetVersion does not exist.

SELECT
id,
name,
connectionName,
dataUri,
description,
isReference,
tags,
type,
version
FROM azure.ai_projects.datasets
WHERE name = '{{ name }}' -- required
AND version = '{{ version }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list_versions

List versions. List all versions of the given DatasetVersion.

SELECT
id,
name,
connectionName,
dataUri,
description,
isReference,
tags,
type,
version
FROM azure.ai_projects.datasets
WHERE name = '{{ name }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list

List latest versions. List the latest version of each DatasetVersion.

SELECT
id,
name,
connectionName,
dataUri,
description,
isReference,
tags,
type,
version
FROM azure.ai_projects.datasets
WHERE endpoint = '{{ endpoint }}' -- required
;

INSERT examples

create_or_update

Create or update a version. Create a new or update an existing DatasetVersion with the given version id.

INSERT INTO azure.ai_projects.datasets (
dataUri,
type,
connectionName,
description,
tags,
name,
version,
endpoint
)
SELECT 
'{{ dataUri }}' /* required */,
'{{ type }}' /* required */,
'{{ connectionName }}',
'{{ description }}',
'{{ tags }}',
'{{ name }}',
'{{ version }}',
'{{ endpoint }}'
RETURNING
id,
name,
connectionName,
dataUri,
description,
isReference,
tags,
type,
version
;

Manifest

# Description fields are for documentation purposes
- name: datasets
props:
  - name: name
    value: "{{ name }}"
    description: Required parameter for the datasets resource.
  - name: version
    value: "{{ version }}"
    description: Required parameter for the datasets resource.
  - name: endpoint
    value: "{{ endpoint }}"
    description: Required parameter for the datasets resource.
  - name: dataUri
    value: "{{ dataUri }}"
    description: |
      URI of the data (`example `_). Required.
  - name: type
    value: "{{ type }}"
    description: |
      Dataset type. Required. Known values are: "uri_file" and "uri_folder".
  - name: connectionName
    value: "{{ connectionName }}"
    description: |
      The Azure Storage Account connection name. Required if startPendingUploadVersion was not called before creating the Dataset.
  - name: description
    value: "{{ description }}"
    description: |
      The asset description text.
  - name: tags
    value: "{{ tags }}"
    description: |
      Tag dictionary. Tags can be added, removed, and updated.

REPLACE examples

create_or_update

Create or update a version. Create a new or update an existing DatasetVersion with the given version id.

REPLACE azure.ai_projects.datasets
SET 
dataUri = '{{ dataUri }}',
type = '{{ type }}',
connectionName = '{{ connectionName }}',
description = '{{ description }}',
tags = '{{ tags }}'
WHERE 
name = '{{ name }}' --required
AND version = '{{ version }}' --required
AND endpoint = '{{ endpoint }}' --required
AND dataUri = '{{ dataUri }}' --required
AND type = '{{ type }}' --required
RETURNING
id,
name,
connectionName,
dataUri,
description,
isReference,
tags,
type,
version;

DELETE examples

delete

Delete a version. Delete the specific version of the DatasetVersion. The service returns 204 No Content if the DatasetVersion was deleted successfully or if the DatasetVersion does not exist.

DELETE FROM azure.ai_projects.datasets
WHERE name = '{{ name }}' --required
AND version = '{{ version }}' --required
AND endpoint = '{{ endpoint }}' --required
;

Lifecycle Methods

get_credentials

Get dataset credentials. Gets the SAS credential to access the storage account associated with a Dataset version.

EXEC azure.ai_projects.datasets.get_credentials 
@name='{{ name }}' --required, 
@version='{{ version }}' --required, 
@endpoint='{{ endpoint }}' --required
;

pending_upload

Start a pending upload. Initiates a new pending upload or retrieves an existing one for the specified dataset version.

EXEC azure.ai_projects.datasets.pending_upload 
@name='{{ name }}' --required, 
@version='{{ version }}' --required, 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"pendingUploadId": "{{ pendingUploadId }}", 
"connectionName": "{{ connectionName }}", 
"pendingUploadType": "{{ pendingUploadType }}"
}'
;