ingestion

Creates, updates, deletes, gets or lists an ingestion resource.

Overview

Name	ingestion
Type	Resource
Id	azure.planetary_computer_dataplane.ingestion

Fields

The following fields are returned by SELECT queries:

get_run

Name	Datatype	Description
id	string	Run id. Required.
creationTime	string (date-time)	Creation time. Required.
keepOriginalAssets	boolean	Keep original source assets.
operation	object	Operation. Required.
parentRunId	string	Run id which this run is associated to because it has been retried or rerun.
skipExistingItems	boolean	Skip any item that already exist in the GeoCatalog.
sourceCatalogUrl	string	URL of the source catalog.

get

Name	Datatype	Description
id	string	Ingestion id. Required.
creationTime	string (date-time)	Ingestion creation time. Required.
displayName	string	Ingestion name.
importType	string	Ingestion type. Required. Known values are: "StaticCatalog" and "StacGeoparquet". (StaticCatalog, StacGeoparquet)
keepOriginalAssets	boolean	Keep original source assets.
skipExistingItems	boolean	Skip processing existing items in the catalog.
sourceCatalogUrl	string	Source catalog URL. Required for StaticCatalog ingestion type.
stacGeoparquetUrl	string	Parquet catalog URL. Required for StacGeoparquet ingestion type.
status	string	Ingestion status. Required. Known values are: "Ready" and "Deleting". (Ready, Deleting)

list

Name	Datatype	Description
id	string	Ingestion id. Required.
creationTime	string (date-time)	Ingestion creation time. Required.
displayName	string	Ingestion name.
importType	string	Ingestion type. Required. Known values are: "StaticCatalog" and "StacGeoparquet". (StaticCatalog, StacGeoparquet)
keepOriginalAssets	boolean	Keep original source assets.
skipExistingItems	boolean	Skip processing existing items in the catalog.
sourceCatalogUrl	string	Source catalog URL. Required for StaticCatalog ingestion type.
stacGeoparquetUrl	string	Parquet catalog URL. Required for StacGeoparquet ingestion type.
status	string	Ingestion status. Required. Known values are: "Ready" and "Deleting". (Ready, Deleting)

get_operation

Name	Datatype	Description
id	string	Operation id. Required.
additionalInformation	object	Additional information elements about the particular operation type.
collectionId	string	Collection ID.
creationTime	string (date-time)	The UTC time at which the operation was created. Required.
error	object	Error information.
finishTime	string (date-time)	The UTC time at which the operation finished its execution.
startTime	string (date-time)	The UTC time at which the operation was started.
status	string	Operation status. Required. Known values are: "Pending", "Running", "Succeeded", "Canceled", "Canceling", and "Failed". (Pending, Running, Succeeded, Canceled, Canceling, Failed)
statusHistory	array	The history of the operation status in time. Required.
type	string	Operation type. Required.

get_source

Name	Datatype	Description
id	string	Ingestion source id. Required.
created	string (date-time)	Created time in UTC format.
kind	string	Discriminator for the ingestion source. Required. Known values are: "SasToken" and "BlobManagedIdentity".

list_operations

Name	Datatype	Description
id	string	Operation id. Required.
additionalInformation	object	Additional information elements about the particular operation type.
collectionId	string	Collection ID.
creationTime	string (date-time)	The UTC time at which the operation was created. Required.
error	object	Error information.
finishTime	string (date-time)	The UTC time at which the operation finished its execution.
startTime	string (date-time)	The UTC time at which the operation was started.
status	string	Operation status. Required. Known values are: "Pending", "Running", "Succeeded", "Canceled", "Canceling", and "Failed". (Pending, Running, Succeeded, Canceled, Canceling, Failed)
statusHistory	array	The history of the operation status in time. Required.
type	string	Operation type. Required.

Methods

The following methods are available for this resource:

Name	Accessible by	Required Params	Optional Params	Description
get_run	select	collection_id, ingestion_id, run_id, endpoint		Get a run of an ingestion.
get	select	collection_id, ingestion_id, endpoint		Get the definition of an ingestion.
list	select	collection_id, endpoint	$top, $skip	Get ingestions of a catalog.
get_operation	select	operation_id, endpoint		Get an operation of a geo-catalog collection.
get_source	select	id, endpoint		Get an ingestion source in a geo-catalog.
list_operations	select	endpoint	$top, $skip, collectionId, status	Get operations of a geo-catalog collection.
create	insert	collection_id, endpoint, importType		Create a new ingestion.
update	update	collection_id, ingestion_id, endpoint, importType		Update an existing ingestion.
delete	delete	collection_id, ingestion_id, endpoint		Delete an ingestion from a catalog. All runs of the ingestion will be deleted. Ingestion must not have any runs in progress or queued.
cancel_operation	delete	operation_id, endpoint		Cancel a running operation of a geo-catalog collection.
cancel_all_operations	delete	endpoint		Cancel all pending and running operations across the entire GeoCatalog instance. This is a catalog-wide operation and is not scoped to a specific collection.
list_runs	exec	collection_id, ingestion_id, endpoint	$top, $skip	Get the runs of an ingestion.
create_run	exec	collection_id, ingestion_id, endpoint		Create a new run of an ingestion.
list_sources	exec	endpoint	$top, $skip	Get ingestion sources in a geo-catalog.
create_source	exec	endpoint, id, kind		Create a new ingestion source in a geo-catalog.
list_managed_identities	exec	endpoint		Get all managed identities with access to storage accounts configured for a geo-catalog.
replace_source	exec	id, endpoint, id, kind		Update an existing ingestion source in a geo-catalog.
delete_source	exec	id, endpoint		Delete an ingestion source from a geo-catalog.

Parameters

Parameters can be passed in the WHERE clause of a query. Check the Methods section to see which parameters are required or optional for each operation.

Name	Datatype	Description
collection_id	string	Catalog collection id. Required.
endpoint	string	The service endpoint host (no scheme), e.g. myaccount.table.cosmos.azure.com:443 - value of the client endpoint parameter. (default: )
id	string	Ingestion source id. Required.
ingestion_id	string	Ingestion id. Required.
operation_id	string	Operation id. Required.
run_id	string	Run id. Required.
$skip	integer	The number of items to skip. Default value is None.
$top	integer	The number of items to return. Default value is None.
collectionId	string	Operation id used to filter the results. Default value is None.
status	string	Operation status used to filter the results. Known values are: "Pending", "Running", "Succeeded", "Canceled", "Canceling", and "Failed". Default value is None.

SELECT examples

get_run

Get a run of an ingestion.

SELECT
id,
creationTime,
keepOriginalAssets,
operation,
parentRunId,
skipExistingItems,
sourceCatalogUrl
FROM azure.planetary_computer_dataplane.ingestion
WHERE collection_id = '{{ collection_id }}' -- required
AND ingestion_id = '{{ ingestion_id }}' -- required
AND run_id = '{{ run_id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

get

Get the definition of an ingestion.

SELECT
id,
creationTime,
displayName,
importType,
keepOriginalAssets,
skipExistingItems,
sourceCatalogUrl,
stacGeoparquetUrl,
status
FROM azure.planetary_computer_dataplane.ingestion
WHERE collection_id = '{{ collection_id }}' -- required
AND ingestion_id = '{{ ingestion_id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list

Get ingestions of a catalog.

SELECT
id,
creationTime,
displayName,
importType,
keepOriginalAssets,
skipExistingItems,
sourceCatalogUrl,
stacGeoparquetUrl,
status
FROM azure.planetary_computer_dataplane.ingestion
WHERE collection_id = '{{ collection_id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
AND $top = '{{ $top }}'
AND $skip = '{{ $skip }}'
;

get_operation

Get an operation of a geo-catalog collection.

SELECT
id,
additionalInformation,
collectionId,
creationTime,
error,
finishTime,
startTime,
status,
statusHistory,
type
FROM azure.planetary_computer_dataplane.ingestion
WHERE operation_id = '{{ operation_id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

get_source

Get an ingestion source in a geo-catalog.

SELECT
id,
created,
kind
FROM azure.planetary_computer_dataplane.ingestion
WHERE id = '{{ id }}' -- required
AND endpoint = '{{ endpoint }}' -- required
;

list_operations

Get operations of a geo-catalog collection.

SELECT
id,
additionalInformation,
collectionId,
creationTime,
error,
finishTime,
startTime,
status,
statusHistory,
type
FROM azure.planetary_computer_dataplane.ingestion
WHERE endpoint = '{{ endpoint }}' -- required
AND $top = '{{ $top }}'
AND $skip = '{{ $skip }}'
AND collectionId = '{{ collectionId }}'
AND status = '{{ status }}'
;

INSERT examples

create

Create a new ingestion.

INSERT INTO azure.planetary_computer_dataplane.ingestion (
importType,
displayName,
sourceCatalogUrl,
stacGeoparquetUrl,
skipExistingItems,
keepOriginalAssets,
collection_id,
endpoint
)
SELECT 
'{{ importType }}' /* required */,
'{{ displayName }}',
'{{ sourceCatalogUrl }}',
'{{ stacGeoparquetUrl }}',
{{ skipExistingItems }},
{{ keepOriginalAssets }},
'{{ collection_id }}',
'{{ endpoint }}'
RETURNING
id,
creationTime,
displayName,
importType,
keepOriginalAssets,
skipExistingItems,
sourceCatalogUrl,
stacGeoparquetUrl,
status
;

Manifest

# Description fields are for documentation purposes
- name: ingestion
props:
  - name: collection_id
    value: "{{ collection_id }}"
    description: Required parameter for the ingestion resource.
  - name: endpoint
    value: "{{ endpoint }}"
    description: Required parameter for the ingestion resource.
  - name: importType
    value: "{{ importType }}"
    description: |
      Ingestion type. Required. Known values are: "StaticCatalog" and "StacGeoparquet".
    valid_values: ['StaticCatalog', 'StacGeoparquet']
  - name: displayName
    value: "{{ displayName }}"
    description: |
      Ingestion name.
  - name: sourceCatalogUrl
    value: "{{ sourceCatalogUrl }}"
    description: |
      Source catalog URL. Required for StaticCatalog ingestion type.
  - name: stacGeoparquetUrl
    value: "{{ stacGeoparquetUrl }}"
    description: |
      Parquet catalog URL. Required for StacGeoparquet ingestion type.
  - name: skipExistingItems
    value: {{ skipExistingItems }}
    description: |
      Skip processing existing items in the catalog.
  - name: keepOriginalAssets
    value: {{ keepOriginalAssets }}
    description: |
      Keep original source assets.

UPDATE examples

update

Update an existing ingestion.

UPDATE azure.planetary_computer_dataplane.ingestion
SET 
importType = '{{ importType }}',
displayName = '{{ displayName }}',
sourceCatalogUrl = '{{ sourceCatalogUrl }}',
stacGeoparquetUrl = '{{ stacGeoparquetUrl }}',
skipExistingItems = {{ skipExistingItems }},
keepOriginalAssets = {{ keepOriginalAssets }}
WHERE 
collection_id = '{{ collection_id }}' --required
AND ingestion_id = '{{ ingestion_id }}' --required
AND endpoint = '{{ endpoint }}' --required
AND importType = '{{ importType }}' --required
RETURNING
id,
creationTime,
displayName,
importType,
keepOriginalAssets,
skipExistingItems,
sourceCatalogUrl,
stacGeoparquetUrl,
status;

DELETE examples

delete

Delete an ingestion from a catalog. All runs of the ingestion will be deleted. Ingestion must not have any runs in progress or queued.

DELETE FROM azure.planetary_computer_dataplane.ingestion
WHERE collection_id = '{{ collection_id }}' --required
AND ingestion_id = '{{ ingestion_id }}' --required
AND endpoint = '{{ endpoint }}' --required
;

cancel_operation

Cancel a running operation of a geo-catalog collection.

DELETE FROM azure.planetary_computer_dataplane.ingestion
WHERE operation_id = '{{ operation_id }}' --required
AND endpoint = '{{ endpoint }}' --required
;

cancel_all_operations

Cancel all pending and running operations across the entire GeoCatalog instance. This is a catalog-wide operation and is not scoped to a specific collection.

DELETE FROM azure.planetary_computer_dataplane.ingestion
WHERE endpoint = '{{ endpoint }}' --required
;

Lifecycle Methods

list_runs

Get the runs of an ingestion.

EXEC azure.planetary_computer_dataplane.ingestion.list_runs 
@collection_id='{{ collection_id }}' --required, 
@ingestion_id='{{ ingestion_id }}' --required, 
@endpoint='{{ endpoint }}' --required, 
@$top='{{ $top }}', 
@$skip='{{ $skip }}'
;

create_run

Create a new run of an ingestion.

EXEC azure.planetary_computer_dataplane.ingestion.create_run 
@collection_id='{{ collection_id }}' --required, 
@ingestion_id='{{ ingestion_id }}' --required, 
@endpoint='{{ endpoint }}' --required
;

list_sources

Get ingestion sources in a geo-catalog.

EXEC azure.planetary_computer_dataplane.ingestion.list_sources 
@endpoint='{{ endpoint }}' --required, 
@$top='{{ $top }}', 
@$skip='{{ $skip }}'
;

create_source

Create a new ingestion source in a geo-catalog.

EXEC azure.planetary_computer_dataplane.ingestion.create_source 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"id": "{{ id }}", 
"kind": "{{ kind }}"
}'
;

list_managed_identities

Get all managed identities with access to storage accounts configured for a geo-catalog.

EXEC azure.planetary_computer_dataplane.ingestion.list_managed_identities 
@endpoint='{{ endpoint }}' --required
;

replace_source

Update an existing ingestion source in a geo-catalog.

EXEC azure.planetary_computer_dataplane.ingestion.replace_source 
@id='{{ id }}' --required, 
@endpoint='{{ endpoint }}' --required 
@@json=
'{
"id": "{{ id }}", 
"kind": "{{ kind }}"
}'
;

delete_source

Delete an ingestion source from a geo-catalog.

EXEC azure.planetary_computer_dataplane.ingestion.delete_source 
@id='{{ id }}' --required, 
@endpoint='{{ endpoint }}' --required
;