DataLakeCatalog - ClickHouse Documentation

The DataLakeCatalog database engine enables you to connect ClickHouse to external data catalogs and query open table format data without the need for data duplication. This transforms ClickHouse into a powerful query engine that works seamlessly with your existing data lake infrastructure.

Supported catalogs

The DataLakeCatalog engine supports the following data catalogs:

AWS Glue Catalog - For Iceberg tables in AWS environments
Databricks Unity Catalog - For Delta Lake and Iceberg tables
Hive Metastore - Traditional Hadoop ecosystem catalog
REST Catalogs - Any catalog supporting the Iceberg REST specification

Creating a database

You will need to enable the relevant settings below to use the DataLakeCatalog engine:

SET allow_experimental_database_iceberg = 1;
SET allow_experimental_database_unity_catalog = 1;
SET allow_experimental_database_glue_catalog = 1;
SET allow_experimental_database_hms_catalog = 1;
SET allow_experimental_database_paimon_rest_catalog = 1;

Databases with the DataLakeCatalog engine can be created using the following syntax:

CREATE DATABASE database_name
ENGINE = DataLakeCatalog(catalog_endpoint[, user, password])
SETTINGS
catalog_type,
[...]

The following settings are supported:

Setting	Description
`catalog_type`	Type of catalog: `glue`, `unity` (Delta), `rest` (Iceberg), `hive`, `onelake` (Iceberg), `delta_sharing` (Iceberg, flat namespaces)
`warehouse`	The warehouse/database name to use in the catalog.
`catalog_credential`	Authentication credential for the catalog (e.g., API key or token)
`auth_header`	Custom HTTP header for authentication with the catalog service
`auth_scope`	OAuth2 scope for authentication (if using OAuth)
`storage_endpoint`	Endpoint URL for the underlying storage
`oauth_server_uri`	URI of the OAuth2 authorization server for authentication
`vended_credentials`	Boolean indicating whether to use vended credentials from the catalog (supports AWS S3 and Azure ADLS Gen2)
`aws_access_key_id`	AWS access key ID for S3/Glue access (if not using vended credentials)
`aws_secret_access_key`	AWS secret access key for S3/Glue access (if not using vended credentials)
`region`	AWS region for the service (e.g., `us-east-1`)
`dlf_access_key_id`	Access key ID for DLF access
`dlf_access_key_secret`	Access key Secret for DLF access
`force_add_bucket`	When constructing object-storage URLs from the catalog-provided table location and `storage_endpoint`, prepend the bucket/container name even if the endpoint already contains it. Default: `false`. Set to `true` for catalogs that hand back paths without the bucket and require it to be added at the URL-construction step (Polaris-style paths).

Examples

See below sections for examples of using the DataLakeCatalog engine:

Unity Catalog
Glue Catalog
OneLake Catalog Can be used by enabling allow_experimental_database_iceberg or allow_database_iceberg.

CREATE DATABASE database_name
ENGINE = DataLakeCatalog(catalog_endpoint)
SETTINGS
    catalog_type = 'onelake',
    warehouse = warehouse,
    onelake_tenant_id = tenant_id,
    oauth_server_uri = server_uri,
    auth_scope = auth_scope,
    onelake_client_id = client_id,
    onelake_client_secret = client_secret;
SHOW TABLES IN database_name;
SELECT count() from database_name.table_name;

To authenticate without sharing a client secret, set onelake_bearer_token to a pre-obtained bearer token (scoped to https://storage.azure.com) instead of onelake_client_id/onelake_client_secret. ClickHouse does not refresh the token, so the database must be recreated after it expires.

​Supported catalogs

​Creating a database

​Examples

Supported catalogs

Creating a database

Examples