> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-detect-table-modification.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> Step-by-step guide to create your first Amazon S3 ClickPipe.

# Creating your first Amazon S3 ClickPipe

export const Image = ({img, alt, size = "lg"}) => {
  const normalizedSize = ["sm", "md", "lg"].includes(size) ? size : "lg";
  return <div className={`ch-image-${normalizedSize}`}>
      <Frame>
        <img src={img} alt={alt} />
      </Frame>
    </div>;
};

The S3 ClickPipe provides a fully-managed and resilient way to ingest data from Amazon S3 and S3-compatible object stores into ClickHouse Cloud. It supports both **one-time** and **continuous ingestion** with exactly-once semantics.

<Steps>
  <Step title="Select the data source" id="1-select-the-data-source">
    **1.** In ClickHouse Cloud, select **Data sources** in the main navigation menu and click **Create ClickPipe**.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step0.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=5c127cc5de6fe7160371dce223f06b60" alt="Select imports" width="2606" height="790" data-path="images/integrations/data-ingestion/clickpipes/cp_step0.webp" />
    </Frame>

    **2.** Click the **Amazon S3** tile. You can also use this tile to connect to other S3-compatible services not listed in the ClickPipes UI.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step1.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=486454546bfb918f198b6ff6bb18dddf" alt="Select imports" width="3016" height="954" data-path="images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step1.webp" />
    </Frame>

    <Tip>
      Due to differences in URL formats and API implementations across object storage service providers, not all S3-compatible services are supported out-of-the-box. If you're running into issues with a service that isn't listed under [supported data sources](/integrations/clickpipes/object-storage/amazon-s3/overview#supported-data-sources), please [reach out to our team](https://clickhouse.com/company/contact?loc=clickpipes).
    </Tip>
  </Step>

  <Step title="Setup your ClickPipe connection" id="2-setup-your-clickpipe-connection">
    **1.** To setup a new ClickPipe, you must provide details on how to connect to and authenticate with your object storage service.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step2.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=779c2452cdf98845aca3576fe469855a" alt="Fill out connection details" width="3014" height="1402" data-path="images/integrations/data-ingestion/clickpipes/object-storage/amazon-s3/cp_step2.webp" />
    </Frame>

    * **Authentication method**: the S3 ClickPipe supports [IAM credentials](/integrations/clickpipes/object-storage/amazon-s3/overview#iam-credentials) (`Credentials`) and [IAM role-based authentication](/integrations/clickpipes/object-storage/amazon-s3/overview#iam-role) (`IAM role`). See the [reference documentation](/integrations/clickpipes/object-storage/amazon-s3/overview#access-control) for guidance on authentication and permissions.

    * **S3 file path**: the S3 ClickPipe expects a [virtual-hosted-style URI](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html#virtual-hosted-style-access).

      ```bash theme={null}
      https://bucket-name.s3.region-code.amazonaws.com/key-name
      ```

      You can use POSIX wildcards to match multiple files or prefixes. See the [reference documentation](/integrations/clickpipes/object-storage/amazon-s3/overview#file-pattern-matching) for guidance on supported patterns.

    **2.** Click **Incoming data**. ClickPipes will fetch metadata from your bucket for the next step.
  </Step>

  <Step title="Select data format" id="3-select-data-format">
    The UI will display a list of files in the specified bucket.
    Select your data format (we currently support a subset of ClickHouse formats) and if you want to enable continuous ingestion.
    See the "continuous ingest" section in the overview page for more details.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step3_object_storage.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=da8ab02f25b125878a2dee63f69a25ec" alt="Set data format and topic" width="2336" height="1566" data-path="images/integrations/data-ingestion/clickpipes/cp_step3_object_storage.webp" />
    </Frame>
  </Step>

  <Step title="Configure table, schema and settings" id="5-configure-table-schema-settings">
    In the next step, you can select whether you want to ingest data into a new ClickHouse table or reuse an existing one.
    Follow the instructions in the screen to modify your table name, schema, and settings.
    You can see a real-time preview of your changes in the sample table at the top.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step4a.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=b9f12f6c0d73035e1aada3ba43ebbad2" alt="Set table, schema, and settings" width="2734" height="1328" data-path="images/integrations/data-ingestion/clickpipes/cp_step4a.webp" />
    </Frame>

    You can also customize the advanced settings using the controls provided

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step4a3.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=082d0fc28d1d2f8af315e1cfa3d1e0d2" alt="Set advanced controls" width="1592" height="338" data-path="images/integrations/data-ingestion/clickpipes/cp_step4a3.webp" />
    </Frame>

    Alternatively, you can decide to ingest your data in an existing ClickHouse table.
    In that case, the UI will allow you to map fields from the source to the ClickHouse fields in the selected destination table.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step4b.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=5a68f31fdac86791722db4a0faa0f298" alt="Use an existing table" width="2738" height="1322" data-path="images/integrations/data-ingestion/clickpipes/cp_step4b.webp" />
    </Frame>

    <Info>
      You can also map [virtual columns](/reference/functions/table-functions/s3#virtual-columns), like `_path` or `_size`, to fields.
    </Info>
  </Step>

  <Step title="Configure permissions" id="6-configure-permissions">
    Finally, you can configure permissions for the internal ClickPipes user.

    **Permissions:** ClickPipes will create a dedicated user for writing data into a destination table. You can select a role for this internal user using a custom role or one of the predefined role:

    * `Full access`: with the full access to the cluster. Required if you use materialized view or Dictionary with the destination table.
    * `Only destination table`: with the `INSERT` permissions to the destination table only.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_step5.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=3a3657fe0c594a1217d71290f27d096a" alt="Permissions" width="2736" height="1318" data-path="images/integrations/data-ingestion/clickpipes/cp_step5.webp" />
    </Frame>
  </Step>

  <Step title="Complete setup" id="7-complete-setup">
    By clicking on "Complete Setup", the system will register your ClickPipe, and you'll be able to see it listed in the summary table.

    <div className="ch-image-sm">
      <Frame>
        <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_success.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=81d5e86ce8d7315d5463f45d54e4f53c" alt="Success notice" width="613" height="212" data-path="images/integrations/data-ingestion/clickpipes/cp_success.webp" />
      </Frame>
    </div>

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_remove.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=c4cc25a78f1bdb4212d42cf8659bd02d" alt="Remove notice" width="2756" height="1320" data-path="images/integrations/data-ingestion/clickpipes/cp_remove.webp" />
    </Frame>

    The summary table provides controls to display sample data from the source or the destination table in ClickHouse

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_destination.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=186557ea6d5b5189f972399519a60165" alt="View destination" width="2758" height="1316" data-path="images/integrations/data-ingestion/clickpipes/cp_destination.webp" />
    </Frame>

    As well as controls to remove the ClickPipe and display a summary of the ingest job.

    <Frame>
      <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/Qdrmbc1T54ihl0_n/images/integrations/data-ingestion/clickpipes/cp_overview.webp?fit=max&auto=format&n=Qdrmbc1T54ihl0_n&q=85&s=1dfc009ce729fb48f46e3e7cd6c83581" alt="View overview" width="3436" height="1076" data-path="images/integrations/data-ingestion/clickpipes/cp_overview.webp" />
    </Frame>

    **Congratulations!** you have successfully set up your first ClickPipe.
    If this is a ClickPipe configure for continuous ingestion, it will be continuously running, ingesting data in real-time from your remote data source.
    Otherwise, it will ingest the batch and complete.
  </Step>
</Steps>
