> ## Documentation Index
> Fetch the complete documentation index at: https://private-7c7dfe99-detect-table-modification.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

> The official Kafka connector from ClickHouse with Amazon MSK

# Integrating Amazon MSK with ClickHouse

<div class="vimeo-container">
  <Frame>
    <iframe
      src="//www.youtube.com/embed/6lKI_WlQ3-s"
      frameborder="0"
      allow="autoplay;
fullscreen;
picture-in-picture"
      allowfullscreen
    />
  </Frame>
</div>

> Note: The policy shown in the video is permissive and intended for quick start only. See least‑privilege IAM guidance below.

<h2 id="prerequisites">
  Prerequisites
</h2>

We assume:

* you're familiar with [ClickHouse Connector Sink](/integrations/connectors/data-ingestion/kafka/kafka-clickhouse-connect-sink),
* you're familiar with Amazon MSK and MSK Connectors. We recommend the Amazon MSK [Getting Started guide](https://docs.aws.amazon.com/msk/latest/developerguide/getting-started.html) and [MSK Connect guide](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect.html).

<h2 id="the-official-kafka-connector-from-clickhouse-with-amazon-msk">
  The official Kafka connector from ClickHouse with Amazon MSK
</h2>

<h3 id="gather-your-connection-details">
  Gather your connection details
</h3>

To connect to ClickHouse with HTTP(S) you need this information:

| Parameter(s)              | Description                                                                                                    |
| ------------------------- | -------------------------------------------------------------------------------------------------------------- |
| `HOST` and `PORT`         | Typically, the port is 8443 when using TLS or 8123 when not using TLS.                                         |
| `DATABASE NAME`           | Out of the box, there is a database named `default`, use the name of the database that you want to connect to. |
| `USERNAME` and `PASSWORD` | Out of the box, the username is `default`. Use the username appropriate for your use case.                     |

The details for your ClickHouse Cloud service are available in the ClickHouse Cloud console.
Select a service and click **Connect**:

<div className="ch-image-md">
  <Frame>
    <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/0yJtAU6cqYAQlCWC/images/_snippets/cloud-connect-button.webp?fit=max&auto=format&n=0yJtAU6cqYAQlCWC&q=85&s=2d1e2340e1ddccf3199ca1e41eeba836" alt="ClickHouse Cloud service connect button" width="998" height="932" data-path="images/_snippets/cloud-connect-button.webp" />
  </Frame>
</div>

Choose **HTTPS**. Connection details are displayed in an example `curl` command.

<div className="ch-image-md">
  <Frame>
    <img src="https://mintcdn.com/private-7c7dfe99-detect-table-modification/0yJtAU6cqYAQlCWC/images/_snippets/connection-details-https.webp?fit=max&auto=format&n=0yJtAU6cqYAQlCWC&q=85&s=2ea2af378046d5ec87ea8e78de573bc1" alt="ClickHouse Cloud HTTPS connection details" width="1320" height="1184" data-path="images/_snippets/connection-details-https.webp" />
  </Frame>
</div>

If you're using self-managed ClickHouse, the connection details are set by your ClickHouse administrator.

<h3 id="steps">
  Steps
</h3>

1. Make sure you're familiar with the [ClickHouse Connector Sink](/integrations/connectors/data-ingestion/kafka/kafka-clickhouse-connect-sink)
2. [Create an MSK instance](https://docs.aws.amazon.com/msk/latest/developerguide/create-cluster.html).
3. [Create and assign IAM role](https://docs.aws.amazon.com/msk/latest/developerguide/create-client-iam-role.html).
4. Download a `jar` file from ClickHouse Connect Sink [Release page](https://github.com/ClickHouse/clickhouse-kafka-connect/releases).
5. Install the downloaded `jar` file on [Custom plugin page](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-plugins.html) of Amazon MSK console.
6. If Connector communicates with a public ClickHouse instance, [enable internet access](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-internet-access.html).
7. Provide a topic name, ClickHouse instance hostname, and password in config.

```yml theme={null}
connector.class=com.clickhouse.kafka.connect.ClickHouseSinkConnector
tasks.max=1
topics=<topic_name>
ssl=true
security.protocol=SSL
hostname=<hostname>
database=<database_name>
password=<password>
ssl.truststore.location=/tmp/kafka.client.truststore.jks
port=8443
value.converter.schemas.enable=false
value.converter=org.apache.kafka.connect.json.JsonConverter
exactlyOnce=true
username=default
schemas.enable=false
```

<h2 id="iam-least-privilege">
  Recommended IAM permissions (least privilege)
</h2>

Use the smallest set of permissions required for your setup. Start with the baseline below and add optional services only if you use them.

```json theme={null}
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "MSKClusterAccess",
      "Effect": "Allow",
      "Action": [
        "kafka:DescribeCluster",
        "kafka:GetBootstrapBrokers",
        "kafka:DescribeClusterV2",
        "kafka:ListClusters",
        "kafka:ListClustersV2"
      ],
      "Resource": "*"
    },
    {
      "Sid": "KafkaAuthorization",
      "Effect": "Allow",
      "Action": [
        "kafka-cluster:Connect",
        "kafka-cluster:DescribeCluster",
        "kafka-cluster:DescribeGroup",
        "kafka-cluster:DescribeTopic",
        "kafka-cluster:ReadData"
      ],
      "Resource": "*"
    },
    {
      "Sid": "OptionalGlueSchemaRegistry",
      "Effect": "Allow",
      "Action": [
        "glue:GetSchema*",
        "glue:ListSchemas",
        "glue:ListSchemaVersions"
      ],
      "Resource": "*"
    },
    {
      "Sid": "OptionalSecretsManager",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "arn:aws:secretsmanager:<region>:<account-id>:secret:<your-secret-name>*"
      ]
    },
    {
      "Sid": "OptionalS3Read",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject"
      ],
      "Resource": "arn:aws:s3:::<your-bucket>/<optional-prefix>/*"
    }
  ]
}
```

* Use the Glue block only if you use AWS Glue Schema Registry.
* Use the Secrets Manager block only if you fetch credentials/truststores from Secrets Manager. Scope the ARN.
* Use the S3 block only if you load artifacts (e.g., truststore) from S3. Scope to bucket/prefix.

See also: [Kafka best practices – IAM](/integrations/clickpipes/kafka/best-practices#iam).

<h2 id="performance-tuning">
  Performance tuning
</h2>

One way of increasing performance is to adjust the batch size and the number of records that are fetched from Kafka by adding the following to the **worker** configuration:

```yml theme={null}
consumer.max.poll.records=[NUMBER OF RECORDS]
consumer.max.partition.fetch.bytes=[NUMBER OF RECORDS * RECORD SIZE IN BYTES]
```

The specific values you use are going to vary, based on desired number of records and record size. For example, the default values are:

```yml theme={null}
consumer.max.poll.records=500
consumer.max.partition.fetch.bytes=1048576
```

You can find more details (both implementation and other considerations) in the official [Kafka](https://kafka.apache.org/documentation/#consumerconfigs) and
[Amazon MSK](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-workers.html#msk-connect-create-custom-worker-config) documentation.

<h2 id="notes-on-networking-for-msk-connect">
  Notes on networking for MSK Connect
</h2>

In order for MSK Connect to connect to ClickHouse, we recommend your MSK cluster to be in a private subnet with a Private NAT connected for internet access. Instructions on how to set this up are provided below. Note that public subnets are supported but not recommended due to the need to constantly assign an Elastic IP address to your ENI, [AWS provides more details here](https://docs.aws.amazon.com/msk/latest/developerguide/msk-connect-internet-access.html)

1. **Create a Private Subnet:** Create a new subnet within your VPC, designating it as a private subnet. This subnet shouldn't have direct access to the internet.
2. **Create a NAT Gateway:** Create a NAT gateway in a public subnet of your VPC. The NAT gateway enables instances in your private subnet to connect to the internet or other AWS services, but prevents the internet from initiating a connection with those instances.
3. **Update the Route Table:** Add a route that directs internet-bound traffic to the NAT gateway
4. **Ensure Security Groups and Network ACLs Configuration:** Configure your [security groups](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-security-groups.html) and [network ACLs (Access Control Lists)](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-network-acls.html) to allow relevant traffic.
   1. From MSK Connect worker ENIs to MSK brokers on TLS port (commonly 9094).
   2. From MSK Connect worker ENIs to ClickHouse endpoint: 9440 (native TLS) or 8443 (HTTPS).
   3. Allow inbound on broker SG from the MSK Connect worker SG.
   4. For self-hosted ClickHouse, open the port configured in your server (default 8123 for HTTP).
5. **Attach Security Groups to MSK:** Ensure that these security groups are attached to your MSK cluster and MSK Connect workers.
6. **Connectivity to ClickHouse Cloud:**
   1. Public endpoint + IP allowlist: requires NAT egress from private subnets.
   2. Private connectivity where available (e.g., VPC peering/PrivateLink/VPN). Ensure VPC DNS hostnames/resolution are enabled and DNS can resolve the private endpoint.
7. **Validate connectivity (quick checklist):**
   1. From the connector environment, resolve MSK bootstrap DNS and connect via TLS to broker port.
   2. Establish TLS connection to ClickHouse on port 9440 (or 8443 for HTTPS).
   3. If using AWS services (Glue/Secrets Manager), allow egress to those endpoints.