How to restore a replica after storage failure

This guide assumes that the <path> parameter in your config.xml file is set to:

<path>/var/lib/clickhouse/</path>

If you have configured a different data path, replace all instances of /var/lib/clickhouse in the below commands with the actual value of your <path> setting.

Copy access configuration from the healthy replica

Copy the contents of the access folder which contains local users from the healthy replica:

/var/lib/clickhouse/access

Back up the metadata folder from the healthy replica

Navigate to the ClickHouse data directory:

cd /var/lib/clickhouse

Create a backup of the metadata folder (including symbolic links): The metadata directory contains DDLs for databases and tables. The database directory has symlinks to /var/lib/clickhouse/store/.. which contains all the table DDLs.

{ find metadata -type f; find metadata -type l; find metadata -type l | xargs readlink -f; } | tar -cPf backup.tar --files-from=-

This command ensures that both the metadata files, and the symlink architecture are preserved in the backup.

Restore the metadata on the faulty replica

Copy the generated backup.tar file to the faulty replica.
Extract it to the ClickHouse data directory:

cd /var/lib/clickhouse/
tar -xvPf backup.tar

Create the force restore flag

To trigger automatic data synchronization from other replicas, create the following flag:

sudo -u clickhouse touch /var/lib/clickhouse/flags/force_restore_data

Restart the faulty replica

Restart the ClickHouse server on the faulty node.
Check the server logs, you should observe parts being downloaded from the healthy replicas:

11.02 00:00:04.047097 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Downloading files 23
11.02 00:00:04.055542 [ 682 ] {} <Debug> analytics.events_local (...) (Fetcher): Download of part 202511_0_0_0 onto disk disk2 finished.
11.02 00:00:04.101888 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2025_0_0_1 onto disk default.
11.02 00:00:04.102005 [ 687 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11
11.02 00:00:04.102210 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2022_0_0_1 onto disk disk1.
11.02 00:00:04.102247 [ 688 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading part 2021_0_0_1 onto disk disk2.
11.02 00:00:04.102331 [ 690 ] {} <Debug> warehouse.customers_local (...) (Fetcher): Downloading files 11