Dump OpenSearch® index using elasticsearch-dump

It is a good practice to perform backups of your OpenSearch® data to another storage service. This way you can access your data and restore it if something unexpected happens to it.

In this article, you can find out how to dump your OpenSearch data to an:

To copy the index data, we will be using elasticsearch-dump tool. You can read the instructions on GitHub on how to install it. From this library, we will use elasticdump command to copy the input index data to an specific output.

Note

Make sure to have elasticsearch-dump tool installed for the next steps.

Copying data from OpenSearch to Aiven for OpenSearch®

Prerequisites

  • OpenSearch cluster as the input (can be in Aiven or elsewhere)

  • Aiven for OpenSearch cluster as the output

Note

The input and ouput can be either an OpenSearch URL or a file path (local or remote file storage). In this particular case, we are using both URLs, one from an OpenSearch cluster and the other one from Aiven for OpenSearch cluster.

Here are some information you need to collect to copy your data:

OpenSearch cluster:

  • INPUT_SERVICE_URI: OpenSearch cluster URL, for e.g. http://opensearch-url:9243/

  • INPUT_INDEX_NAME: the index that you aim to copy from your input source.

Aiven for OpenSearch:

  • OUTPUT_INDEX_NAME: the index that you want to have in your output with the copied data.

  • SERVICE_URI: OpenSearch service URI. You can find it in Aiven’s dashboard.

Import mapping

The process of defining how a document and the fields are stored and indexed is called mapping. When no data structure is specified, we rely on OpenSearch to automatically detect the fields using dynamic mapping. However, we can set our data mapping before the data is sent:

elasticdump \
--input=INPUT_SERVICE_URI/INPUT_INDEX_NAME \
--output=SERVICE_URI/OUTPUT_INDEX_NAME \
--type=mapping

Import data

This is how you can copy your index data from an OpenSearch cluster (can be in Aiven or elsewhere) to an Aiven for OpenSearch one.

elasticdump \
--input=INPUT_SERVICE_URI/INPUT_INDEX_NAME \
--output=SERVICE_URI/OUTPUT_INDEX_NAME \
--type=data

When the dump is completed, you can check that the index is available in the OpenSearch service you send it to. You will be able to find it under the Indexes tab in your Aiven console.

Copying data from Aiven for OpenSearch to AWS S3

Prerequisites

  • Aiven for OpenSearch cluster as the input

  • AWS S3 bucket as the output

You need to collect the following information about your Aiven for OpenSearch cluster and your AWS service:

Aiven for OpenSearch:

  • SERVICE_URI: OpenSearch service URI, which you can find in Aiven’s dashboard.

  • INPUT_INDEX_NAME: the index that you aim to copy from your input source.

AWS S3:

  • AWS credentials (ACCESS_KEY_ID and SECRET_ACCESS_KEY).

  • S3 file path. This includes bucket name and file name, for e.g. s3://${BUCKET_NAME}/${FILE_NAME}.json

See also

You can find more information about AWS credentials in the AWS documentation.

Export OpenSearch index data to S3

Use elasticsearch-dump command to copy the data from your Aiven for OpenSearch cluster to your AWS S3 bucket. Use your Aiven for OpenSearch SERVICE_URI for the input . For the output, choose an AWS S3 file path including the file name that you want for your document.

elasticdump \
--s3AccessKeyId "${ACCESS_KEY_ID}" \
--s3SecretAccessKey "${SECRET_ACCESS_KEY}" \
--input=SERVICE_URI/INPUT_INDEX_NAME --output "s3://${BUCKET_NAME}/${FILE_NAME}.json"

Resources

Aiven for OpenSearch databases are automatically backed up, so you can check more information about how the Backup process works.


Elasticsearch is a trademark of Elasticsearch B.V., registered in the U.S. and in other countries.