Define OpenSearch® timestamp data in SQL pipeline

Frequently results in Apache Flink® data pipelines include one or more timestamps, either contained in the source events or generated by window aggregations.

When the output of the Apache Flink® data pipeline is an Aiven for OpenSearch® index, you need to convert the Flink timestamps to a format recognizable by OpenSearch®, otherwise they will be interpreted as strings, losing the benefits of time filtering.

OpenSearch® recognises the following as correct date/time formats:

  • yyyy/MM/dd for a date field

  • HH:mm:ss for a time field

  • yyyy/MM/dd HH:mm:ss for a timestamp field

Therefore you need to structure the data pipeline output to follow one of the acceptable formats