Tutorial: Viewing Kafka Messages and Integrating with AWS S3 on the Data Lake Project

Published: Mar 16, 2024 by

Prerequisites:

  • Access to the OpenShift environment.
  • Familiarity with Kafka, Camel K, and AWS S3.

Step 1: Access Kafka Messages

  1. Open your OpenShift dashboard.
  2. Navigate to the edge-datalake project route.

  3. Click on the route for kafdrop.

    Kafdrop route

  4. Once inside, click on the kafdrop route to view the messages.

    Kafdrop messages

  5. To view specific messages, click on a ship name, such as Ship name.

    Ship messages

    Ship messages details

  6. Explore other ships within Kafka as needed.

Step 2: View Camel-K Integration

  1. Within the edge-datalake project, click on the camel-k operator.

    Camel-K operator

    Camel-K details

  2. Review the code for kafka-to-s3-integration-olympic.

    Kafka to S3 integration

  3. Navigate to Resources and click on Pod.

    Resources Pod

  4. View the logs within the selected pod.

    Pod logs

Step 3: Access AWS Console and View S3 Bucket

  1. Log in to your AWS console.

  2. Navigate to the S3 service and locate the relevant bucket.

    AWS S3 bucket

    S3 bucket details

  3. View the data within one of the files.

    S3 data

  4. Scroll to the bottom and click on Run SQL query.

    Run SQL query

  5. Review the results of the data.

    SQL results

Step 4: Create an Instance for Data Push

Attempt to create an instance of either QueenMary or Titanic to push data to the S3 bucket. For detailed steps on this, refer to the provided Camel K Ship integration documentation.

By following this tutorial, you should be able to view Kafka messages, integrate with AWS S3, and push data to an S3 bucket using the Data Lake project on OpenShift. Happy exploring!

Share