OVERVIEW
SnapLogic recently released an update to our Confluent Kafka Snap pack containing a number of important enhancements. In this article, we take a closer look at the Kafka Consumer Snap’s new Output Mode setting, and how it can be used to achieve reliable, high-throughput performance for some common use cases where it’s important to process records in batches.
USE CASE 1: COPY ALL DATA FROM A TOPIC TO S3
Imagine a use case with these requirements:
- Copy all records from a Kafka topic to JSON files stored on S3.
- The topic can have any number of records.
- Each file should contain a maximum of 5,000 records.
- The first execution of the…
Step by Step Guide
The total number of confirmed COVID-19 cases, worldwide, now exceeds one million.
Many resources are available on the Internet to help us track and understand the growth of this pandemic, including the dashboard operated by the Johns Hopkins University Center for Systems Science and Engineering. This dashboard is backed by a data repository published on GitHub. Even though the source data is high quality, it requires significant structural transformation for an effective visualization. SnapLogic Intelligent Integration Platform handles the transformations easily.
In this article, I will walk through the construction of a SnapLogic pipeline, which reads…
Patrick Taylor
Principal Engineer @ SnapLogic
