Using Spark Structured Streaming to upsert Kafka messages into a database

I wrote a detailed and technical blog post demonstrating an integration of Spark Structured Streaming with Apache Kafka messages and Snowflake.

An overview of the content is:

• querying Twitter API for realtime tweets
• setting up a Kafka server
• producing messages with Kafka
• consuming and parsing Kafka messages with Spark Structured Streaming
• explanation of the streaming model of Spark Structured Streaming
• upserting latest data to Snowflake

You can find the full blog post here.

A small preview: