Using Kafka for Real-Time Data Ingestion with .NET


Remember that first time you saw Service Broker and thought of all the great things you could do with it? Remember when you actually tried Service Broker and found all the limitations and pain points? Enter Apache Kafka. Apache Kafka is a distributed publish-subscribe messaging system which can scale out to handle millions of messages per second and support a distributed, microservices-oriented architecture. In this talk, we will cover the basics of this powerful system, including general architectural and design principles. We will also build a .NET project which uses Kafka to produce and consume messages quickly and efficiently.


I presented a version of this talk at NDC Oslo in 2018. You can get the recording on the NDC Youtube channel.


Click here to access demo code for this presentation. This includes all of the console applications and Kafka scripts to work with topics.

To get the flight data we loaded and the airport data we used to enrich our data sets, check out the Data expo '09 contest's set of flight data, including the 2008 flight data and airports CSV file

The source code is licensed under the terms offered by the GPL.