Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to content

seanpquig/confluent-platform-spark-streaming

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

confluent-platform-spark-streaming

Example of consuming Avro data encoded with Confluent serialziers from Kafka with Spark Streaming.

Build it

$ sbt assembly

Running it

Export env variables for required configuration.

$ export KAFKA_TOPIC=example_topic
$ export BOOTSTRAP_SERVERS=example.server1.net:9092,example.server2.net:9092,example.server3.net:9092
$ export SCHEMA_REGISTRY_URL=http://kafka-schema.registry.net:8081

Spark submit it

$ spark-submit --class example.StreamingJob --driver-java-options "-Dconfig.file=conf/application.conf -Dlog4j.configuration=file:conf/log4j.properties" target/scala-2.10/StreamingExample-assembly-1.0.jar

About

Working example of consuming Avro data from Kafka with Spark Streaming

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages