Scylla Student Projects, Part III: Kafka Client for Seastar and Scylla

Intro: Student Projects

  • Parquet support for Seastar and Scylla (see Part I)
  • SeastarFS: an asynchronous userspace file system for Seastar (see Part II)
  • Kafka client for Seastar and Scylla (this post)

Apache Kafka: Core Concepts

Our Goals and Feature Set

  • Future-based API built on top of Seastar
  • High performance
  • Proper error handling and resiliency
  • Retries, with configurable backoff strategies (exponential backoff with jitter as default)
  • Partitioners (multiple implemented out-of-the-box)
  • Batching
  • ACK (configurable number of acknowledgments to wait to receive after sending a message)
  • Compatibility with older Kafka clusters (by supporting many versions of the protocol)
  • Support for multi-broker clusters

Why Seastar?

The Kafka protocol

Handling Errors

Batching

Optimization

Benchmarks

Example Use Case: Scylla CDC Replication Service

Wrapping it up

--

--

--

The monstrously-fast NoSQL database.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Generics Basics

Reduce Cost and Increase Productivity with Value Added IT Services from buzinessware — {link} -

[Book Study] Clean Architecture — Part 1

AWS Certified SysOps Administrator Associate SOA-C02 Exam Questions 2021 Part 5

AWS Certified SysOps Administrator Associate SOA-C02 Exam Questions 2021 Part 5

Warp’s dev log #001

How to Install Hyper Terminal for MacBook

More ways to make side income as a coder

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
ScyllaDB

ScyllaDB

The monstrously-fast NoSQL database.

More from Medium

Evaluating TiDB against the MySQL Test Suite

Setup HTTPs Authentication for Presto / Trino

Load Balancing for gRPC

Distributed System Consensus