Atlassian introduced Lithium - an in-house #ETL platform designed to meet the requirements of dynamic data movement.

Lithium simplifies cloud migrations, scheduled backups, and in-flight data validations with ephemeral pipelines and tenant-level isolation - ensuring efficiency, scalability & cost savings.

InfoQ spoke with Niraj Mishra, Principal Engineer at Atlassian, about Lithium’s implementation and future.

Read more here: https://bit.ly/415RPYZ

#DataPipelines #KafkaStreams #ApacheKafka #ApacheFlink #SoftwareArchitecture

Как в Sidec благодаря exactly-once сократили потребление ресурсов без потери производительности

Меня зовут Сергей Гребенюк, я лидер разработки Sidec ( Росреестр ). Расскажу, как решили задачу объединения двух топиков с соотношением один ко многим и почему не устроило решение на Kafka-streams ( kafka docs ) и RocksDB ( github ). А также о том, как, опираясь на гарантии доставки exactly-once (EOS) ( confluent docs ), смогли снизить требования к ресурсам в несколько раз. На иллюстрации показаны два подхода к объединению топиков: с persistent cache и in-memory cache. Мы перейдём от первой схемы ко второй.

https://habr.com/ru/companies/sberbank/articles/863688/

Kafka Streams In Action - Baue deine erste Streaming-App

In seinem Vortrag auf der #BaselOne24 zeigt Thomas Müller, wie man mit der hashtag#Kafka-Streaming-API eine leistungsfähige Streaming-Anwendung erstellen kann – und das auf einfache Weise!
https://lnkd.in/ggjmzerN

Wir freuen uns auf dich!

When talking about Typestream, I often get the question "how does it compare to Kafka Streams? Can you build microservices with it?"

My latest article answers these questions: https://lucapette.me/writing/event-driven-microservices-a-comparative-analysis-of-kafka-streams-and-typestream/

Meet Amazon MSK Replicator - a new feature of Amazon Managed Streaming for #ApacheKafka.

MSK Replicator allows customers to reliably replicate data across Amazon MSK clusters in different or the same AWS region(s) in a few clicks.

To learn more about it, read #InfoQ! https://bit.ly/3tXtlD7

It’s fun to work with talented people who care! Thanks for brainstorming ChrisE.

And “thanks” again for breaking #KafkaStreams

For anybody wondering: https://issues.apache.org/jira/browse/KAFKA-15259

Seems #KafkaStreams is on recently!

- KIP-960: IQ for versioned-stores
- KIP-955: stream-table FK-join ()
- KIP-954: custom stores for DSL default store config
- KIP-941: allow `null` for IQ range queries
- KIP-930: add grace-period to stream-table join
- KIP-925: rack aware task assignment
- KIP-892: TX stores
- KIP-759: repartition canceling

Way to go!

Having fun with AVRO schemas evolution early in the morning... Not even ironic. Its 7:47 and I think AVRO schema evolution checks by schema registry already saved the day!
#kafka #kafkastreams #avro

#CaseStudy - Cloudflare has generated over 1 trillion messages to #ApacheKafka for inter-service communication in less than six years!

Discover real incidents faced in recent years due to scalability limitations and the steps and patterns applied to deal with the ever-increasing demand: https://bit.ly/43yEp5T

Did you know that there will be a Meetup Hub at #KafkaSummit next week?

I am going to give an introduction to #KafkaStreams. If you are new to Kafka and stream processing, hope to see you there!

After all the toots about #RTASummit I am getting really excited about #KafkaSummit London. My first in-person conference after 3.5 years

Btw: Did you know that almost everyone from Confluent’s #KafkaStreams engineering team will be there? Hope you’ll hit us up!

While I am already excited about #KafkaSummit London, I can now also get excited about #Current2023.

I am going to talk about, you guessed it, #KafkaStreams

"The Nuts and Bolts of Kafka Streams---An Architectural Deep Dive"

Hope to see you there. Registration is open.

@ApacheKafka 3.5 feature freeze deadline passed. The community landed 6 KIPs for #KafkaStreams.

- KIP-399: extend production-exception handler to cover serialization errors
- KIP-889 + KIP-914: add versioned state stores and use them in the DSL
- KIP-844: add config for KafkaClientSupplier
- KIP-904: improve adder/subtractor for table-aggregations
- KIP-907: add BooleanSerde

Thanks and congrats for landing all this to all contributors!

Brainstorming topics for #Current23 talks. Anything you always wanted to know about #KafkaStreams (or @ApacheKafka in general)?

CfP closes soon (3/20) so it’s time to start working on abstracts.

So Confluent is acquiring Immerok and diving into the world of #Flink. What does that mean for #KafkaStreams?

@jbfletch explains:
https://twitter.com/jbfletch_/status/1611493927470735362?s=20&t=CdcyN5Qn-aYEBSkC2Ley6w

Dear @ApacheKafka community! Happy holidays! On to new adventures in 2023! #KafkaStreams