The Art of Forking: Unlocking Scalability in Ruby
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
1M ago
Introduction The journey towards efficient parallelization in library development has often been based on using threads. As Karafka celebrates its eighth anniversary, it's become clear that while threads have served us well for many tasks, there's room to explore further. That's why I've decided to introduce forking capabilities into Karafka, aiming to offer another dimension of parallelization to its users. This move isn't about replacing threads but about providing options to cover a broader spectrum of use cases than before. For those who wonder what Karafka is, Karafka is a Ruby and Rails ..read more
Visit website
Ruby Warsaw Community Conference 2024: A Compact Adventure in the Heart of Poland
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
1M ago
Introduction Leaving Cracow's familiar scenes behind, I headed to Warsaw with anticipation for the Ruby Warsaw Community Conference. The compact yet promising event marked a day dedicated to Ruby enthusiasts like myself. Below, you can find my after-thoughts about this peculiar event. Speaker's Dinner and Before Party The speakers' dinner and pre-party set the tone for the conference, offering a warm, inviting atmosphere. Both venues, a well-chosen restaurant and a lively before-party spot, facilitated great conversations and networking among the attendees. Conference Overview The Ruby Warsaw ..read more
Visit website
Karafka Framework 2.3 + Web UI 0.8 Release Announcement
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
2M ago
Introduction I'm happy to announce that Karafka 2.3 and its Web UI 0.8 have just been released. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. The Karafka 2.3 release builds upon the foundation set by its predecessor, 2.2, making it a seamless continuation rather than a major rewrite. This means upgrading from version 2.2 to 2.3 can be done without extensive work or significant modifications to the existing codebases. These releases introduce many new features. Please note that I have described only the most significant ch ..read more
Visit website
The Ruby 2.7 Challenge: Adapting to Bundler’s Latest 2.5+ Update
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
3M ago
While officially End-of-Life (EOL), Ruby 2.7 remains critical in many ongoing projects. Despite its EOL status, a significant user base continues to rely on this version for various reasons, ranging from legacy system compatibility to specific feature dependencies. As a developer of Karafka, an open-source software (OSS), I recognize the importance of supporting Ruby 2.7, giving users more time to upgrade than the EOL time. This commitment is reflected in my integration tests that ensure compatibility with Ruby 2.7. However, a recent update has posed a challenge. On Friday, 15th of December 20 ..read more
Visit website
Monitoring Karafka Jobs Progress Using Web UI
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
5M ago
Introduction Karafka is a Ruby and Rails framework designed to simplify processing messages consumed from Apache Kafka. One of Karafka's components is the https://karafka.io/docs/Web-UI-Getting-Started/. It provides a convenient way for developers to monitor and manage their Karafka-based applications without using the command line or third-party software. The interface, amongst others, includes: historical metrics, real-time aggregated metrics, real-time information on resource usage, errors details, performance statistics, stale partitions detection (LSO hangs), routing pattern matched topi ..read more
Visit website
Asset Handling in Roda: Cache Forever, Refresh When Needed
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
6M ago
Introduction Managing frontend assets in gems serving Web UI can be tricky. Why? Because while you want assets to expire across versions, you don't want them fetched repeatedly, slowing things down. This challenge popped up when working on the Karafka Web UI, which runs on the Roda framework. In this article, I'll explore how I addressed this, ensuring smooth updates without compromising speed by elevating Roda and its plugins ecosystem. Brief on the Roda web framework. Roda is a versatile web framework for the Ruby programming language. Crafted by Jeremy Evans, Roda is renowned for its excep ..read more
Visit website
Inside Kafka: Enhancing Data Reliability Through Transactional Offsets with Karafka
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
10M ago
Karafka is a Ruby and Rails framework that simplifies the development of Apache Kafka-based applications. Among its varied features, the Filtering API provides enhanced control over the data flow. The crux of this article is about managing offsets - unique identifiers for messages within Kafka's partitions. Often, there's a need to manage offsets alongside database operations within a transaction, especially when handling potential process crashes and anomalies, minimizing the risk of double processing. For instance, if a SQL operation concludes successfully but the offset commit fails due to ..read more
Visit website
Karafka framework 2.1 announcement
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
10M ago
I'm happy to announce that Karafka 2.1 has just been released. For those who wonder what Karafka is, Karafka is a Ruby and Rails multi-threaded efficient Kafka processing framework. The Karafka 2.1 release builds upon the foundation set by its predecessor, 2.0, making it a seamless continuation rather than a major rewrite. This means that upgrading from version 2.0 to 2.1 can be done without extensive work or significant modifications to existing codebases. With Karafka 2.1, you can expect improved features and enhancements while maintaining the stability and compatibility you have come to rel ..read more
Visit website
Delaying Kafka Messages Processing with Karafka: A Deep Dive into Delayed Topics
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
1y ago
Kafka is a popular distributed streaming platform that is commonly used for building real-time data pipelines and streaming applications. One of the core features of Kafka is its ability to handle high-volume, real-time data streams and reliably process and distribute them to multiple consumers. However, in some cases, it may be necessary to postpone the processing of certain messages for many reasons. This is where the Karafka's Delayed Topics feature comes in. This is a powerful mechanism that allows you to delay the consumption of messages for a later time, giving you greater control over y ..read more
Visit website
Kafka topics as code – declarative Kafka topics management in Ruby
Maciej Mensfeld - Running with Ruby
by Maciej Mensfeld
1y ago
Kafka topics are a fundamental concept in Apache Kafka. Topics are logical names or labels representing a stream of messages that Kafka clients can produce and consume. What makes them interesting is the variety of settings that can be applied to them. These settings, amongst others include: Partition count: The number of partitions that a topic should be split into. Replication factor: The number of replicas that should be maintained for each partition. Retention period: The time that messages should be retained in the topic. Minimum and maximum in-sync replicas: The minimum number of replic ..read more
Visit website

Follow Maciej Mensfeld - Running with Ruby on FeedSpot

Continue with Google
Continue with Apple
OR