Recently, I came across an interesting scenario: one application had a cron job constantly polling an API for active offers, just to refresh a Redis cache that powered the offer view. So, I started thinking—isn’t there a better way to handle this? Or at least a way to offload such repetitive tasks outside the application itself?
That’s when it hit me: this pattern looks way too similar to the CDC flows we already implement with Kafka Connect JDBC source connectors. So why not apply the same idea to HTTP? After a bit of digging, I found the answer was yes—we can definitely do it. But there’s a catch. The official Confluent HTTP source connector requires a license, and most open-source alternatives are either too complex or don’t quite match the use case.