I think this is a clever happy path solution, but it isn't very defensive against problems. If the cloud cron doesn't fire on time or the consumer fails then the queue won't be drained in time. That will cause the events to be batched in the wrong hour.
I don't see how you are building the batches, but there's a lot of complexity around batch sizes, acknowledging that the messages were consumed, etc. Having a single dataset.json.gz per hour feels like a hint this isn't done safely. I'd double check that carefully to avoid losing data.
One tip about compression, I've been really pleased with zstd compression of json/jsonl data.
Cool pattern. Critics will say “falsehoods programmers believe about time” but it’s a cloud service, any time oddity that would break this would also break a lot more important stuff, so it’s fine.
I couldn’t tell you if AI code assistants would write this, or catch it - ironically enough, “the perils of programming in plain text” is one of my stated reasons for eschewing AI assistance.
Given that the succinctness just bit me, I’m inclined to sacrifice it and go with
// over the range 0..23, +1 gives the same result%2 as -1, and also handles edge case of 0
const lastHour = new Date().getUTCHours() + 1
…
env.QUEUES[lastHour % 2]
Fundamentally I guess I was just objecting to doing arithmetic in the time domain to achieve negating a boolean.
Cloudflare R2 bills on writes ($4.50/M), reads ($0.36/M), and storage ($0.015/GB/mo). The free tier gives you 1M writes, 10M reads, and 10GB per month. If you batch uploads hourly, you only make ~720 writes a month. That completely eliminates write costs, leaving storage as the only expense.
12 comments:
I think this is a clever happy path solution, but it isn't very defensive against problems. If the cloud cron doesn't fire on time or the consumer fails then the queue won't be drained in time. That will cause the events to be batched in the wrong hour.
I don't see how you are building the batches, but there's a lot of complexity around batch sizes, acknowledging that the messages were consumed, etc. Having a single dataset.json.gz per hour feels like a hint this isn't done safely. I'd double check that carefully to avoid losing data.
One tip about compression, I've been really pleased with zstd compression of json/jsonl data.
[flagged]
- your blog page has errored out
- https://bellone.com/blog
[flagged]
A stateless pattern I use to batch telemetry at the edge while driving Cloudflare write costs to zero.
Cool pattern. Critics will say “falsehoods programmers believe about time” but it’s a cloud service, any time oddity that would break this would also break a lot more important stuff, so it’s fine.
Ugly code though.
> env.QUEUES[1 + hour % 2]
// results in index 1 or 2 as js has precedence for */% over +-
So something like the following, for your desired visual succinctness? ...
env.QUEUES[hour % 2 ^ 1]
// results in desired js index 0 or 1
I wonder if any of you using AI code assistance regularly might tell an old hack like me if your code review tools would have caught such a bug?
Ah, the perils of programming in plain text.
I couldn’t tell you if AI code assistants would write this, or catch it - ironically enough, “the perils of programming in plain text” is one of my stated reasons for eschewing AI assistance.
Given that the succinctness just bit me, I’m inclined to sacrifice it and go with
Fundamentally I guess I was just objecting to doing arithmetic in the time domain to achieve negating a boolean.[flagged]
Can you explain the pricing here?
Cloudflare R2 bills on writes ($4.50/M), reads ($0.36/M), and storage ($0.015/GB/mo). The free tier gives you 1M writes, 10M reads, and 10GB per month. If you batch uploads hourly, you only make ~720 writes a month. That completely eliminates write costs, leaving storage as the only expense.