This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible.
This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
At no point do we claim that it is a bug with the model. We specifically outline how it was a mis-configuration on Cloudflare's part and a bug in SGLang. We name the model in the title because our customers need to know which of our endpoints was affected. In fact, we say "the GLM-5.3 Workers AI model", not "GLM-5.3".
Not buying that. You explicitly mention the model name in the title the 'GLM 5.3 workers AI model' suggests it is something specific to GLM 5.3 which it is not. You mention Sglang and are all surprised that it didn't throw an error. But the issue was yours all along.
Not sure if the title was edited but as is it is very clear to me this is a bug on their side and I have no idea how a technically literate person (someone using the model via cloudflare?) who would misinterpret given the reference to the cache.
I have read this post, and it appears that this is indeed an issue specific to the GLM 5.3 model on their platform. I am unsure whether this is because only GLM 5.3 uses SGLang or because only its configuration is problematic.
I was surprised in reading the article that it had nothing to do with the contents of the model. It could have been any model capable of 1M context that triggered this bug/misconfig on Cloudflare/SGLangs side. The title is extremely misleading.
Which part of that is not claiming the exposure is in X model? Cloudflare is stopping to be cool, this is the start. I guess, cf is now big enough for enshitification to start.
It seems like you're trying to imply this was an intentional bias due to the nationality of the model authors. But the first sentence of the article is:
"On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026."
Meaning the issue was a cache exposure with the GLM-5.3 Workers AI model. If the title was, "Cache exposure issue with GLM-5.3," I would agree it's ambiguous and maybe intentionally harmful. But it literally says, "Workers AI model" which is clearly a Cloudflare product.
I did not mention anybody's nationality, nor did I imply that that was important.
Downthread I suggested "Cloudflare configuration error leaked data between user sessions" which I think is far better. You could improve on it further: "Cloudflare AI worker configuration error leaked data between user sessions". But this could have happened with any model.
I like this RCA a lot more than most. I appreciate the ownership and clear explanation of why their testing did not catch this bug.
Also impressed with the speed to address the issue unplug the model and then pretty quick remediation, the speed to remediation matters less to me, but good to see they can act quickly on their platform to kill the serving of the model.
"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "
Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?
So they knowingly deployed a configuration with an overflow risk, much like writing code with a buffer overflow exploit.
They assumed that it wasn't like an overflow exploit, despite the fact that's exactly what they'd set up to happen, on the grounds of thinking the equivalent of, "oh this probably uses smart pointers, we'll be fine".
Next sentence: "We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue."
Doesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.
why would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.
Sounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success.
I read that as you can only do a single prefill up to 681k tokens, separate from cache hit or generation. I think people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare. If you run a normal coding session with many turns it would get to 1M ok.
Last year it happened to me more than once that dictating something in chatgpt would become a completely different message after processing, not in a misheard type of way but just a separate conversation more easily explained by the same type of KV cache issue here.
I think in the last year I've had it happen once where I seemingly got someone else's response, it was very odd. I rewound and got what I expected. It was correlated to Anthropic's roll out of new models at the time which always seems to cause instability in their apis.
Reminds me of a bug I found in a very popular browser testing service that spun up iOS vm's/devices for on-device testing.
I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other.
I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back.
I kind of forgot about the shared tenancy attack model. I think I was assuming/hoping the software was magically isolating my tokens, when in reality that's impossible for shared tenants. There's whole classes of attacks on things like shared cache, before we get to the LLM-specific features and potential bugs. Enterprises with highly sensitive data should really be on non-shared hosts.
42 comments:
This doesn't really have anything to do with GLM-5.3. This is what happens when you share models between users and don't do your bookkeeping properly, the actual model doesn't matter. What it shows is that there isn't enough isolation between the user sessions and that memory can overlap when that should never be possible.
This is an SGLANG and a Cloudflare bug long before it is a Z.AI model bug, I don't see what they could have done about it. Cloudflare should own it instead of naming other (imo innocent) parties in the title.
At no point do we claim that it is a bug with the model. We specifically outline how it was a mis-configuration on Cloudflare's part and a bug in SGLang. We name the model in the title because our customers need to know which of our endpoints was affected. In fact, we say "the GLM-5.3 Workers AI model", not "GLM-5.3".
Not buying that. You explicitly mention the model name in the title the 'GLM 5.3 workers AI model' suggests it is something specific to GLM 5.3 which it is not. You mention Sglang and are all surprised that it didn't throw an error. But the issue was yours all along.
Yeah, their title is absolutely implying a GLM bug to anyone who doesn't read it super closely.
There's a huge amount of unnecessary extraneous details in the headline that distract from "this is a Cloudflare bug".
Not sure if the title was edited but as is it is very clear to me this is a bug on their side and I have no idea how a technically literate person (someone using the model via cloudflare?) who would misinterpret given the reference to the cache.
I have read this post, and it appears that this is indeed an issue specific to the GLM 5.3 model on their platform. I am unsure whether this is because only GLM 5.3 uses SGLang or because only its configuration is problematic.
I was surprised in reading the article that it had nothing to do with the contents of the model. It could have been any model capable of 1M context that triggered this bug/misconfig on Cloudflare/SGLangs side. The title is extremely misleading.
"Cross-request KV cache exposure in X model".
Which part of that is not claiming the exposure is in X model? Cloudflare is stopping to be cool, this is the start. I guess, cf is now big enough for enshitification to start.
Then remove GLM-5.3 from the title.
It seems like you're trying to imply this was an intentional bias due to the nationality of the model authors. But the first sentence of the article is:
"On September 8, 2026, we were alerted by a security researcher, Gadi Evron from Knostic, to a tenant isolation issue affecting the GLM-5.3 Workers AI model (@cf/zai-org/glm-5.3) starting on August 27, 2026."
Meaning the issue was a cache exposure with the GLM-5.3 Workers AI model. If the title was, "Cache exposure issue with GLM-5.3," I would agree it's ambiguous and maybe intentionally harmful. But it literally says, "Workers AI model" which is clearly a Cloudflare product.
I did not mention anybody's nationality, nor did I imply that that was important.
Downthread I suggested "Cloudflare configuration error leaked data between user sessions" which I think is far better. You could improve on it further: "Cloudflare AI worker configuration error leaked data between user sessions". But this could have happened with any model.
Why are you bringing up people's nationalities? This seems incredibly inappropriate
I like this RCA a lot more than most. I appreciate the ownership and clear explanation of why their testing did not catch this bug.
Also impressed with the speed to address the issue unplug the model and then pretty quick remediation, the speed to remediation matters less to me, but good to see they can act quickly on their platform to kill the serving of the model.
"We deployed GLM 5.3 with a 1,048,576-token context window, but the KV cache pool on our prefill GPUs only had room for 681,280 tokens. We were aware of this mismatch before launch. "
Did they advertise it as 1,048,576 token when they only had room for 681,280 tokens?
So they knowingly deployed a configuration with an overflow risk, much like writing code with a buffer overflow exploit.
They assumed that it wasn't like an overflow exploit, despite the fact that's exactly what they'd set up to happen, on the grounds of thinking the equivalent of, "oh this probably uses smart pointers, we'll be fine".
That's how I read it anyway.
Next sentence: "We expected the engine to return an error for any prompt it could not process, and our long-context tests returned successful responses, so we did not believe this to be an issue."
Which is pointing the finger at Sglang, who also weren't responsible for this.
At a minimum someone should have realized during testing that their utilization figures are off by 40%.
A much better title would be "Cloudflare configuration error leaked data between user sessions"
I mean they are responsible for this. It's definitely a bug on their end.
Doesn't matter, they could have marketed it as 1 million token window, you would then infer, get to 600k and get an error. You won't be able to finish your work, and you won't get credit for 600k tokens.
They thought it worked up to 1m token window
why would they think that? I'm not a pro, but I know you can't serve more than kv cache pool that you have. If you want to run concurrent requests, you need to have more kv cache pool than your context window. The entire thing is weird, my bet is they were thinking most people will be using about 100k cache, compacting, etc and not nearing 600k ever.
Bro, I had to read this sentence in the context of the previous one, 3 fcking times.. and it sill does not make any sense to me.
Sounds like they are saying they tested with a long prompt and just tested for a failure, but the test likely would have seen that same KV cache leakage and just interpreted it as success.
thank you...
I read that as you can only do a single prefill up to 681k tokens, separate from cache hit or generation. I think people try pretty hard to avoid a cache miss on sessions that long and rarely do standalone requests that big so the situation triggering this condition is rare. If you run a normal coding session with many turns it would get to 1M ok.
Other providers probably don't even know/care that this is happening
Last year it happened to me more than once that dictating something in chatgpt would become a completely different message after processing, not in a misheard type of way but just a separate conversation more easily explained by the same type of KV cache issue here.
I think in the last year I've had it happen once where I seemingly got someone else's response, it was very odd. I rewound and got what I expected. It was correlated to Anthropic's roll out of new models at the time which always seems to cause instability in their apis.
I wonder if the chatgpt shirtless guy is from the same kind of issue
Other providers might have better isolated KV stores?
I've had this happen with ChatGPT too.
It seems more specific to the platform than the model. Maybe Cross-request KV cache exposure in the AI model workers would be a better fitting title?
Please consider to update the title in: Cross-request KV cache exposure in the AI model workers
CloudBleed huh?
Interesting. That might explain the few times I've received completely unrelated responses from a certain provider's API.
Upshot: even though sglang is written in Rust, they still got burned by undefined behavior from uninitialized memory.
Reminds me of a bug I found in a very popular browser testing service that spun up iOS vm's/devices for on-device testing.
I copied the clipboard from the device, and my clipboard was full of another users credentials that they were obviously trying to paste into their own VM. Their devices were leaking clipboard data to each other.
I reported the vulnerability and asked if there was any bounty program (I wasn't looking for vulnerabilities but didn't hurt to ask), to which they said no, but they could send me a signed letter thanking me of my contributions. I said, great, thanks - and never heard back.
All pretty disappointing.
Is anyone even offering single tenant inference as a service?
That’s just renting GPU clusters
Is that a new class of tenant isolation vulnerability?
Seems not? Just good old gpu allocations colliding
I kind of forgot about the shared tenancy attack model. I think I was assuming/hoping the software was magically isolating my tokens, when in reality that's impossible for shared tenants. There's whole classes of attacks on things like shared cache, before we get to the LLM-specific features and potential bugs. Enterprises with highly sensitive data should really be on non-shared hosts.