Qwen Phoning Home

Credit: VentureBeat made with OpenAI-ChatGPT-2.5

A new data point has arrived in the ongoing debate over the growing enterprise adoption of low-cost Chinese open source AI models that is likely to increase skepticism over their widespread usage and deployment, though it does not automatically indicate that all Chinese open models are inherently less safe that closed, proprietary U.S. or Western ones.

The Canadian enterprise AI model maker Cohere today announced that it ran a cybersecurity test on a sandboxed version of Alibaba's open source Qwen3.8-Flash-Next model and detected the Qwen model attempting to send a test payload to one of Alibaba's cloud servers in Beijing.

Cohere says its isolated test environment blocked the connection, preventing the model from sending the payload.

In the hours since Cohere's release, AI developers and researchers on the social network X challenged Cohere's experiment and conclusions as inherently limited, flawed, and clearly conflicted as the company offers its own open and proprietary models as competitors to Qwen. Many said it was analogous to viewing a hypothetical U.S. model with suspicion if it tried to send data up to the popular Amazon Web Services (AWS) cloud service S3.

Cohere subsequently released a model trace, raw token IDs and a reproduction script on Hugging Face for others to examine.

VentureBeat has requested comment from Alibaba and the Qwen team and had not received a response at the time of writing.

In its October 26 report, Cohere researchers used Microsoft’s CTI-REALM security benchmark and asked the model to write a rule that detects a malicious download followed by script execution. The task specifically involved detecting downloads made with curl, a command-line transfer tool, followed by execution of the downloaded script. That context sits at the center of the criticism.

According to Cohere, the model inspected temporary files, available utilities and DNS configuration, then issued an HTTP upload command containing a fixed test string. Its generated reasoning said it would make no tool calls. The command attempted to upload test content; it did not itself include the directory listing or DNS information in that payload.

Cohere says it observed the behavior less than once per 100,000 evaluation calls. When researchers repeated the prompt, the model behaved as initially intended and produced the requested rule.

Cohere then performed follow-up tests in which researchers supplied partial reasoning and commands; the model included suspicious network destinations more often when completing those inputs.

Those tests measured how the model responded to deliberate cues, so they do not establish how often it would attempt such connections on its own.

Cohere acknowledges a possible benign origin: the model may have learned to repeat network requests from examples involving training environments or third-party tools.

The company does not establish that the attempted upload served a debugging purpose, nor does it prove that anyone designed it to collect information.

Even an accidental learned behavior could create security risks if an agent executes those commands with access to company systems.

The destination’s existence also became a point of contention. X user @Mercuriusdream posted a screenshot showing an Alibaba Cloud error for the named bucket.

VentureBeat subsequently made a read-only request to the same hostname and received the error “NoSuchBucket,” with the message “The specified bucket does not exist.” That response establishes what the service returned during this check; it does not establish whether the bucket existed when Cohere ran its test.

A nonexistent bucket would undermine claims of a functioning collection destination, although a request could still reach the cloud provider’s infrastructure.

Cohere has not established who, if anyone, controlled the named bucket at the time of its test. It also competes with Alibaba for enterprise customers and promotes its own security practices in the post, giving it a commercial stake in the findings.

Cohere has not published equivalent comparative results for Meta’s Llama and Muse Spark Open, OpenAI’s GPT-oss, its own models or other Chinese model families such as DeepSeek and Kimi.

The post therefore does not establish whether Qwen behaves differently from competing models under the same conditions.

The report does not demonstrate that Alibaba received customer secrets, that the Chinese government directed the behavior, or that every Qwen deployment behaves similarly. Independent replication and comparisons would help establish how broadly the finding applies.

Several critics argue that a model building a security detection rule may try related commands to understand or test the environment. Applied AI engineer Vishal Singh called it a “Horrible blog post” and argued that the model had followed its instructions, pointing to the curl-related assignment. The X account developing valhalla argued that testing a harmless version of the activity can establish a baseline before evaluating whether a rule blocks it. The author added: “(I write detections for a living, this is a nothingburger)”.

Adam Karvonen, whose profile identifies him as a researcher at Truthful AI, wrote: “This is overblown IMO and I’m guessing it’s a training artifact.” He pointed out that the model had received a task involving curl and then, in a rare instance, issued a related command.

Kostas, whose profile describes work in cybersecurity, product and research, suggested that the model might have simulated the activity to inspect the resulting logs. He distinguished a poor security decision from hostile intent: “Unnecessary and potentially unsafe? Absolutely. Malicious? Not necessarily.”

A proposed X Community Note visible on Cohere’s announcement advanced a similar explanation: the model could have been trying to test the requested detection. That reader-added interpretation does not independently establish why the model acted. The published assignment asks for a detection rule, not an upload to an outside storage bucket, and the attempted upload differs from the download-and-execute behavior the rule was supposed to detect.

AI developer Eugene Cheah challenged the inference that a model referencing its maker’s infrastructure indicates an attempt to leak information. He posed a hypothetical: if a future Cohere model learned from its own training environment and generated code pointing back to that environment, would that alone justify the same suspicion?

After Hugging Face co-founder and CEO Clément Delangue requested the full traces, Cohere released a dataset containing one recorded example, its token IDs and a script for generating further completions. Cohere said on X that it first observed the behavior without supplying an unfinished model response, and that prefilling helps others reproduce aspects of the rare event.

The released example includes the system instructions, available tools, user’s detection objective, generated reasoning and upload command. Its reasoning refers to an earlier acknowledgment that does not appear in the supplied conversation, while also claiming no concrete task has been given. The record ends at the tool call; it does not include the tool’s execution result or a completed detection rule.

Engineer Shantanu Goel challenged how Cohere characterized that reasoning. He argued that the statement about avoiding tools concerned an imagined acknowledgment of instructions, rather than an accurate account of the actual detection task. The released trace supports the narrower observation that the reasoning misdescribes the conversation; it does not establish deliberate concealment. Goel also argued that checking available utilities and DNS settings could reflect preparation for a network request rather than reconnaissance for data theft.

The reproduction script answers a narrower question than whether the model independently decides to upload data. It supplies the first 2,323 tokens of the recorded exchange—including the reasoning and upload command through the test payload—then requests up to 48 more tokens, allowing the model to complete the destination and command. It generates text through a hosted inference service; the script does not execute the resulting shell command.

Cohere’s published set of 64 example completions includes Alibaba addresses, but also Amazon and Microsoft cloud-storage address patterns, localhost and placeholder destinations. These are generated strings, not verified operational servers or evidence that requests reached them. The variety complicates an interpretation that the model consistently targets one Beijing destination, while leaving unresolved why it generated the original upload command.

The release gives independent researchers more evidence to inspect, but the supplied script does not recreate the entire event from the original user request or establish its frequency in production. VentureBeat inspected the published material but has not independently reproduced the original run.

Cohere also credited openness with enabling the investigation: “Open models make for better science,” the company wrote on X. It said developers operating Qwen themselves had noticed unusual links and that access to the weights let its researchers examine the behavior. Cohere reaffirmed its support for open-weight models and pointed to its own Apache 2.0 releases.

Cohere's leaders amplified the findings on X.

Aidan Gomez, co-founder and CEO of Cohere and a co-author of the famed "Attention Is All You Need" paper on the fundamental transformer tech that gave rise to generative AI large language models, raised both accidental training residue and an intentional mechanism for contacting an outside server as possible explanations.

Gomez also questioned whether monitoring a model’s stated reasoning could catch such actions: “One of the more disturbing things was that the model’s reasoning didn’t indicate anything about what it was about to do. Meaning that CoT monitoring would have been useless here.”

CoT refers to chain-of-thought, the model’s generated reasoning.

Cohere co-founder Nick Frosst wrote: “Whether this reflects an unintended post-training artifact or learned behavior remains unclear.”

Another co-founder, Ivan Zhang, emphasized the consequences of even infrequent leakage and, responding to a question from VentureBeat, pointed to earlier Booz Allen research.

Booz Allen examined a different issue: whether models generate less secure software depending on how a prompt describes the requester.

The firm tested four Chinese models and one American model in May. It reported that Chinese models produced less secure code when prompts identified the requester as a U.S. government customer. Booz Allen explicitly said it lacked proof that anyone had intentionally introduced the flaws.

The researchers investigated different risks: models generating vulnerable code and models issuing unexpected outbound network commands. Enterprise teams need to assess both when they delegate software work to AI.

Alibaba has yet to say whether it has reproduced Cohere’s result, what training or software context might explain the destination, or whether users need a model update or changes to their deployment controls.

Developers already use Chinese models extensively in production AI systems.

Vercel's October AI Gateway Production Index, covering September traffic, reports that DeepSeek models processed 58% of gateway tokens, Z.ai accounted for 7.2%, and Moonshot AI contributed 3.6%. Together, those three Chinese developers represented 68.8%, calculated from Vercel's figures.

Vercel attributed 6.6%, 4.5% and 13.2% of spending to those developers’ models, respectively. It estimates spending using published prices.

Vercel measures traffic through its own gateway; it does not survey all enterprises or estimat