Research · Preprint · September 2026

Semantic Drift Across Qwen Generations

Reproducibility, Thinking Mode, Serving Stack, and What Cosine Similarity Misses in Iterated Paraphrase Chains

James H. Smith · Joshua8.AI · DOI 10.5281/zenodo.22646931 · CC BY 4.0

Download PDF (1.8 MB) View on Zenodo Data & Code on GitHub

Abstract

We extend the telephone-game study of semantic drift in iterated LLM paraphrase chains to the Qwen3.6 and Qwen3.8 model generations, and we use the fast, cheap inference these models afford to ask questions the earlier work could not: whether a sampling seed reproduces a chain at all, what reasoning (“thinking”) mode does to drift, whether speculative decoding changes it, how prompts and drift behave across sixteen knowledge domains, and whether embedding similarity measures the loss of information that practitioners care about. Across 2,717 completed chains and 87,110 model calls on eight configurations served by vLLM, SGLang and llama.cpp, we find:

  1. cosine similarity to the original, the drift measure used by this line of work, does not track the information a document carries: across sixteen domains with ten checkable facts each it is only weakly related to the survival of those facts (ρ = 0.32), and on construction statements of work it is negatively related (ρ = −0.42), because outputs inflate two- to seven-fold and reframe as memos while every number survives; dates are the most fragile fact type (35% survival, on six date anchors) and identifiers the most robust (81%);
  2. seeds do not pin chains on any of these stacks — identical requests diverge at the first token even at temperature 0 and even with one request in flight — so drift must be reported over samples, and we do so at 15 to 30 chains per condition;
  3. thinking mode is a large effect whose sign depends on the model, costing Qwen3.6-35B up to 0.16 in final similarity and never reaching a fixed point over 100 iterations, while lifting Qwen3.8-27B from 0.58 to 0.88 on the short stimulus; no system prompt rescues the model it hurts, and it retains the runaway-thinking failure of the previous generation at a low rate, which a presence penalty of 1.5 in the sampler removed in the one condition tested without changing the output otherwise;
  4. multi-token-prediction speculative decoding shows no mean shift we can detect at 15 samples, and temperature 0 narrows the spread without moving the mean;
  5. system-prompt rankings replicate across hosts (ρ = 0.95–0.98) and transfer across models within this generation (ρ = 0.70–0.90), where the earlier work found no transfer across families or sizes, and one domain-neutral fidelity prompt raises every one of sixteen domains to 0.88–0.98; and
  6. most chains lock byte-for-byte within ten iterations, so “drift” on these models is largely where a copying attractor caught the chain.

We conclude that similarity-based drift scores measure genre and length change far more than information loss, and we release the full chain texts so that other metrics can be applied.

Read the Paper

Your browser cannot display the PDF inline.

Download the PDF

Citation

@misc{smith2026generations,
  author       = {Smith, James H.},
  title        = {Semantic Drift Across {Qwen} Generations:
                  Reproducibility, Thinking Mode, Serving Stack, and What
                  Cosine Similarity Misses in Iterated Paraphrase Chains},
  year         = {2026},
  howpublished = {Preprint, Zenodo},
  doi          = {10.5281/zenodo.22646931},
  url          = {https://doi.org/10.5281/zenodo.22646931}
}

Companion paper: Semantic Drift in Iterated LLM Paraphrase Chains. The full chain texts and analysis code are archived as a Zenodo dataset.