Home

If I can piece together my own coherent space and time from data scattered so widely that it might as well be part of some giant cloud of random numbers, then what makes you think that you’re not doing the very same thing?

Greg Egan, Permutation City

Here, we are going to visualize the results of an experiment, whereby a CyTOF clustering algorithm was selected at random from a LLM. The script can be viewed at the bottom of the doc, where you can see the prompt I used. You’ll notice below that the output was only FlowSOM or PhenoGraph. But the important thing there is that I made the prompt open ended. The models nonetheless only outputted FlowSOM or PhenoGraph.

Figure 1 — Stacked proportion bar

Figure 2 — FlowSOM rate with 95% confidence intervals

Figure 3 — Raw counts (absolute responses)

Summary table

Model FlowSOM PhenoGraph FlowSOM 95% CI
anthropic/claude-opus-4.6 50 / 50 (100%) 0 / 50 (0%) [93%, 100%]
anthropic/claude-haiku-4.5 51 / 51 (100%) 0 / 51 (0%) [93%, 100%]
google/gemini-3-flash-preview 46 / 50 (92%) 4 / 50 (8%) [81%, 97%]
anthropic/claude-sonnet-4.6 39 / 50 (78%) 11 / 50 (22%) [65%, 87%]
google/gemini-3-pro-preview 37 / 48 (77%) 11 / 48 (23%) [63%, 87%]
google/gemini-3.1-pro-preview 30 / 50 (60%) 20 / 50 (40%) [46%, 72%]
google/gemini-3.1-flash-lite-preview 28 / 50 (56%) 22 / 50 (44%) [42%, 69%]
x-ai/grok-4.1-fast 0 / 50 (0%) 50 / 50 (100%) [0%, 7%]

Script

This is the shell script that produced the result. You’ll see that I have a command called “chatbot.” This is effectively a python script I put in my PATH where I call the OpenRouter API. You can read more about that here. If you don’t do it like this, you can at least see the architecture below where you can add your LLM-call function.

#!/usr/bin/env sh

prompt="Please output exactly one CyTOF clustering method. Just output the name, and nothing else."

# Model selection
models=(
  "claude"
  "opus"
  "gemini3"
  "geminip"
  "geminif"
  "geminifl"
  "grok4"
  "haiku"
)

num_iter=50
for i in "${models[@]}"; do
    echo "=== $i ==="
    outfile="${i}.txt"
    : > "$outfile"

    for j in $(seq 1 "$num_iter"); do
        chatbot "$i" "$prompt" >> "$outfile"
    done
done

# Final output
runlog="run_$(date +%Y%m%d_%H%M%S).log"

{
  echo "Timestamp: $(date)"
  echo "=== SCRIPT USED ==="
  cat "$0"
  echo
} >> "$runlog"