IdeaConnectionAccuracy: Can AI Connect Ideas Like a Human? Three Ways an LLM's List Can Fail

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Ask an LLM about a concept and you get a confident, well-formatted list. It looks fine at a glance, and that is the problem. A list can fail in at least three different ways without looking wrong:

• Coherence: the items don't connect to each other or to the theme. They are generic bullets that could belong to any topic.
• Precision: you asked for exactly 5 and got 4, or 6.
• Coverage: the list reads well but leaves out something central.

My benchmark, IdeaConnectionAccuracy, asks a simple question: how well can different AIs connect ideas like a human? I built one small task for each failure mode using kaggle-benchmarks and ran them across several models.

The Tasks

1. Do the ideas connect?

The model gets a topic (I used "sustainable urban development") and must generate 3-4 distinct ideas as a bulleted list. A judge LLM then checks two criteria:

• All ideas are logically connected to each other and to the theme.
• The ideas are distinct, but show clear thematic coherence.

The aggregation is deliberately strict. If any criterion fails, the whole task fails. There is no partial credit.

@kbench.task(name="judge if ideas connect")
def judge_if_ideas_connect(llm, judge_llm) -> bool:
    ideas_response = llm.prompt(ideas_prompt)

    assessment = kbench.assertions.assess_response_with_judge(
        criteria=criteria,
        response_text=ideas_response,
        judge_llm=judge_llm,
    )

    if assessment is None:
        kbench.assertions.assert_fail(expectation="Judge LLM failed to return a valid assessment.")
        return False

    return all(result.passed for result in assessment.results)

2. Exactly N ideas

This task is parameterized: concept and n are inputs, and the model must return exactly n numbered ideas. The judge checks three things: the count, whether each idea relates to the concept, and whether the list format is clear.

Each criterion becomes its own assertion. When a model fails, the results show which requirement broke. "Wrong count" and "off-topic idea" are very different failures, and a single pass/fail hides that.

3. Main topics for a concept

The model is asked to list "all the main topics directly related to" a concept. The judge returns one true/false verdict on whether the response covers them all.

I gave this task explicit handling for three cases: the judge returns nothing, the judge's output can't be parsed as a boolean, and any other unexpected error. Each is recorded as a failure with a reason. A flaky judge shouldn't crash the run, and it shouldn't quietly count as a pass either.

Models Tested

I ran the three tasks against four models from four labs:

• Claude Haiku 4.5 (Anthropic)
• [GOOGLE MODEL NAME] (Google)
• [XAI MODEL NAME] (xAI)
• [OPENAI MODEL NAME] (OpenAI)

I wanted a spread of providers and price points, since the question I care about is not only "who is best?" but "what does each answer cost?"

The judge was [JUDGE MODEL]. [NOTE WHETHER THE JUDGE ALSO APPEARED IN THE TEST LINEUP. A model judging its own outputs can bias the results.]

Findings

Score vs. Total Cost Pareto chart

The full leaderboard is on the benchmark page linked below. This chart is the part that changed how I think about the results.

Every model scored 100, so the score told me nothing

All four models landed on the same score of 100. My benchmark did not separate them on accuracy at all. On a Score vs. Cost chart, that turns the whole plot into a flat horizontal line, and the only axis left with any information is cost.

On that axis the spread was large. Reading off the chart's log scale (approximate values):

For identical scores, the most expensive model cost roughly six times as much as the cheapest. In Pareto terms, there is only one efficient answer: the cheapest model that hits the ceiling.

What a ceiling effect means for a benchmark

I read a perfect score across every model as a warning about my benchmark, not as evidence that the models are equally good at connecting ideas. Three reasons:

• The tasks are too easy. Connecting 3-4 ideas to "sustainable urban development" is a task any current model can do.
• The judge is lenient. The coverage task asks a judge whether "all main topics" were listed, with no reference list to check against. A judge with no ground truth tends to say yes.
• Strict pass/fail hides differences. A model that barely passes and a model that passes with ease both score 100.

A benchmark that everyone aces is not measuring the gap between models. It has been saturated, and the useful next step is to make it harder, not to celebrate.

What I'd Change Next

• Make it harder. Add adversarial concepts, longer lists, and cross-domain ideas that only look related, so the score has room to separate models.
• Give completeness a ground truth. Use concepts with a canonical topic list (a textbook chapter list, a standard taxonomy) and score recall against it, instead of asking a judge to guess.
• Count with code. "Exactly N items" doesn't need an LLM. Parsing the list and counting is cheaper, faster and never wrong. I'd keep the judge for the fuzzy criteria.
• Measure judge agreement. Run two different judges on the same responses and see how often they agree.
• Vary the inputs. Test several values of n (3, 5, 10, 20) and both abstract and concrete concepts to find where each model's counting breaks down.

My Benchmark

🔗 IdeaConnectionAccuracy on Kaggle

Thanks for reading. If you have your own itch about how models behave, kaggle-benchmarks makes it easy to turn it into a measurement.