How we predict what a room will hold

Every promoter answers the same question before they sign anything. Can this artist fill this room, in this city, on this night? Get it right and a 300-cap room in Leeds hosts the act that headlines a festival two summers later. Get it wrong and someone loses a deposit on a half-empty venue.

Most of the industry answers it with gut feel, a Spotify listener count and a phone call to a friend. We built a system that answers it better. It is called Antonello. This is how it works.

The target is not what you would guess

The obvious framing is a regression. Features in, audience size out. That framing is wrong twice, and understanding why is most of the design.

The label is a decision, not a measurement. We do not have "how many people would have come". We have the capacity of rooms this artist has played, and each of those numbers is one some other promoter chose months in advance, under the same uncertainty we are in now. The training signal is other people's judgement, mistakes included.

The output space is also lumpy. No venue holds 1,340 people. Real rooms cluster at 100, 150, 200, 250, 300, 400, 500, 750, 1000, 1200, 1500, 2000 and up. A model that emits 1,340 has told you less than one that emits 1,200, because 1,200 is a room you can book. So the last step snaps the raw number to the nearest real venue size.

Capacity is heavily right-skewed. Thousands of 200-cap rooms, a handful of arenas. We train on log1p(capacity) and invert with expm1, so the model optimises proportional error. Being 100 seats out matters at 200 cap. At 12,000 it does not.

The shape of the system

                   LLM?   what decides
  1  evidence      YES    scrapers, plus an agent that reads the open web
        |
        v
  2  prior          no    a tier ladder, pure Python
        |
        v
  3  model          no    XGBoost over ~40 features
        |
        v
  4  guards         no    RESCUE raises, VETO lowers, GATE abstains
        |
        v
  5  adjudicate     no    a rule engine, no I/O
        |
        v
     publish

One of those five stages is machine learning. Exactly one of them touches an LLM, and it is not the same one. That is deliberate, and the rest of this post is mostly about why.

The model is good at turning streaming and social signal into a number. It is bad at knowing that the 2,000-cap show it just found was a festival slot and not a headline. Everything around it exists to feed it honest inputs and to catch how it goes wrong.

It runs as a Python 3.13 container Lambda with a hard 540-second timeout, invoked asynchronously by a Node orchestrator. That deadline shapes more of the design than any modelling choice.

Stage 1: evidence, and what it costs to get

We gather the artist's gigography from public listings and partner APIs, then resolve a capacity for every show. A listing row often carries no capacity at all, so each one goes through a chain: our own venue database, a scraped-venue table, a known-venues list, then an LLM estimate as a last resort.

The natural implementation is a loop. For each unresolved venue, look it up. On an artist with a long gigography that loop makes a hundred network calls, blows the 540 seconds and the Lambda dies. The cache lives in ephemeral /tmp, so every retry pays full cost and dies the same way. That is a permanent failure loop, and it hits the most-booked artists first.

So every external call sits behind three bounds:

• A call budget. At most N distinct-venue lookups per run, deduplicated by (venue, city), so twelve shows at one venue cost one call.
• A wall-clock sub-budget, itself capped by a deadline shared across stages, so one greedy phase cannot starve the phase after it.
• A circuit breaker on consecutive transient failures.

Two details matter more than they look. A 404 is a definitive answer, "this venue is not in our database", so it is cached, never retried, and counts as a breaker success. Treat it as a failure and the breaker trips on any tour through venues we happen not to know. And when a budget runs out, the run records how many calls were blocked, so a missing capacity is distinguishable from a capacity we never tried to fetch.

That principle runs through the pipeline. An excluded show is always excluded with a reason: no_capacity, headline_range, scaled_below_min, missing_city, future_event. Those reasons reach the UI. When a booker asks why a number is low, the answer has to be reconstructible.

Stage 2: the prior, and why no model touches it

The strongest signal is not streaming. It is what size room this artist last filled, and where. We call that the prior. Computing it involves no model at all.

Shows sort into an evidence-tier ladder, geography first and recency second:

prev_london > recent_past_london > prev_uk > recent_past_uk > old_london
  > prev_intl(>=2) > recent_past_intl(>=2) > old_uk > old_intl > mixed_past

London wins because London is the target market. A recent UK tour beats an old London room. International tiers need at least two shows, because a capacity scaled across markets is noisier than one measured at home.

The ladder encodes a few things a learned ranker would have to rediscover from sparse data.

Two shows are not one show twice. Corroboration counts distinct dates, not rows, or the same gig reported by three scrapers looks like a trend. We collapse duplicates on day :: fuzzy-venue :: city before anything counts them.

Big claims need real backing. A venue above 5,000 capacity may anchor the prior only if it is independently verified, or backed by two distinct high-cap dates with at least one that is not an LLM estimate. Two guessed arenas must never corroborate each other into a stadium-sized prior.

The most recent London show is not always the right one. A rising act's latest London date is often a small support slot while a bigger headline sits in the same window, so when fresh London evidence spans several dates we anchor to the largest fresh room rather than the newest. The reverse holds too. When London evidence is thin and the recent touring pool medians well above it, the touring median wins.

That last pair took two attempts. The first version only corrected downwards, for a stale anchor that was too big. A rising act pinned to a small old London room was never corrected, so we under-predicted exactly the artists worth finding early.

Stage 3: the model

The ensemble is gradient-boosted trees, about 3.8MB of pickled XGBoost, over roughly forty features in four families.

Family	Examples
Audience size	total listeners, streaming followers, follower ratio, country and city listeners, local audience share
Cross-platform reach	Instagram and TikTok creator reach in-country, YouTube city views, SoundCloud followers, platforms with engagement
Proof of intent	Bandsintown and Songkick followers, Shazam cities and tracks, chart placements, editorial playlist counts
Momentum	30-day velocity, acceleration, days since the listener peak

follower_ratio, followers over monthly listeners, is one of the load-bearing ones. Monthly listeners are easy to inflate with one playlist placement or a collaboration. A follow is a deliberate act. A ratio under about 0.05 says the audience is borrowed rather than owned, and borrowed audiences do not buy tickets. Weighting followers over listeners was one of the largest single accuracy gains we made.

Two heads run over the vector, a scale head and a trajectory head, and the trajectory head supplies the number. We keep the scale head's agreement as a diagnostic rather than routing on it. It tells us when two views of the same artist disagree, which is worth knowing on its own.

Stage 4: guards that only move one way

The model over-predicts in patterns. A festival slot reads as a headline show. A co-bill reads as a solo draw. Rather than complicate the model we wrapped it in guards that run on already-cached evidence, with no new network calls, and each guard moves in one direction only.

RESCUE only raises. It recovers a corroborated clean headline venue tier that the recency discount threw away, capped at 1.4x the prior, and it needs at least two corroborating capacities.

VETO only lowers. It caps the final number at 1.3x the 75th percentile of clean London headline capacities, which trims festival over-extrapolation.

GATE changes nothing. It flags a run as review rather than predict when there is no past venue evidence, or when the artist's identity is ambiguous. Two artists sharing a name is common and it wrecks a prediction quietly.

One-way movement is what makes this tractable. Because each guard moves one direction, you can reason about them separately and compose them in any order with no fixed point to solve. On a 72-artist benchmark the three together lifted point accuracy within 25% from roughly 42% to 46%, while cutting over-predictions. The headline gain is modest. The errors it removed were the expensive ones.

GATE is worth its own line. Abstention is an output. A model that says it does not know on a cold-start artist is worth more than one that guesses, and the flag routes that run to a human instead of shipping a number quietly.

Stage 5: a second opinion that cannot overrule

The last stage is a rule engine. Twelve rules, one per file, no I/O and no model. It re-derives a defensible range from the same evidence and compares the model's point against it. Three rules define it:

It never mutates the prediction. The number from the pipeline stays the baseline. This is a labelled second opinion, not a correction.

If it throws, it is omitted. A failure logs a warning. It never fails or delays the baseline write.

A rule with no defensible bound raises a reason, not a range. It is allowed to say it has nothing to add.

Because it is pure we replay it over stored historical records and measure verdict agreement, known-miss capture and false-suspect rate against human review, through the same adapter the live orchestrator uses. The backtest measures what actually runs.

Where the LLM is, and where it is not

Stages 2 through 5 contain no LLM. selective.py, which holds all three guards, imports statistics.quantiles and a config file and nothing else. The rule engine does no I/O at all. Those four stages decide the number, and a number a booker bets money on has to come out the same way twice.

The LLM lives in stage 1, where the job is reading the open web. Enumerating shows our scrapers missed. Estimating a capacity for a venue nobody has in a database. Checking whether a room actually sold out. That work has no deterministic implementation. It is reading, and models read.

An LLM on its own would be useless here. Ask one "how big a room can this artist fill in London" and you get a confident number with no provenance, different every time you ask, drawn from whatever it half-remembers about an artist it may be confusing with a similarly named one. That is not a forecast. It is a vibe with a decimal point.

What makes it work is everything we wrap around it.

  the repair agent gets:                    the repair agent is bound by:

  the algorithm's own INFO log              a deadline, killed by process group
  the raw scraped events                    a dollar budget as the turn bound
  the prior and how it was chosen           a schema it must fill
  a target city and a date                  a cache key over the raw evidence

                        any error => no repair
                        the algorithmic number stands

Four of those deserve spelling out.

It reads our log, not just our data. capture_algo_log() pipes the prior's own INFO lines into the prompt. The agent does not just see which show we anchored to. It sees the reasoning that picked it: which tier won, which shows were excluded and why. Giving a model the log is the difference between asking it to guess and asking it to review.

It fills a schema. The output is a structured verdict, not prose. A parse failure or a schema failure is treated the same as a crash.

It is bounded in time and money. The subprocess is clamped to a deadline shared with the rest of the pipeline and killed by process group if it overruns. The CLI version we run has no turn limit, so the dollar budget is the turn bound instead.

It is deterministic on replay. Every verdict is cached under artist | spotify_id | city | as_of | raw_events_hash | prompt_version. New evidence changes the hash and invalidates the verdict. Set LLM_REPAIR_OFFLINE=1 and the agent is never spawned at all, so a backtest over stored records replays exactly.

And the rule that ties it together: fail open, everywhere. A spawn error, a parse error, a schema error or a timeout all produce the same outcome, which is no repair. The algorithmic number stands. The LLM can improve a prediction. It cannot break one.

That is the whole argument in one sentence. A model with no structure produces confident noise. The same model, given the log of what the system already decided, a schema to answer in, a clock, a budget, and a default that ignores it when it fails, becomes the most useful component in the pipeline. The power is not in the model. It is in the leash.

The confidence interval is not around the number

This is the part I would most want a new engineer to take away.

The band we show is computed around the prior, not around the model point. They are two different estimators. So the model's number can sit outside its own displayed range.

The tempting fix is to widen the band until the point falls inside. We do not do that. Widening invents a confidence interval nobody measured and hides a real disagreement between two estimators, which is the signal a booker most needs. The UI keeps the true band and shows a below range or above range pill saying the two methods disagree.

A number that admits it is unsure beats a number that always looks certain. The exclusion reasons, the abstention gate, the second opinion and the honest band all exist to serve that.

What is worth stealing

Put the deterministic layer where the domain knowledge is. Geography beats recency. Distinct dates beat rows. A guessed capacity cannot corroborate another guess. Those are rules a human can argue with, and arguing with them is how they got better.

Make your guards one-way. Only-raise and only-lower compose. Bidirectional adjustments do not.

Budget every external call three ways, count, wall-clock and a breaker, when you live inside a hard timeout. Make exhaustion observable.

Never drop a row silently. Every exclusion carries a reason to the UI.

Let the model abstain. The prediction you decline to make is free. The confident wrong one costs a promoter a deposit.

We are hiring the person whose whole job is making this better: the models, the pipelines and the data that decide which artist fills which room. Come and find us.