If you build a probabilistic model for events that also trade on a prediction market, the market price is your natural baseline, and an unforgiving one. Saying "our model is 60% accurate" means very little; the question that matters is whether your probabilities carry information the price does not.

This walkthrough answers that question end to end on a real dataset. It covers four steps that are worth doing in this order for any forecaster:

1. Score both sides with a proper scoring rule.
2. Decompose the score to see why one side wins.
3. Put an honest confidence interval on the gap.
4. Test whether the model adds anything on top of the price.

The data are model forecasts on Polymarket questions published by NeuPortal (disclosure: that is my project). Each forecast was stored next to the market price at the moment it was made and timestamped into Bitcoin, so none of the numbers can have been edited after the outcome. The dataset is open (CC BY 4.0).

Setup

import numpy as np
import pandas as pd

URL = "https://huggingface.co/datasets/neuportal/forecasts-vs-market/resolve/main/forecasts.csv"
df = pd.read_csv(URL)

# keep resolved questions whose forecast was timestamped before the outcome was known
df = df[(df.status == "final") & (df.sealed_before_resolution.astype(str).str.lower() == "true")]
y = df.outcome.to_numpy()          # 1 = Yes, 0 = No
model = df.our_p.to_numpy()        # model probability of Yes
market = df.market_p_at_call.to_numpy()  # market price at the same moment
print(len(df))                     # 5654

The filter on sealed_before_resolution matters. The dataset also contains an early batch that was timestamped late, after many outcomes were already known. Those rows are flagged and should never be used as evidence of forecasting skill. Whatever data you work with, check when each forecast provably existed before you score it.

Step 1: score both sides

The Brier score is the mean squared error between the probability and the 0/1 outcome. Lower is better; always saying 50% scores 0.25.

def brier(p, y):
    return np.mean((p - y) ** 2)

print(brier(model, y), brier(market, y))   # 0.1959  0.1598

The market wins by a wide margin. That is the headline, but on its own it tells you nothing about what to fix.

Step 2: decompose the score

The Murphy decomposition splits the Brier score into three parts:

• Uncertainty depends only on the outcomes. It is the same for both forecasters.
• Reliability is the calibration error: when you say 30%, does it happen 30% of the time? Lower is better.
• Resolution measures how much your forecasts separate events that happen from events that don't. Higher is better.

Brier ≈ uncertainty − resolution + reliability.

def murphy(p, y, bins=10):
    idx = np.clip((p * bins).astype(int), 0, bins - 1)
    base = y.mean()
    rel = res = 0.0
    for k in range(bins):
        m = idx == k
        if m.any():
            w = m.mean()
            rel += w * (p[m].mean() - y[m].mean()) ** 2
            res += w * (y[m].mean() - base) ** 2
    return rel, res, base * (1 - base)

for name, p in [("model", model), ("market", market)]:
    rel, res, unc = murphy(p, y)
    print(f"{name}: reliability {rel:.4f}  resolution {res:.4f}  uncertainty {unc:.4f}")

This table is more useful than the headline. The market is almost perfectly calibrated, with reliability close to zero. The model is not, but calibration is the smaller half of its problem. Most of the gap is resolution: the market separates Yes from No outcomes more than twice as well. Recalibrating the model, for example with isotonic regression, would close only part of the gap. The model is missing information, not just mislabelling its confidence.

A quick look at the reliability table shows where the calibration error sits:

bins = pd.cut(model, np.linspace(0, 1, 11), include_lowest=True)
print(pd.DataFrame({"p": model, "y": y}).groupby(bins, observed=True).agg(n=("y", "size"), mean_p=("p", "mean"), hit=("y", "mean")))

In this data the worst bucket is the bottom one. When the model said 10% or less, events happened far more often than it claimed. That is a classic sign of a model whose predictive distribution is too narrow.

Step 3: an honest confidence interval

It is tempting to bootstrap the Brier difference over rows. Here that would be wrong, because rows are not independent. Many questions belong to the same event: several temperature bands for one city and day, for instance, all resolve together. Resample events, not rows:

df["d"] = (df.our_p - df.outcome) ** 2 - (df.market_p_at_call - df.outcome) ** 2
g = df.groupby("event").d.agg(["sum", "count"])

rng = np.random.default_rng(0)
boot = []
for _ in range(2000):
    s = g.iloc[rng.integers(0, len(g), len(g))]
    boot.append(s["sum"].sum() / s["count"].sum())
print(df.d.mean(), np.percentile(boot, [2.5, 97.5]))   # 0.0361  [0.0317  0.0407]

The model's excess Brier is 0.036, with a 95% interval of roughly 0.032–0.041 across 1,938 events. The interval does not come close to zero, so this is not noise. Clustering is also what keeps you honest when the gap is small. A row-level bootstrap would report an interval that is too narrow, and that is how "significant" edges are found where none exist.

Step 4: does the model add anything to the price?

A model can lose to the market overall and still be useful, if its errors differ from the market's. The cleanest test is to blend the two and see whether any weight on the model improves on the price alone. Blending in log-odds space is the usual choice for probabilities:

def logit(p):
    p = np.clip(p, 1e-3, 1 - 1e-3)
    return np.log(p / (1 - p))

for w in [0, 0.1, 0.2, 0.3, 0.5]:
    p = 1 / (1 + np.exp(-(w * logit(model) + (1 - w) * logit(market))))
    print(w, round(brier(p, y), 4))

Every positive weight makes things worse. On this sample the model carries no information that the price does not already contain. Run this test before any other. If the optimal weight on your model is zero, no amount of tuning the threshold or position sizing will rescue it.

What to take away

• Score against the price, not against a coin flip. Beating 0.25 is easy; beating a liquid market is the actual bar.
• Decompose before you fix. Here the market won mostly on resolution, which points to missing inputs rather than a calibration patch.
• Bootstrap the unit that is actually independent. In prediction-market data that is usually the event, not the question.
• The blending test is the verdict. If adding your model to the price never helps, the model is not yet adding information.
• Check when each forecast provably existed. Scores computed on forecasts recorded after the outcome measure nothing.

The data and the proof files are public if you want to rerun any of this or try your own model against the same questions:

• Data (CC BY 4.0, DOI): https://doi.org/10.5281/zenodo.24051884
• Timestamp proofs: https://github.com/neuportalai-ui/experiment-proofs

More: the full experiment at https://neuportal.ai/experiment and the write-up 7,426 forecasts against Polymarket: exactly where the market beat our models.

Educational transparency data, not financial advice.

On the 7,048 calls that have resolved, the market scores 0.161 and our models score 0.194. The market wins. This post is about where it wins, because the answer is more specific, and more useful, than "markets are efficient".

Everything below can be reproduced from the public dataset in a few lines of pandas (code at the end).

Headline numbers

Weather is 87% of the sample, so it dominates the total. Without weather the gap is small (0.130 vs 0.123 on 920 calls). The only track where we are ahead is politics, on 23 calls, which proves nothing.

One more number worth keeping in mind: our call scored better than the market price on 50.5% of individual questions. We win about half the coin flips, and lose the total anyway. The losses are bigger than the wins, and the rest of this post is about why.

Finding 1: we were far too sure that unlikely things would not happen

Take every call where our model said 10% or less. There are 2,176 of them.

• our average probability: 3.8%
• the market's average price on the same questions: 16.5%
• how often the event actually happened: 17.0%

The market was almost exactly right; we were off by a factor of four. Most of these are weather "band" questions (will the high temperature land in a given 2-degree range). Our models put a narrow, confident distribution on the temperature and gave neighbouring bands almost nothing. Reality was wider than the model.

The market's own calibration, by contrast, is close to the diagonal everywhere it has data:

And ours:

So the error runs both ways: too low at the bottom, too high in the middle. That is the signature of a model that is overconfident about which band wins, not one that is biased in a single direction.

Finding 2: the more we disagreed with the market, the worse we did

Group the calls by the gap between our probability and the market price:

When we roughly agreed with the price, we tied it. Every step of disagreement cost us, and the large disagreements, the ones that would have looked like "edge", were the worst calls in the set.

Finding 3: mixing our model into the price only made it worse

A natural rescue is to blend: take some weighted average of our probability and the market's. If the model carried any information the price did not, some blend should beat the price alone.

Every amount of our model hurt. On this sample, the honest conclusion is that the models added nothing the price did not already know.

What we are not claiming

• This is one set of models, mostly on one kind of question. It says little about forecasting in general and nothing about other teams.
• Our first batch (calls from 14 to 26 September, 1,613 of them) was sealed late, on 26 September, after 1,394 had already resolved. Those rows are flagged sealed_before_resolution = false and are not evidence of forecasting in advance. They do not flatter us: on the 5,654 resolved calls sealed before their outcome, the result is the same (0.196 vs 0.160). Since 26 September every call is sealed minutes after it is made.
• Nothing here is a trading signal or advice. It is a record of misses, kept so it cannot be edited later.

Reproduce it

import pandas as pd
from datasets import load_dataset

f = load_dataset("neuportal/forecasts-vs-market", split="train").to_pandas()
f = f[f.status == "final"]

# headline
print(f[["brier_model", "brier_market"]].mean())

# finding 1: our 0-10% calls
low = f[f.our_p <= 0.10]
print(len(low), low.our_p.mean(), low.market_p_at_call.mean(), low.outcome.mean())

# finding 2: by disagreement
gap = (f.our_p - f.market_p_at_call).abs()
bins = pd.cut(gap, [0, .05, .1, .2, .3, 1], include_lowest=True)
print(f.groupby(bins)[["brier_model", "brier_market"]].mean())

# finding 3: blends
for w in [0, .25, .5, .75, 1]:
    p = w * f.our_p + (1 - w) * f.market_p_at_call
    print(w, ((p - f.outcome) ** 2).mean())

To check that a call existed when we say it did, open its proof_file in the proofs repository and drop the .ots file, then the .json, on opentimestamps.org. The Bitcoin block it reports is the latest moment that call could have been written.

• Data (DOI, CC BY 4.0): https://doi.org/10.5281/zenodo.24051884
• Proof files: https://github.com/neuportalai-ui/experiment-proofs
• Project page: https://neuportal.ai/experiment

Educational transparency data, not financial advice.

Production is the same code executing at 06:40 on a Sunday, against an input nobody anticipated, with no human in the loop and a customer waiting on the other end of it.

Almost every conversation about earning money with AI skips the distance between those two sentences. That distance is the entire job.

Prompting stopped being the product

The change in the market over the past couple of years is not that models got better. It is that knowing how to prompt one became a baseline expectation instead of a service. Buyers now assume the person across the table can get a reasonable answer out of a model, in the same way they assume you can use a spreadsheet. What they are paying for is somebody who can put that capability inside a workflow that already exists and then point at a business number that moved.

Good news for anyone with an engineering background, bad news for the prompt-library genre. A prompt is a configuration value, and nobody signs a purchase order for a configuration value.

The four properties that make a build billable

Treat these as acceptance criteria. They hold whether the delivery is a support agent, a reporting pipeline or a content system.

It runs unattended. A scheduler or an event source fires it, not a terminal you open. That buys you retries with backoff, a dead-letter path for the inputs that fail every attempt, and idempotency on anything that writes. If your job re-processes the same webhook twice because the receiver got redelivered, you have shipped a machine that double-emails the client's customers. Idempotency keys are not architecture astronautics here - they are the difference between an invoice and an apology.

It degrades safely. The interesting failure is not the crash. A crash is loud and gets fixed. The expensive failure is the one that keeps producing plausible output after the ground truth is gone: a stale cache, an empty retrieval result treated as "no relevant policy", a rate-limited call that silently returns the previous response. Decide in advance what the system does when it does not know, and make that path explicit. Refusing and escalating is a feature. Guessing confidently is the defect.

Somebody owns the failure cases. Not "the model handles it" - a named human, a documented escalation, an alert that reaches somebody who can act. Buyers in any regulated or customer-facing process are partly purchasing accountability. A tool is accountable for nothing. A supplier with a contract is.

It reports a number the buyer already tracks. Not tokens consumed, not messages handled, not a satisfaction score you invented for the occasion. If the client's operations lead already has a weekly figure on a dashboard, your system's output belongs in that same figure. First-response time. Tickets closed without human touch. Percentage of orders needing manual correction. Instrument it from day one, because the renewal conversation is decided by that number and by nothing else you built.

Where the price spread actually comes from

Published pricing for this work in 2026: businesses pay somewhere between 300 and 1,500 dollars for a working 24/7 support agent built on Voiceflow, Botpress or Stack AI.

That is a five-fold range for what is nominally the same deliverable, and the spread is not model quality. All four of the properties above sit inside it. The 300-dollar version answers common questions from a static FAQ and quietly does something unhelpful with the rest. The 1,500-dollar version has a defined escalation path, keeps a transcript the client can audit, knows its own business hours, and drops a deflection rate into the reporting the support lead already reads on Monday.

Same builder tool. Same underlying model. The delta is operations work that nobody demos because it does not demo well.

The hand-off checklist

This is the artifact I would hand over with any delivery, and it is what turns a one-time build fee into a monthly one. If none of these exist, you have not finished; you have stopped.

[ ] runbook: what this system does, in the buyer's vocabulary, one page
[ ] trigger: cron / webhook / queue - named, with its schedule written down
[ ] credentials: whose account, whose card, what expires and on what date
[ ] failure path: where a rejected input goes, who is notified, expected latency
[ ] degraded mode: exact behaviour when the API is down or the source is stale
[ ] rollback: how to switch it off in under a minute, and who is allowed to
[ ] metric: the single business number, where it appears, who reads it weekly
[ ] eval set: the frozen inputs it must keep handling, with expected outputs
[ ] known-bad inputs: the cases we deliberately route to a human, and why
[ ] change log: every prompt or threshold edit, dated, with the reason

The last three lines are the ones that turn a project into a relationship. Everything above them is hygiene.

After twenty deliveries you own something the client cannot buy

Here is the part that took me longest to understand, and it is the honest answer to "what stops the client doing this themselves".

It is not access. Your client can subscribe to exactly the same tools at exactly the same list price, and increasingly they know it.

What accumulates instead is failure knowledge. Deliver twenty of these and you have a private catalogue of the inputs that break the thing. Product names containing measurements that the extractor reads as quantities. Scanned documents rotated 180 degrees where the text layer is present but upside down. Customers who send four short messages in a row, so the first turn arrives with no context and the agent answers a fragment. The word "cancel" meaning the subscription in one vertical and a single order in another. Dates written day-first by a supplier and month-first by the buyer, in the same feed. Voice models that mangle a specific regional accent that happens to be half of the client's customer base.

None of that is in the documentation. You paid for it one incident at a time, and every item on the list becomes a row in an eval set that a competitor starting today does not have and cannot shortcut. That is the moat: a regression suite built out of other people's outages. It is also why the second and third clients in a niche are more profitable than the first - the same catalogue applies, and the delivery time collapses.

The published margin data points the same way. The highest-margin categories in this market are ghostwriting, prompt engineering and UGC video - and the reason prompt engineering can appear on a margin list while prompting is simultaneously a commodity is that the paid version of it is not writing the prompt. It is owning the evaluation, the versioning and the failure catalogue around it.

The disclosure I owe you

I do not run an AI services agency, and this company reports no income from the path described above. We build automated forecasting systems, which is a different business with worse odds on a twelve-month horizon. Every figure I have quoted comes from published 2026 market data, presented as such, and not from our books.

What transfers between the two is the engineering discipline rather than the revenue. Our own forecasts are hashed and anchored to a Bitcoin block before publication, so the record cannot be edited after the outcome, and the scoring page shows the misses beside the hits. Our stated 50% intervals have been catching far more outcomes than that label permits; we filed that as a failure and shipped the diagnosis before we had the fix, because an interval that catches nearly everything carries no information.

That is the same instinct as the checklist above. Decide what failure looks like, instrument it, and put the number somewhere you cannot quietly ignore it. The scored record is at neuportal.ai/experiment.

Educational content - not financial advice.

This is the honest home of AI in markets, and it is a very different thing from prediction. It is worth walking through carefully, because the difference between a volatility forecast that means something and one that is decoration is measurable, and most of the genre fails the measurement.

Why direction is the wrong question for a liquid market

A deep market is not a puzzle sitting still. It is an adversary that has already priced whatever your model just discovered. By the time a directional pattern is visible in the data, it is visible to everyone with the same data, and the price reflects it. Forecasting the direction of the next move, in that setting, is close to calling a coin the market has already flipped.

This is not a limitation that a bigger model removes. It is the structure of the problem. The information that would tell you which way the price is about to go is exactly the information a liquid market competes away fastest. So a system built to answer that question is built to lose, slowly, in a way that only shows up over enough calls to be inconvenient to count.

Volatility is predictable in a way returns are not

Volatility is different, and the difference is a real statistical property: it persists. Calm days cluster with calm days, violent days with violent days, and a shock today raises the odds of a large move tomorrow. Returns are close to unpredictable; the size of the moves is not. That autocorrelation of magnitude — volatility clustering — is stable enough to learn from.

Give a model realised volatility over several lookbacks, options-implied surfaces where they exist, funding rates and open interest, and it can return a forward range that carries information even when the centre of that range is genuinely unknowable. Notice how much humbler that output is than an arrow on a chart. It does not say what will happen. It says how wide to expect the outcomes to be. And that single estimate is what everything downstream depends on: how large a position holds risk constant, where a stop is noise and where it is real, when to brace before a violent session instead of flinching after it.

The band most people draw is wrong in both directions

Here is where measurement separates from vibes. The standard way to turn a volatility number into a band is to multiply by the square root of the horizon — sigma times root-t. It is one line of code, it is everywhere, and for fat-tailed assets it misprices the distribution in a way that is worth stating precisely.

We measured it against the entire Binance history rather than a flattering recent window — 3,261 daily bars for Bitcoin back to 2017. The quantity of interest is the ratio of an empirically-measured 80% band to the sigma-root-t band at each horizon. For Bitcoin it runs about 0.80 at one day, roughly 0.88 at seven days, and about 1.00 by thirty days. Read that carefully: at short horizons the parametric band is too wide, and by a month it is about right. The error changes sign as the horizon extends, so there is no single correction factor that fixes it.

The reason the short-horizon band is too wide despite genuinely fat tails is that the excess kurtosis — around sixteen on daily returns, against three for a normal distribution — lives in the extreme tails, not in the tenth-to-ninetieth-percentile shoulders. So the 80% interval is actually narrower than a Gaussian would imply, while the 99% interval is much wider. Fat tails and a narrow 80% band coexist. A parametric shortcut hides exactly that, and hiding it is how a band ends up quietly lying about what it knows.

Read the band off the data, and count your samples honestly

The fix is to stop parameterising and read the interval straight off the empirical distribution of realised moves over the matching horizon, tilting the midpoint only with a momentum lean that engages when a trend gate clears — never with a hand-drawn line. Every number then has a stated source: it is a quantile of real history, not an assumption.

There is one trap in doing this, and it is a subtle one. The multi-day moves overlap — consecutive thirty-day windows share twenty-nine days of data — so the samples are heavily autocorrelated. If you report the raw count of overlapping windows as your sample size, you overstate your evidence by roughly the horizon. Bitcoin's thirty-day band, drawn from about 3,231 overlapping windows, rests on only around 107 independent months. That is a materially different epistemic object, and collapsing the two is how a backtest manufactures confidence it has not earned. We print the independent count on every chart for exactly this reason: a band should show how much history actually stands behind it, not how much it can appear to.

Coverage: the honesty metric that cuts both ways

The metric for an interval forecast is coverage, and its most important feature is that it fails in both directions. If you claim a 50% range, the outcome should land inside it about half the time across many days — not most of the time. A band that contains the price ninety percent of the time is not precise, it is padded, and padding is cowardice dressed as confidence: it can never be caught being wrong, which is exactly why it is worthless. A band too narrow gets caught immediately. Both are failures, and the only way to tell which one you are looking at is to score the same forecaster over many out-of-sample days against outcomes fixed in advance.

This is the measure a volatility model lives or dies by, and it is the one almost no public market analysis reports, because reporting it means publishing the times the band was wrong. Over-coverage has to count as a miss or the whole exercise is theatre. Say so in those words, or the number means nothing.

Why a volatility forecast has to be committed before the fact

A forecast is only evidence if it existed before the event. This is the plainest thing in the field and the most routinely ignored, because the entire "AI called this move" genre survives on screenshots taken afterward, on ranges that were never written down until they looked good.

The fix is not a better model. It is a timestamp. We write each forecast down first, serialise it, hash it with SHA-256, and anchor that hash to the Bitcoin blockchain through OpenTimestamps before any of it is public. Then we score it openly, the misses on the same page as the hits, with no filter that hides them. The Bitcoin block does not prove the forecast was good — the coverage score does that. It proves the number existed before the outcome did, which is the one claim no amount of after-the-fact narration can fake. One practical note from building this, because it is the kind of detail that quietly discredits an honest record: hash the exact bytes you publish. Write the file, hash the file, timestamp the file — if a reader runs the hash themselves and gets a different digest because you re-serialised in between, it reads as fraud even when nothing was wrong.

What a volatility model is not

The deflation belongs here, because leaving it out is how the genre gets away with itself. None of this is an edge. Reading volatility well lowers the cost of being wrong; it does not tell you the future, and it will not beat the market. No method reliably beats a liquid market, and anyone promising that is selling something — usually a subscription, sometimes a token, always a screenshot.

What an honest volatility model buys you is not prophecy. It is a band whose width means what it says, scored in the open where it is allowed to look bad, committed before the candle closed so the record cannot be curated later. A forecast is a risk object before it is anything else, and the machine earns its keep not in the arrow on the chart but in the honest width of the band around it — and in being able to prove, afterwards, that the width was honest.

Educational content — not financial advice, and not a betting tip.

#Artificial Intelligence · #Machine Learning · #Volatility · #Cryptocurrency · #Data Science

That's calibration — of all the times a model said 70%, do ~70% actually happen? You measure it with a reliability diagram: bucket predictions by probability, then compare predicted vs. observed frequency. The diagonal is perfect honesty; below it means overconfidence.

import numpy as np

def reliability(p, y, bins=10): edges = np.linspace(0, 1, bins + 1) idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1) for b in range(bins): m = idx == b if m.any(): print(f"{edges[b]:.1f}-{edges[b+1]:.1f}: " f"pred={p[m].mean():.2f} actual={y[m].mean():.2f} n={m.sum()}")

p = np.random.rand(1000) y = (np.random.rand(1000) < p).astype(int) # calibrated by construction reliability(p, y)

A model can be highly accurate and badly calibrated — great on easy cases, overconfident on hard ones — and in risk-sensitive settings that calibration failure is the one that bites. Pair the diagram with a proper scoring rule (Brier, log loss) and you're grading honesty, not hit-rate.

We track calibration on live forecasts in public at neuportal.ai/experiment.

Educational content — not financial advice.

The honest answer is more interesting than a simple yes or no. This article explains how prediction markets work, how AI forecasting models work, and where machine learning genuinely helps versus where it runs into the hard wall of market efficiency. It is educational content, not betting or financial advice.

What Are Prediction Markets?

A prediction market is a marketplace where people trade contracts tied to the outcome of a future event. Platforms like Polymarket and Kalshi list questions with clearly defined resolutions — "Will X happen by date Y?" — and let participants buy and sell shares in each outcome.

Kalshi is a US-regulated exchange that offers event contracts on subjects like economic data, weather, and public events. Polymarket is a crypto-based platform where users trade on a wide range of world events. The mechanics differ, but the core idea is the same: a contract that pays out a fixed amount if an event happens, and nothing if it doesn't.

Because the payout is fixed, the price of the contract carries information. If a "Yes" share that pays $1 is trading at 65 cents, the market is effectively saying the event has roughly a 65% chance of happening.

How Prices Become Probabilities

This is the elegant part. In a prediction market, the price is the probability — at least approximately. A contract trading near 90 cents reflects a near-certain outcome; one near 10 cents reflects a long shot. Prediction market odds are simply these prices expressed as percentages.

Why should a price track a real probability? Because traders have an incentive to correct it. If you believe an event is genuinely 80% likely but the market says 60%, you can buy the underpriced "Yes" shares. When enough informed people do this, the price moves toward the consensus estimate. The market aggregates scattered knowledge — news, expertise, intuition, and money at risk — into a single number.

That aggregation is exactly why prediction markets are hard to beat, as we'll see.

How AI Forecasting Models Work

AI forecasting covers a broad family of techniques, but most share a common goal: estimate the probability of an outcome from data. At a high level, a model looks at historical examples, learns statistical relationships between inputs (features) and outcomes, and produces a probability for a new, unseen case.

Approaches range from classical statistical models and gradient-boosted trees to modern neural networks and large language models that can read and summarize text. A sports model might weigh team form, injuries, and venue. A macroeconomic model might track leading indicators. A language model can ingest news articles and reason about what they imply for an event.

Crucially, a good AI forecast is not a crystal ball. It is a calibrated estimate of probability. The best forecasting systems are judged not on whether they were "right" on a single call, but on whether their stated probabilities match reality over many predictions — a concept we'll return to.

Can AI Beat Prediction Market Prices?

Here is the question everyone actually wants answered. The realistic response: beating a liquid, active prediction market is very hard, and no method does it reliably.

The reason is market efficiency. A price on a popular Polymarket or Kalshi contract already reflects the combined reasoning of many participants — including people running their own sophisticated models. For an AI to profit, it needs an edge the market hasn't already priced in. In deep, liquid markets, most obvious edges have already been competed away. The price is a moving target that absorbs new information quickly.

This doesn't mean AI is useless. It means the naive dream — point a model at a market and print money — doesn't survive contact with reality. Where AI can add value is subtler:

• Speed and coverage. A model can monitor thousands of markets at once and react to news faster than any human, flagging where prices may lag fresh information.

• Consistency. Machines don't get bored, emotional, or attached to a losing position.

• Calibration analysis. AI can study whether a market's prices have historically matched real-world frequencies, and where they systematically drift.

• Spotting known biases. Markets have documented quirks, and models can help identify where they appear.

Notice that none of these is a guaranteed profit engine. They are analytical advantages, and in thin or illiquid markets they may matter more than in heavily traded ones. But an edge in analysis is not the same as a reliable way to make money, and anyone claiming otherwise is overselling the technology.

The Favourite-Longshot Bias

One of the most studied quirks in betting and prediction markets is the favourite-longshot bias. Across many markets, longshots — low-probability outcomes — tend to be overpriced relative to how often they actually occur, while heavy favourites tend to be slightly underpriced.

In plain terms, people tend to overpay for the small chance of a big payout, the same instinct behind lottery tickets. This is a well-known general pattern rather than a precise law, and its size varies by market and era. AI can help measure whether a bias like this shows up in a given set of prediction market odds — but the bias being known also means it is often already partly corrected by savvy traders. Inefficiencies have a way of shrinking once everyone is looking for them.

Why "Accuracy" Is the Wrong Question

People love to ask how "accurate" an AI forecaster is, but for probabilities, raw accuracy is a misleading yardstick. If a model says an event is 70% likely and it doesn't happen, was the model wrong? Not necessarily — 30% things happen all the time.

The better lens is calibration. A well-calibrated forecaster is one whose 70% predictions come true about 70% of the time, across many forecasts. You can only judge calibration over a large sample, not a single dramatic call. Scoring rules such as the Brier score exist precisely to reward forecasts that are both confident and correct while penalizing overconfidence.

This is why a forecaster who looks "wrong" on one flashy event can still be excellent, and why someone who nails one bold prediction may simply have been lucky. Prediction markets themselves tend to be reasonably well-calibrated, which is another reason they make such a tough benchmark.

How to Think Critically About AI Predictions

If you encounter an AI prediction — from a model, a newsletter, or a dashboard — a few questions keep you grounded:

• What's the track record, over many predictions? One correct call proves almost nothing.

• Is it calibrated? Do its stated probabilities hold up across a large sample?

• Compared to what? An AI forecast is only impressive if it beats a simple baseline — often the market price itself.

• What's the incentive? Anyone promising guaranteed winners is selling something, not forecasting.

Treat AI predictions as one input among many, not an oracle. The market price is frequently the toughest baseline to beat, precisely because it already blends the crowd's collective judgment.

The Honest Takeaway

So, can AI predict prediction markets? It can forecast the underlying events, analyze how prices behave over time, and highlight where markets may be biased or slow to react — genuinely useful things. What it can't do is reliably beat efficient, liquid markets or turn forecasting into a money machine. The value of AI here is clarity, coverage, and calibration, not magic.

That framing is why the most interesting work in this space is transparent about its results. One public experiment, NeuPortal, scores AI forecasts against prediction-market prices in the open (neuportal.ai), which is a healthier way to think about the question than any promise of guaranteed edges. Approached honestly, AI is a powerful lens for understanding markets like Polymarket and Kalshi — as long as you remember that understanding a market and beating it are two very different things.

#AI #PredictionMarkets #Polymarket #Kalshi #Forecasting #MachineLearning