As part of the Spring 2026 FutureEval tournament, we ran a survey. Among other things, this survey asked participants what other bot makers should learn from their bot-making experience. 37 bot makers answered this question. We share their anonymized/randomized responses in this piece along with overarching themes. The thematic analysis was run and written by AI. All quotes are taken directly from the survey. Note that many participants aren’t able to do extensive A/B testing, and some responses are more postulations than data-backed conclusions. A list of open-source bots is also shared. This post is complementary to our Spring Pro vs Bots Analysis and our more extensive Spring Bot Survey Analysis.
Top Themes
1. Simplicity beats complexity
Additional systems, roles, and "clever" structure hurt rather than helped. Several people explicitly regressed after adding complexity between seasons.
- #45: "more diversity != more performance"; more structure worked worse
- #63: dropped from 20th → ~63rd after adding logit aggregation, extremization, crowd-blending, etc.
- #90: collapsed from 13 named LLM roles to 2 models — simpler and more predictable
- #87: "A simple reliable baseline is hard to beat"
- #78, #23, #21, #30 echo this
2. Research quality is a bottleneck
Many said forecasting accuracy is gated by finding/parsing/synthesizing information, and that this matters more than prompting.
- #54: forecasts are only as good as your ability to find/parse/synthesize info; sophisticated prompting is pointless until then
- #90: stripping "LLM-as-researcher" in favor of raw retrieval was the biggest win
- #52, #60, #44, #33, #50, #26 all emphasize research; #44 notes deep research works even with non-frontier models
- #98, #81, #63: prediction-market / community signals as a research input (though #63 found crowd-blending introduced correlated bias)
3. Backtest against real resolved questions. Don't trust the sample set or naive backtests
- #70: sample/test questions are too few and narrow; test across binary/numeric/MC resolved questions
- #63, #21: complexity without backtesting hurts; log traces so you can backtest
- #81: be skeptical of backtests on resolved questions. Retrieval leaks the answer, so scores look artificially good
- #67: track Brier/log scores inside the bot for immediate calibration feedback
4. Newer Models may not mean better for forecasting
- #52: switched to Claude Opus 4.6 mid-season, turned out overconfident; labs optimize for coding, not forecasting
- #48: the system prompt can turn a great LLM into a terrible one
- #28, #30: frontier models weren't much better than earlier checkpoints
- Counterpoint: #90: GPT-4o → GPT-5.2 had more measurable impact than any prompt change
- Author’s note: Every other analysis we have done has indicated that newer flagship models perform better than older ones, so we believe this theme is interesting, but misleading.
Recurring secondary themes
5. Use tactics to encourage calibration & avoid overconfidence
- #81: anchor on an explicit empirical base rate before the model sees news; treat the LLM as an adjustment with a hard cap
- #11: many bots would benefit from capping/clipping predictions
- #26: top performance is about limiting mistakes, then getting well-calibrated
- #45: Monte-Carlo approaches were "atrocious for calibration"
6. Extremization / post-processing calibration is contested
- Split verdict: #50 found post-processing valuable; #33, #15, #21 tried extremization and rolled it back as harmful or unhelpful. #59 allowed extreme predictions but is unsure it mattered.
7. Ensembling & role separation
- Cheap + diverse beats expensive + singular (#81); multi-model committees (#87, #63, #18); using different models for different roles (research vs. forecasting vs. parsing) (#24, #51, #90). Aggregate in logit space, not probability space (#81).
8. Resolution criteria & edge cases cause catastrophic misses
- #45: lost 90 peer score by not telling the bot Metaculus's "assume it hasn't happened yet" assumption; "reasonable assumptions" are unreasonable
- #33: worst miss (2x the next) from not handling the "before X date questions are forward-looking" edge case
- #26, #60: careful resolution interpretation avoids big errors
9. Cost / budget management
- #81: a per-call cost ceiling is "not optional". One expensive model silently drained the budget, then everything fell back to 0.5; two-thirds of spend was wasted on unscored practice questions
- #17, #78, #24, #90 also flag cost/API-limit awareness
10. Invest in the feedback loop, logging & instrumentation
- #15: make minibench results easy to review; the feedback is limited and infrequent, so extract maximum value
- #79, #50, #21, #90: prioritize good data collection and structured logs over pure testability
11. Prompting has diminishing returns; teach judgment over rules
- #30, #33: returns on prompt optimization are rapidly declining
- #79: teach general forecasting judgment rather than specifying every situation; there are too many exceptions
12. Patience, persistence & disciplined iteration
- #80, #64, #17; #23: run bots in parallel to distinguish signal from noise instead of iterating too fast; #63/#21: change one thing at a time and measure it
Notable one-offs
- Agentic / tool-use scaffolding (#23): plugging tools into the Claude Agent SDK worked well; tension between single-agent bloat and over-relying on subagents
- Code execution is high-risk in a no-human-in-the-loop system (#54)
- Test-time scaling likely useful with deeper pockets (#28)
- Different pipelines per question type (#13)
- Several are early-stage/first-season and hadn't rigorously reviewed performance yet (#32, #69, #43, #72)
Open source bots
These developers said we could share their repositories publicly. Ranks are participant ranks, with Metac Bots removed.
A longer list of open source bots is on the FutureEval resources page.
Responses in full
All 37 responses to "what should other bot makers learn from your experience", verbatim and in random order. Participants were also asked, “Tell us about any approaches you tried and abandoned, and why”. We also share responses to this question when a developer also gave permission to publish them.
Bot maker #19: When starting out build for reliability (submit for all questions) and review each forecast and fix anything systemic that causes a catastrophic miss. Hopefully, I've gotten most of these out now that I'm on my 3rd minibench :)
Bot maker #67: Adding Brier score and log score tracking against resolved questions directly inside the bot (rather than as a separate offline step) was the single most useful development tool for us. it made calibration problems visible immediately.
Bot maker #45: #1 more diversity != more performance. I added lots and lots of systems, added instrumentation, and assumed that "they can't exactly hurt, right?" - a ridiculous idea in retrospect. It also turns out that almost all the new systems were absolutely atrocious, which surprised me, given I thought of them the same way I thought of all the others #2 more structure isn't necessarily better. the more structured systems worked worse #3 monte-carlo approaches are generally atrocious for calibration. I tried multiple different pipelines that attempted to solve this. They were all pretty awful! I'm very curious how others have approached this. #4 Making your own internet research bot probably isn't worth it. I'm giving it a second shot, but so far it's ended up quite pricey for middling performance. #5 Tell the system about the metaculus "assume it's not happened yet" assumption! I lost 90 peer score for that. #6 Pay more attention to the resolution criteria. My bot made a very silly assumption across every quasi-independent predictor, which was a bit of a wakeup call. I already had quite a literal bot about resolution criteria, but I needed to teach it that "reasonable assumptions" are unreasonable!
Bot maker #98: Checking prediction markets for relevant questions and putting guide rails in the prompt for how the model should and should not make its prediction were the two most impactful additions to my bot.
Bot maker #81: The single biggest lever was anchoring every forecast on an explicit empirical base rate before the model sees any news. We built a library of reference-class modules that inject a base rate for the question type, then treated the LLM as an adjustment on top of that anchor rather than the primary estimator, with a hard cap on how far it could move us from the anchor. This buys calibration and protects against confident nonsense on questions where the model has no real signal. If you do one thing, do this.
Ensembling helped, but cheap and diverse beat expensive and singular. We ran an equal-weight logit average of three or four low-cost models from different labs, and reconciled them with a stronger "adjudicator" model only when they disagreed sharply. Aggregate in logit space, not probability space. Watch per-call cost obsessively: adding one expensive reasoning model to the ensemble once exhausted our daily budget in about four questions, after which every later forecast silently fell back to 0.5. A cost guard that rejects any model above a per-call ceiling is not optional.
Two lessons we learned the hard way. First, measure where your money actually goes. We found roughly two thirds of our spend was being wasted on unscored practice questions we did not need to forecast. Second, be deeply skeptical of backtests on already-resolved questions: retrieval leaks the answer, so backtest scores look far better than live. The honest evaluation is on the future, or on near-range self-resolving questions you generate and resolve yourself.
What we would not repeat: chasing model breadth without a cost ceiling, over-engineering elaborate reasoning steps that a tight spend cap then starves into no-ops, and trusting absolute backtest Brier scores. Promising direction: strong reference-class base rates combined with prediction-market and community-prediction signals, blended conservatively, consistently mattered more for us than clever prompting.
Bot maker #43: Because I know more about the limitation of multi models forecasting
Bot maker #33: I spent a lot of time focusing on optimizing prompts via pastcasting which did help to some extent. However because the prompting wasn't the bottleneck once I upgraded models, I think it would've been better spent on just improving research. A lot of the mistakes that my bot made were related to simply not having the latest specific relevant value or underestimating the variance. I also abandoned post-prediction calibration (static extremizing and hedging) about halfway through because they both distorted the original prediction in ways that seemed more damaging than helpful. I think post-prediction calibration has a role, but probably makes more sense being done in a more rigorous llm-driven validation/calibration phase. I built my own crude research agent that did add value over the base Asknews article summaries, but it wasn't enough. Doing research in a more natural, dynamic, human-like way either before or during the forecasting step makes sense to me. And finally, my worst-scoring question (almost 2x the next largest miss) was on one where I would have had to include in the prompt that "before __ date questions are forward looking" so it's crucial to understand the unintuitive edge cases that the bot needs to handle to avoid massive mistakes.
Tried and abandoned: I tried temperature scaling for multiple choice questions but removed it when it seemed that it hurt calibration by reallocating too much probability away from the true outcomes. The opposite was true for numeric questions, where I extremized predictions by scaling up the normal distribution's SD, but it hurt calibration as well, but on the tails. (Both forms of attempts at calibration seemed like a good idea at the time but I ended up abandoning.)
Bot maker #44: Deep research works very well, even if you don't use a frontier model
Bot maker #63: I placed 20th in Fall with a relatively simple bot: Tavily + Perplexity for research, gpt-5 × 2 + claude-sonnet-4 committee, no crowd blending, no extremization. In Spring I added significant complexity — logit-space aggregation, mathematical extremization, crowd-blending with community predictions, multi-query research expansion, TinyFish, and stricter error handling that refused to forecast when research was insufficient. I dropped to ~63rd.
The lesson I draw: complexity without backtesting can hurt badly. Each addition seemed reasonable in isolation but I never rigorously validated whether it actually improved calibration on resolved questions. The REQUIRE_RESEARCH hard-fail likely caused missed forecasts that hurt my score. And extremization + crowd-blending together introduced correlated biases. If I were starting over, I'd add one thing at a time and measure each change against past resolved questions or test questions before going live.
Bot maker #52: Having solid research really helped my bot. Debate is powerful. Don't assume newer llms are better at forecasting. Labs are optimizing for coding/office work not forecasting. I did much worse than I should have this season because I switched to Claude Opus 4.6 halfway through which turned out to be overconfident.
Bot maker #32: first season, used the template with light customization, didn't do extensive testing
Bot maker #23: My general approach was to just plug all the tools I could code in the claude agent SDK, and that worked quite well with little scaffolding overall.
Having the bot create a cohesive view (read its past output, run tools to correct them, etc) worked surprisingly well, I think this is what shot me up at the very end of the tournament.
I started with one single agent who made all the tool calls (reasoning being bitter-lesson-pilled), however I soon noticed that doing this could bloat its reasoning stream and make it less coherent with time. However there's a tension with splitting it to subagent, as the bot can start relying too much on their results. This is a tension I'm still trying to figure out.
If I were to do it again, I would run several bots in parallel just to compare their progress, I was itterating too fast at the beginning and that made it hard for me to know whether there was signal or noise in the bot's improvement
Bot maker #72: news web search not working well
Bot maker #54: 1. your forecasts are only as good as your ability to find, parse, and accurately synthesize pertinent information - until you have mastered this, sophisticated prompting strategies are pointless. 2. code execution in a no-human-in-the-loop system is exceptionally error-prone and high-risk as a strategy.
Bot maker #64: Getting started is easy, iterating and refining and trying all the ideas is hard. I fell way short of what I wanted to achieve but still had fun.
Bot maker #24: using the same model for both research decomposition and final forecasting. separating roles across models (Opus for forecasting, lighter models for supporting tasks) improved both cost and output quality.
Bot maker #70: Don't rely only on the provided test questions to evaluate your bot. They are too few and too narrow to reveal how your bot will actually perform across the full range of tournament questions. Test against resolved Metaculus questions across different types — binary, numeric, and multiple choice — before going live. A bot that looks good on the sample questions can still perform poorly on the tournament's actual questions.
Bot maker #51: Using separate models for different roles as the Metaculus template has is best. I had parser failures, which were the biggest source of errors, and a dedicated stronger parser would have recovered more forecasts cleanly.(although I used free tier models; this is did to test the tournament and my bot though)
Bot maker #28: I suspect that test time scaling is going to be very useful if you have deeper pockets than me! And surprisingly, the most frontier’ed models were not all that much better than their earlier checkpoints.
Bot maker #69: There isn't much for me to say unfortunately, I haven't really updated my system majorly since Fall AIB (except for switching to newer LLMs), and haven't properly reviewed where my bot did not perform well.
Bot maker #78: Since I am new to software tinkering I think new person's or hobbyist can just use the metac template code by Ben Wilson and the team to run and understand. I tried to over engineer my bot and it wasn't really working fora great part of the spring contest.
Tried and abandoned: I tried to use only one research route tavily but always hit the monthly cap, I explored using llm own research and also let it run
Bot maker #80: You should have patience to see the reaults of your improvements.
Bot maker #59: Only thing I did was tweaking the prompt based on the question category and especially for poilitics ask the bot to put less weight on any public statements and put more weight on observed actions. This helped filtering out some noise. Additionally I asked it to assume that 2026 will be more chaotic than the previous 2 decades. That was basically it - a couple of guidelines in the prompt, the rest was the standard template. I also allowed it to make extreme (0.1 / 99.9) predictions, still need to check it it was harmful or not, but seems it didn't change much. Single model, multipass to fight extremes.
Bot maker #87: A simple reliable baseline is hard to beat. What seemed to work well was current-information retrieval and diverse model ensembling. Complex scenario prompts/agents didn't seem to help.
Bot maker #18: Multi runs across multi bots.
Bot maker #17: Be persistent and aware of the API calls.
Tried and abandoned: None, this was a prototype.
Bot maker #79: Spend the time reading up about how all the parts of your system work. LLMs have lots of different features and new ones are being added all the time. Some of the new things coming out are quite useful.
Being able to test your system is important, but don't sacrifice on your ability to collect data in order to make your system more testable. Getting good quality data into your system is one of the most important things.
It is generally a better strategy to teach/instruct your system on how to approach forecasting generally rather than being specific about doing certain things in certain situations. There are too many things in the world and too many exceptions to rules that you can never specify them all. Teach AI good forecasting judgement and it can figure out how to do the rest on its own.
Bot maker #26: Being a top bot still primarily requires limiting mistakes. This takes the form of robust research capabilities and careful prompting to avoid hallucinations or incorrect resolution interpretation. Once that is mostly achieved, my sense is that there is a lot of room to excel when it comes to producing well-calibrated forecasts. This is challenging and can be expensive to properly optimize, so advances in this area are highly valuable.
Bot maker #90: The biggest win was progressively stripping LLM-as-researcher in favor of raw retrieval. Starting with 13 named model roles (proponent, opponent, domain analysts, synthesizers) added cost and latency without proportional accuracy gains collapsing to 2 models with a fixed 0.70/0.30 weighted blend was simpler and more predictable. Making Tavily required rather than optional forced cleaner code and eliminated silent research failures. The optional extremization infrastructure logit scaling for binary, power transform for MC, both CLI-configurable was the right abstraction: tune aggressiveness without touching model logic. Drop-count tracking per model and failure mode was underrated for debugging parse errors at scale. What I wouldn't repeat: the 13-role v1 architecture, the performance uplift didn't justify the complexity or cost. The jump from GPT-4o to GPT-5.2 as primary forecaster had a more measurable impact than any prompt engineering change. Had improved performance from fall tournament
Tried and abandoned: Started with 13 named LLM roles (proponent, opponent, domain analysts, synthesizers) which added cost and latency without proportional accuracy gains. Collapsed to 2-model setup. Also abandoned LLM-as-researcher (GPT writing research summaries) in favor of raw Tavily outputs
Bot maker #60: I think depth of research and quality is important in making the bots accurately understand the resolving criteria and not misjudge a question
Bot maker #11: Many bots would benefit from capping/clipping their predictions.
Bot maker #30: Subjectively, the returns on prompt optimization/manipulation seems to be rapidly declining and out of the box models seem to perform increasingly well.
Bot maker #48: The models improve quickly over time, but the system prompt alone can turn a great LLM into a terrible one.
Tried and abandoned: 5cast-v1 (https://www.metaculus.com/accounts/profile/283198/) was a custom implementation of an agentic loop framework, but nevertheless took a lot of inspiration from other implementations out there:
- the provided metaculus framework for the prompts, which where then iterated upon
- a benchmarking framework against community questions inspired by https://github.com/Panshul42/Forecasting_Bot_Q2/blob/main/Bot/benchmark.py
- integrated Ask News (News & Wikipedia Search) support into the bot, worked well
- Serp API for Stocks and Google Trends, purely for more accurate Minibench results
- briefly tried Exa News, but landed on the native OpenAI search endpoint for cost instead
5cast-v2 (https://www.metaculus.com/accounts/profile/295029) tried to iterate on the previous design by replacing the custom agentic loop with an orchestrator of Opus Agents running in the Anthropic Agents SDK. This iteration proved to be worse, however, likely due to much worse prompts.
Bot maker #50: Calibration worked, post processing after LLM response(s) is valuable. Keep extensive, structured logs and/or database records to help identify and improve on weak areas. My main area of improvement for the next tournament will be better external research processes.
Bot maker #13: To have very different processes depending on the question type
Bot maker #15: Put in work to make it easier to review minibench results and get feedback. It's quite hard to pull info from this manually (and quite variable). Don't neglect investing in the feedback loop, especially since you only get limited and infrequent feedback it's important to get as much out of it as you can.
Tried and abandoned: tried extremization and (as attempted anyway) it seemed to be unhelpful or problematic, so rolled it back
Bot maker #21: I've had good results doing the basic stuff: research augmentation, better continuous/numeric question handling, model ensembles. However, getting reliably good results from more sophisticated approaches has been challenging. To the extent that I've tried more clever stuff, I haven't found solid boosts in backtested performance. And benchmarking backtests require logging upfront, so I'd encourage other bot makers to log your research and bot traces for future analysis and backtesting! And please share if you get anything cool/fun/creative to work.
Tried and abandoned: I experimented with stacking via Opus 4.5/4.6 this season but ultimately toggled it off; found it wasn't reliably better than basic median agg. Main change for Spring '26 compared to Fall '25 was adding XAI/Grok research as a secondary source compared to just AskNews in Fall. (I've augmented my research sources even more for Summer '26.)
