Four AI forecasts still need one clear decision rule

More opinions do not choose the action for you

Imagine asking four forecasting models how many people will attend a workshop. They return different answers, and a combined forecast produces one tidy number. The tidy number is useful only if you know what you are going to do with it. Ordering supplies, choosing a room, and deciding whether to hold the event are different decisions.

IBM Research’s October 22 article presents Granite Timeseries Ensemble, a recipe combining four pretrained Granite forecasting models. It describes a default method that pools predictive distributions with equal influence and an alternative that changes influence based on forecast spread. The release is a useful starting point for a methods question: how should a team inspect disagreement before relying on an ensemble?

This article offers an illustrative decision exercise, not results from running Granite or a recommendation for a consequential forecasting deployment. Every attendance number below is invented for explanation and labeled as such. The aim is to make the relationship between a prediction, its uncertainty, and the action visible.

Source: IBM Research: Ensemble forecasting with IBM Granite time-series models ↗ · IBM Granite: Timeseries Ensemble R1 model card ↗ · Wang, Hyndman, Li and Kang: Forecast combinations, an over 50-year review ↗ · Hyndman and Athanasopoulos: Forecasting, Principles and Practice, the forecaster's toolbox ↗ · Hyndman and Athanasopoulos: Practical forecasting issues and forecast combinations ↗

Connect the cost of error to visible disagreement

For the exercise, suppose you are planning a small public workshop and counting how many printed activity sheets to prepare. Too few sheets would require sharing or printing more. Too many would leave reusable extras. That gives overestimation and underestimation different consequences, even before you choose a model.

Write those consequences without pretending you know exact monetary costs. Perhaps spare sheets can be used next month, while the printer is unavailable during the event. That context could justify a different preparation rule from one used for perishable refreshments. The forecast can be identical while the sensible action changes.

Now separate the number you want to predict from the number you will prepare. Predicted attendance is a statement about an uncertain outcome. Printed quantity is a choice under that uncertainty. If your spreadsheet stores both in one cell called expected, someone may later mistake the preparation buffer for the model’s prediction.

Make the decision rule explicit before seeing which model looks best. For instance, you could compare a central estimate with a more cautious preparation scenario. Do not present that scenario as a confidence guarantee. It is a planning choice whose usefulness you will assess against what actually happened.

Consider four hypothetical point predictions: 40, 42, 43, and 75 attendees. Their arithmetic average is 50. That calculation is easy to verify, but it does not tell you why the last prediction differs or whether the average is the best basis for preparation.

One model might be responding to a pattern the others missed. It might also be responding badly to an unusual input. You cannot decide between those explanations merely by counting how many models agree. Three forecasts close together are not three independent witnesses if their information or limitations overlap.

Now consider a second hypothetical set: 49, 50, 50, and 51. The average is also 50. These two examples have the same central value and very different visible disagreement. If you retain only the average, you discard information that could prompt a useful investigation.

These arithmetic examples are not an implementation of Granite’s probability-pooling method. They are deliberately simpler so you can see the decision problem without confusing it with the released algorithm. Keep that boundary clear when explaining an ensemble to teammates: a convenient illustration should not quietly become a technical description of the product.

Read the combination rule and check its uncertainty

The research review by Wang, Hyndman, Li, and Kang places forecast combinations in a much longer methodological history. It discusses both point and probabilistic forecasts and the benefits and limitations of combining them. An ensemble is therefore a family of approaches, not a single guarantee that more models will always help.

For the current Granite recipe, IBM’s model card is the place to check the actual configuration, supported outputs, and constraints. It describes loading member checkpoints rather than a single merged checkpoint. It also notes that the default recipe processes channels independently, which matters if your application depends on relationships between them.

Our interpretation is that you should be able to explain the combination rule at the level your decision requires. Are you combining central values, distributions, or something else? Are member weights fixed or responsive to a property of the predictions? What happens when a member is missing or outside its supported input range?

If you cannot answer those questions from the documentation, keep the method in exploration. You can still learn from its outputs, but avoid writing a confident operational story around a combination you do not understand. Naming the unknown is a useful engineering result, especially before other people begin treating the output as authoritative.

A forecast range is more informative than an unexplained single number, but a narrow range is not automatically a better one. The forecasting textbook by Hyndman and Athanasopoulos distinguishes point predictions, distributions, and prediction intervals and provides methods for evaluating them. The useful question is whether the uncertainty description behaves appropriately for the task.

Here is a hypothetical coverage exercise. Suppose you have 20 comparable past forecasts, each with a nominal 80 percent interval, and only 10 observed outcomes fall inside those intervals. That is 50 percent observed coverage in this small sample. It is a warning to investigate, not a precise verdict about the model’s long-run behavior.

The reverse problem matters too. An interval that includes almost every imaginable outcome can cover the observations while doing little to guide preparation. In the workshop example, a range from zero to several thousand attendees would not help you decide how many sheets to print.

Look at both coverage and usefulness, and retain the number of cases behind any percentage. Twenty cases and two thousand cases do not carry the same evidential weight. Also inspect which cases missed. A method that repeatedly understates unusually busy events may create a particular operational problem even if its overall summary looks acceptable.

Keep disagreement available for diagnosis

For your experiment, save the individual member outputs alongside the combined output when your tools support doing so. Do not assume the application exposes them automatically. The purpose is to make later questions answerable: which members differed, when did they differ, and did that difference correspond to a meaningful feature of the data?

Add short context notes for unusual periods. A workshop moved venues, a registration form closed early, or an announcement was sent to a new audience. These notes should describe observed events, not explanations invented after seeing the error. They can help you distinguish a model problem from a data-definition problem.

For example, imagine the recorded target sometimes counts registrations and sometimes actual arrivals. Combining four models will not repair that ambiguity. Before comparing predictions, define the outcome consistently. Otherwise an apparently sophisticated ensemble evaluation may be measuring changes in what the spreadsheet means.

Avoid turning every disagreement into an exception rule immediately. With a small sample, it is easy to write a special case for each surprising result. Keep a distinction between observations worth investigating and patterns supported well enough to influence the next version of your method.

Compare decisions and budget for the complete workflow

The forecasting textbook’s evaluation material provides a foundation for checking forecasts on data not used to fit them. For this exercise, extend that thinking to the decision you care about. Record what each candidate rule would have caused you to prepare, then compare the consequences against the observed attendance.

Use a simple baseline as well as the ensemble. You might start with the attendance from the most comparable previous event, clearly noting why you considered it comparable. The baseline gives a new method something concrete to improve on. An elaborate forecast is not useful merely because it is more difficult to produce.

In a hypothetical event with 46 actual attendees, a preparation rule that prints 50 sheets leaves four extras. A rule that prints 42 leaves a shortage of four. The absolute difference from attendance is the same, while the practical effect differs. That arithmetic explains why a model score and a planning score can answer different questions.

Decide how you will describe these outcomes before comparing the methods. You do not need to invent dollar values for convenience. Counts of shortages, reusable extras, and additional preparation actions may be enough for the exercise. Use terms the person responsible for the event can understand and challenge.

IBM’s model card cautions that loading multiple members adds resource demands relative to a single model. That is a reason to measure the complete workflow you would actually use. A benchmark ranking does not tell you whether the method fits the machine, time window, or budget available to your project.

For a trial, separate installation and first-download work from routine execution. Record the environment, relevant versions, input size, and what you measured. If a result took longer because a dependency was missing, preserve that observation instead of quietly treating the time as model inference.

Also account for human work. How long did it take to prepare consistent data, inspect unusual cases, and explain the range to the person making the decision? Those tasks can matter more than a small difference in automated runtime for a modest recurring project.

This does not mean the simpler model always wins. A more demanding workflow might be worthwhile if it produces a meaningful and reliable improvement in the decisions you care about. The comparison should make that tradeoff visible. The right question is what useful outcome you gained for the additional effort and resources.

Decide how the system should behave when evidence is weak

A forecast workflow needs an answer for insufficient history, missing inputs, or a change in how the event operates. For the workshop exercise, you could choose to display a planning range and a note asking the organizer to review assumptions. That is different from silently presenting the latest numerical output as equally dependable in every situation.

Keep the fallback understandable. If a member fails, say whether the system stops, uses a documented alternative, or produces an explicitly incomplete result. Do not let an implementation accidentally choose the policy by averaging whatever numbers happened to arrive.

Consider a change in audience as another test. A method based on previous local workshops may be less informative for a newly advertised regional event. You do not need to claim a specific model failure to recognize that the decision context has changed. Flag the change and collect evidence before assuming continuity.

Assign someone to review these conditions. The point is not to add a human signature to every routine calculation. It is to ensure that a meaningful change in data or context reaches a person who can alter the plan. A forecast can assist that person without hiding the conditions under which it was produced.

Try a small exercise with a clear decision rule

You can explore the core ideas without installing Granite. Create a small table with a few explicitly fictional events, four proposed attendance values per event, a combined value, a preparation quantity, and the actual attendance you choose for the scenario. Label the table as illustrative so it cannot be mistaken for performance evidence.

Include one case with close agreement, one with a single outlier, and one where all four predictions are wrong in the same direction. For the last case, imagine every prediction is near 40 while actual attendance is 70. The example makes a key question visible: would your plan notice a shared miss, or would agreement make it look reassuring?

Ask another person to choose the preparation rule before revealing the illustrative outcomes. Then discuss which information they found useful. Did they want member values, a range, a note about the event, or an explanation of the cost of a shortage? Their answer can guide the interface of a future forecasting tool.

If you later run the real recipe, keep this teaching exercise separate from your evaluation data. The fabricated numbers explain concepts; they cannot establish accuracy. A real evaluation needs observed outcomes, a fixed comparison plan, and a record of the model configuration that actually produced the forecasts.

The current release makes it easier to try several forecasting architectures through a common recipe. That is useful access to a method, not a completed argument for using it in every project. Your own task still determines what outcome matters and what evidence would justify a change.

A good review should be able to answer three plain questions. Did the forecasts improve on a relevant alternative? Did the uncertainty description help you handle difficult cases? Did the resulting decision justify the extra work? If the answers differ, preserve the disagreement rather than compressing it into one overall score.

For the workshop example, you might keep the baseline while continuing to study the ensemble, or use the ensemble only as a second view during planning. Those are legitimate outcomes of an honest trial. Learning that a tool is interesting but not yet useful for your decision is still progress.

The practical lesson is to keep individual predictions, combination rules, uncertainty checks, and actions connected. An ensemble can summarize several views of the future. A clear decision rule explains what you will do with that summary and gives you something meaningful to evaluate after the future arrives.