Designing the Next Generation of Forecasting: A Series

The energy grid is changing faster than most forecasting tools were designed to handle. Behind-the-meter assets — rooftop solar, battery storage, electric vehicles and smart HVAC — are now large enough to make traditional point-forecast models unreliable for operational decisions. Utilities need forecasts that communicate uncertainty honestly, adapt to new load shapes and support the kind of probabilistic reasoning that modern grid management demands. Over the past several months, I’ve been exploring what it would take to get there — and I’ve been using Claude as a research and prototyping partner throughout. This is the first in a series of short papers documenting that work: the questions I asked, the concepts I had to learn, the architectures I sketched out and the prototypes I built to test them with real data.

The goal of this series isn’t to describe finished products. It’s to document the process of moving from a problem statement to a working prototype — a process that, in my experience, is where most of the real design decisions get made. I’ve found that using an AI assistant changes that process in meaningful ways: it compresses the research phase, makes it easier to think through unfamiliar technical territory and turns what used to be a solitary design exercise into something more like a conversation. Whether that’s useful depends on how you use it, and part of what I want to capture in this series is the specific prompts, pivots and dead ends that shaped the work.

The first paper covers my initial work on Ensemble Forecasting — the idea that instead of running a single load forecast against a single weather forecast, you run many forecasts across many weather scenarios and let the spread tell you something meaningful about uncertainty. I used Claude to research the topic, develop a presentation, sketch out a two-branch application architecture combining simulation ensembles with Bayesian deep learning, and build a working prototype of the first branch. That paper is available to download here. This series covers applied forecasting research at the intersection of utility operations and machine learning.