![\"NASA](\"https://dynamical.org/assets/catalog-thumbnails/nasa-imerg-analysis-early.jpg\")
Who's the acronym artist at NASA responsible for this tomfoolery? Yes, IMERG is constructed from Integrated Multi-satellitE (!!!) Retrievals for GPM. Next level.
\nIMERG combines observations from a constellation of satellites into a global precipitation analysis. We now have two live-updating IMERG datasets in the catalog:
\nBoth are global, 0.1° (~10 km), half-hourly, and extend back to 1998. They include precipitation rate and a precipitation quality index.
\n
We are releasing the Early and Late runs of IMERG V07. When NASA releases V08, we plan to add the Final run as well.
\nEvery catalog page now includes two ways to open a dataset: with dynamical-catalog, or directly from our STAC catalog with Icechunk.
The examples use pystac because it keeps the code concise, but any HTTP client can read the same STAC metadata. The important part is that the STAC is the source of truth for the current Icechunk asset:
import icechunk\nimport pystac\nimport xarray as xr\n\ncatalog = pystac.Catalog.from_file(\"https://stac.dynamical.org/catalog.json\")\ncollection = catalog.get_child(\"noaa-gfs-forecast\")\nasset = collection.assets[\"icechunk-https\"]\n\nrepo = icechunk.Repository.open(icechunk.http_storage(asset.href))\nsession = repo.readonly_session(\"main\")\nds = xr.open_zarr(session.store, chunks=None)One request: don't copy the asset URL out of STAC and hard-code it into your application. We may move or version stores in the future. Resolve the current location from STAC when you open the dataset, or let dynamical-catalog do that for you.
This is a good time to remind everyone still accessing data products via data.dynamical.org: that endpoint is deprecated and will be discontinued at the end of August.
Zarr-Python 3.3.0 is out. Among many awesome improvements, Alden's partial-shard read optimization replaces serial requests for chunks within a shard with one Store.get_ranges() call, letting stores fetch ranges concurrently and coalesce nearby ranges into fewer requests. The coalescing controls are tunable, but the optimization is now on by default.
Every materialized dynamical.org dataset is sharded, so they all stand to benefit. The biggest gains should appear when a read spans multiple chunks within each shard, especially on remote storage.
\nPeople are saying "four times faster!" One real-world benchmark of regional reads from a sharded weather reforecast cut the median from 10.8 seconds to 2.75 seconds (that's 3.9x faster!).
\n"What then is time? If no one asks me, I know; if I wish to explain it, I do not."
\nYou all made Saint Augustine proud with your questions, and prompted a deeper dive into when a forecast should be considered "delayed" (among other explorations).
\nThe short of it:
\nThe longer version, including what we learned from replaying roughly a year of arrivals, is in our living lab notes.
\nThe findings resulted in improvements to our pipeline status.
\nWe now ingest dissemination advisories from NOAA and ECMWF, map them to the products we monitor, and show them alongside observed delays. We backfilled the advisory archive to early 2022 and now track new and resolved notices in real time.
\nThese are useful as two separate signals: our arrival monitoring tells us what is happening in the files, while an agency advisory can tell us why. Advisories are available on the pipeline status page, in the machine readable status feed, and as a webhook event.
\nThank you for your requests! We added a few requested variables to existing datasets:
\nThe exact variable names and coverage are in each catalog entry.
\nNVIDIA Earth2Studio now includes native dynamical.org data sources. Earth2Studio resolves collections through our public STAC catalog and reads the anonymous Icechunk repositories directly.
\n\nThanks to Duncan Anderson (who made an excellent book recommendation), Nick Geneva (who has not yet discussed books with me -- yet --, but was very pleasant to chat with in the PR!) and the NVIDIA Earth2Studio team for suggesting and working through the integration with us.
\nEnjoy the weather,\n^ MM
\n", "date_published": "2026-07-30T00:00:00Z" } , { "id": "https://dynamical.org/research/virtual-data-products/", "url": "https://dynamical.org/research/virtual-data-products/", "title": "Having it all: virtual and materialized data products", "content_html": "We want it all:
\nBut when your data is 14 PB and 1 billion GRIB messages, making that dream a reality takes some work. Virtual Icechunk Zarrs are a key complement to our materialized (rechunked) Zarrs. Together they get us one step closer to the catalog of our dreams.
\nWe recently made a big push to build up our virtual Icechunk capabilities. This note describes why we create them, how we do it, and the lessons we've learned along the way.
\nFor gridded weather data, thoughtfully designed virtual Zarrs offer three main benefits.
\nAdding virtual Zarrs gives us a catalog of options tuned to different jobs: our materialized (rechunked) products for training a regional time-series model across a forecast archive, our virtual products for low-latency inference, an analysis product as a proxy for observations, and so on.
\nThe geospatial data sector spends a lot of time optimizing, sometimes prematurely. Is this one of those moments? We think it isn't: we started with the problems (latency, completeness) and worked backward, exhausting our existing techniques before picking up a virtual approach.
\n"Virtual Zarr?" you may be saying, "sounds like a Zuckerberg fever dream." Nay, I say: a virtual Zarr is nothing more than a Zarr whose chunks are references to data stored somewhere else.
\nEvery chunk in a virtual Zarr is backed by a virtual reference, and the reference is just a URL saying where to find the file, and the byte offset range to read within the file. The Zarr's metadata also stores which codec to use to decode the bytes (netCDF, HDF5, GRIB2, etc.). If you are familiar with GRIB indexes, imagine someone combined every GRIB index from your favorite weather model into one gigantic index.
The virtual Zarr concept grew out of the challenge: "We have these huge piles of data in legacy file formats. We'd like to use it as an analysis-ready, cloud-optimized data cube, but we don't want to reprocess and double-store all the data." The kerchunk project pioneered this concept with the realization that by storing references that point to those legacy files, client libraries could read them on the fly, and the whole experience would act like a modern data cube. The VirtualiZarr project built on this, improving the ergonomics of constructing a virtual Zarr. Virtual capabilities were a natural addition to Icechunk. After all, references to chunks of data are core to how Icechunk achieves atomic updates and data versioning for virtual and materialized Zarrs alike.
\nSo what happens when you read ds["temperature_2m"].sel(time="…") from a virtual Zarr?
temperature_2m/c/0/1/2 and temperature_2m/c/3/4/5) and asks the Zarr store for those chunks.Up to this point, virtual and materialized Zarrs follow the same path. Here's where they diverge:
\ntemperature_2m/c/0/1/2 and looks it up in its manifest to find a source URL and byte range, then fetches those bytes (from local disk, object storage, wherever the URL points).codec named in the Zarr's metadata. In a materialized Zarr, that codec is often a standard compressor like zstd. In a virtual Zarr, it's a codec that knows how to read the native file format: netCDF, HDF5, or for GRIB weather data, the GribberishCodec.Once the bytes are decoded into an array, everything is back to standard Zarr: the array is indexed as usual, returning the values you asked for.
\nThe short answer: try both, and see which is fastest for each of your access patterns. Within the same project it may be optimal to use both together.
\nFor dynamical.org's catalog, here's a rule of thumb:
\n\n\nDoes an individual unit of work read the model's whole geographic domain? If yes, use virtual, else use materialized.
\n
If a variable you need is only available in the virtual Zarr, try that and let us know at feedback@dynamical.org if the performance isn't what you need. We add variables to our materialized Zarrs based on demand.
\nMore generally, use the product that will fetch the fewest chunks of data for your access patterns. You can find the chunking of our datasets in the catalog documentation and in our STAC.
\nThere's nothing about virtual Zarrs that requires them to be faster for single-time-step, map, or spatial access patterns; it just happens that almost all native gridded weather data files follow that shape thanks to the mechanics of running a weather model. If someday we find native, time-series-optimized files, we'll virtualize those and have our materialized dataset rechunk to optimize for spatial access.
\nEnough talk. Let us show you the numbers.
\nWe define latency as the time between the moment a file first becomes available at the source and the moment that data is first available in our Zarrs — specifically, from a file's "modified" or "created" timestamp at the source to the timestamp of the first Icechunk snapshot containing the same data.
\nThese statistics are calculated on a relatively short history of our first virtual Zarr, NOAA HRRR forecast, 48 hour, virtual. As we build up a longer track record, we'll add them to our data product pipeline status page, where you can audit the details and see them covered under tight latency thresholds in our SLA.
\n| \n | p50 | \np95 | \np99 | \nmax | \n
|---|---|---|---|---|
| all | \n2.9 | \n4.2 | \n5.9 | \n7.7 | \n
| surface | \n3.2 | \n4.5 | \n5.8 | \n6.0 | \n
| pressure | \n3.1 | \n4.2 | \n6.1 | \n7.7 | \n
| model | \n1.7 | \n3.7 | \n4.7 | \n5.9 | \n
![\"Virtual](\"https://dynamical.org/assets/notes/latency_kde_by_type.png\")
Three priorities guided the design of our virtual Zarr update pipeline: fast, correct, resilient.
\nPush or poll? This is one of the core questions when designing a system that reacts to new data as it arrives. "Get notified right away" makes push notifications sound like the obvious choice wherever they're available. As you get into the details, the story becomes more complex: a dataset that produces many files at once (e.g., an ensemble run) needs debouncing regardless, since committing takes about a second and many files can land in that same second. And most national NWP agencies' operational file servers, the sources with the lowest latency, don't support push notifications at all. Backfills would need their own handling too, since there's no "new file" event to subscribe to for data that already exists.
\nPolling at a high rate (a second or less) collapses all of these cases into a single path. A backfill polls once, stores what's there, and moves on; an update keeps polling until the full forecast has arrived. The same loop works across HTTP, FTP, and object stores, and across backfills and updates alike, eliminating a whole class of "two code paths drifted apart" bugs. It's also resilient: if the process restarts, it catches back up on the next request.
\nAn update in our code follows these steps:
\nIf you'd like to get even more into the implementation, this is all in our open-source code.
\nIn the end, the bottlenecks to low-latency updates come down to:
\nEven for large ensemble forecasts, an incremental update with well-tuned manifest splits (see below) can do this work in a couple of seconds on modest hardware.
\nThat's not to say it was fast on the first try. Lots of N² or serial loops that don't matter at small N rear their heads at the scale of a billion references and over 100 Zarr arrays. But the good news is that they're all solvable: (1) ultimately this is small data per update and computers are fast, and (2) AI does great with a verifiable objective, and wall-clock time is exactly that. We're having fun working with AI to optimize this one.
\nHere's an incomplete list of bumps we had to work out:
\nmain by the time you read this.It's not enough to make them fast; we also wanted these virtual Zarrs to be as simple to use as the rest of our catalog. Much of that happens for free by virtue of them being an Icechunk Zarr: Icechunk exposes a nearly uniform interface regardless of how the dataset is constructed.
\nA key element of ease of use for us was drop-in compatibility with our existing materialized Zarrs. For any variable in our materialized data products you can grab the same variable from the virtual product and you'll get the same values out. To make this happen we apply a few select, on-the-fly transformations to make the values you get back from our virtual Zarrs exactly match those from our materialized Zarrs. Matt Iannucci, the creator of Gribberish (also working at Earthmover), has been very helpful along the way as we contributed a couple new GribberishCodec options that do those on the fly.
\nThere's a limit to how we can manipulate data to improve UX (although we do this sparingly). The two most salient: you won't find a deaccumulated precipitation rate like in our materialized datasets, and we can't homogenize a variable that switched units sometime in its history (e.g., hPa to Pa back in 2019).
\ngrib2.bz2? (e.g., NOAA MRMS, DWD ICON-EU) You bet, thanks to the power of Zarr codec pipelines.Are we living inside a Bostrom-esque simulation? Does that simulation exist inside another simulation?
\nThere's only one way to find out: let's add increasing layers of derivative abstraction on already-digital signals that inadequately capture what the sheeple call "reality." Yeah?! Yeah.
\nIf anyone has Morpheus' phone number let him know I'm ready to be picked up on the Nebuchadnezzar.
\nIt is our first virtual dataset. It's low-latency, optimized for spatial access (i.e. good for maps) and has all variables and vertical levels. Try it out, and check out the release trailer.
\n![\"noaa-hrrr-forecast-48-hour-virtual\"](\"https://dynamical.org/assets/catalog-thumbnails/noaa-hrrr-forecast-48-hour-virtual.jpg\")
We now have the machinery for rigorous, high-performance virtual data products. You will see many more of these, and they will be as thoughtfully designed as our materialized data products.
\nYou can read more about our design process in Alden's lab note: Having it all: virtual and materialized data products.
\nWe stood up status.dynamical.org to give insight into both:
\nThe latter initially focuses on upstream observability (e.g. GEFS dissemination on NOAA NOMADS) and our data products (materialized and virtual) will land shortly. We just wanted to make sure they were heckin' fast first ;)
\n![\"NOAA](\"https://dynamical.org/assets/status-pipeline-2026-07.png\")
We later added webhooks for subscribers to be notified on dissemination milestones or delays. We're still tuning what counts as "delayed", but overall it's been an invaluable tool for our team (and others) to learn more about the dissemination process.
\nYou can read the lab notes here: Knowing the moment a forecast is ready.
\nThis level of rigor is helping us prepare to offer an SLA to organizations who need additional support and latency/uptime assurance. Don't worry, free public access stays free. Email me at marshall@upstream.tech if you're interested in learning more about our SLA and/or supporting the sustainable operations & growth of dynamical.org <3
\nThe per-dataset validation reports we started in June got richer. I mean just look at this absolutely ridiculous one for the noaa-hrrr-forecast-48-hour-virtual data product.
\nllms.txt, structured metadata, sitemap). Lab notes inspired by the inimitable Ink & Switch.Enjoy the weather -- MM
\n", "date_published": "2026-07-13T00:00:00Z" } , { "id": "https://dynamical.org/research/when-the-forecast-is-ready/", "url": "https://dynamical.org/research/when-the-forecast-is-ready/", "title": "Knowing the moment a forecast is ready", "content_html": "Forecast production is a factory assembly line, a fulfillment center, and a delivery route all in one. Lewis Fry Richardson's Weather Forecasting Factory was not too far off.
\n![\"\"](\"https://dynamical.org/assets/notes/conlin-1.jpg\")
Every dataset in the dynamical.org catalog (so far) is downstream of a model run that\nsomeone else produces on their own cadence. And the initialization is just the beginning (literally and philosophically). Then the files start landing, one by one, eventually\ntrickling in over tens of minutes (or even hours) rather than appearing all at once. If your\npipeline depends on that data, you have two bad options: pretend you can divine a cron schedule that will "always work," or poll wildly.
\nAnd to add to that, we had questions like:
\nWe sought to answer these questions and many more so that the dynamical.org catalog would be resilient, low-latency, and designed around the minute details of its upstream sources. The prompt for actually sitting down and answering them was a question Will Hobbs posted on LinkedIn about NOAA model file availability.
\nSo, we built a tool called wxopticon to remove that guesswork (I pronounce it "waxopticon", and I say it in a slightly mischievous voice and picture Saruman reaching for the Palantir — no, not THAT Palantir. Oh never mind). It watches upstream weather sources and dynamical.org's own catalog stores, and it answers three operational questions:
\nYou can see all of this on the pipeline status page.
\nThe second component is a system that enables consumers to create subscriptions\n(via webhooks, Slack notifications, etc.) to meaningful events (e.g., "notify me\nwhen IFS ENS progress:f024 is complete" or "warn me when GEFS on AWS looks like it might arrive late").
\nwxopticon is a set of stateless functions over a single append-only event log in\nobject storage. The log is the source of truth, and\neverything else (the dashboard, status and timing, the readiness milestones) is a\npure replay of it.
\nA lean detection scan runs every two minutes: it replays the log to find the runs\nstill expected, probes their upstream locations, appends any new state\ntransitions, and fans each new milestone out to subscribers. A separate\nsummarize pass runs every five minutes to refresh the status feed and seed\n"delayed" signals. For products with a notification stream (e.g., AWS SNS), a\ncontinuous listener catches arrivals within seconds instead of waiting for the\nnext scan.
\nBecause every arrival is in the log, we can replay the whole dang thing. Over the\nlast year, wxopticon recorded roughly 1.9 million file arrivals across\n9,150 completed product-runs of thirteen upstream feeds.
\nA run arrives over time. The moment a run starts and the\nmoment it's complete can be many hours apart, and the shape of that arrival looks\ndifferent for every model.
\n![\"Scatter](\"https://dynamical.org/assets/notes/arrival-staircase.png\")
This is why "ready" is a series of milestones; a short-range\nconsumer can start using GEFS the instant the early lead groups land, long before the full run completes.
\nThe feeds are punctualish! Measured from init time to the last file of the run, the median completion time is about 1h47m for HRRR, 3h37m for DWD's ICON-EU, 5h15m for AIFS, 5h22m for GFS, and a full ~26h for GEFS's 35-day run.
\n![\"A](\"https://dynamical.org/assets/notes/time-to-complete.png\")
GFS completes within a 13-minute band from its median to its 99th percentile, run after run. Regularity is exactly what makes a learned next-run expectation meaningful. A "late" signal is useful because on-time performance is so consistent.
\nwxopticon models each run's progress as a sequence of readiness boundaries.\nRather than a single "done" flag, a run crosses named milestones as its lead\nhours become available:
\n| kind | \nfires when | \n
|---|---|
progress | \nevery lead ≤ an intermediate lead-group horizon is available (e.g., progress:f240) | \n
complete | \nthe full run is available — you don't need to know group names | \n
in_flight | \na still-running run is behind its learned schedule | \n
advisory | \nthe upstream agency opens or resolves a dissemination advisory | \n
That table describes events, not run states. We initially mixed those concepts\ntogether, which made it hard to say whether delayed meant "still running" or\n"finished, but late." The current model has two orthogonal axes:
status is lifecycle: pending, in_flight, complete, failed, or\nunobserved.timing is the judgment: on_time or delayed, when there is enough\nhistory to make one.I went back and forth on the correct threshold for "delayed." Our first pass\nsplit the question in two: an in-flight run was delayed at p95 plus one minute,\nwhile a completed run was judged against p99. Raw p95 was obviously too harsh:\nby definition, it would flag roughly one ordinary run in twenty. But adding one\nminute still hugged very consistent feeds too tightly, and p99 was a volatile\ntail statistic answering a different question.
\nWe replayed a year of arrivals and compared several buffers above p95:
\np95 + (p95 - p50), which adapts to dispersion but collapses back toward p95\non a very tight feed.![\"A](\"https://dynamical.org/assets/notes/threshold-candidates.png\")
The challenge was to come up with something formulaic that behaved sensibly across tight and wide distributions. What worked best was something not on our original list: a dispersion buffer with a floor:
\nspreadf = p95 + max(p95 - p50, 15 minutes)
Each product gets its own spreadf from a trailing 90-day window. Each lead\ngroup gets one too. This is useful: if GFS is still working toward\nday 10 after its day-10 group would normally be ready, wxopticon can mark the\nrun in_flight and delayed before the full 16-day run reaches its later\ndeadline. The event names the lead group that triggered the warning.
![\"A](\"https://dynamical.org/assets/notes/delay-threshold-comparison.png\")
The spreadf approach now drives the dashboard, in-flight warnings, and the\ncompleted run's timing.
\nWe also wanted to explore the question: can delayed lead groups foretell the full run being delayed? Using the spreadf methodology, lead groups produced 46 observable early warnings, or about 0.5% of runs:
\n![\"A](\"https://dynamical.org/assets/notes/lead-group-stagger.png\")
An in_flight event triggered by\na lead group might be a useful warning depending on one's sensitivity to delays. Or one could subscribe to\ncomplete_delayed if they only wanted confirmed late completions.
From time to time, ECMWF and NOAA, for example, issue dissemination advisories: memorable ones include the dramatic power outage impacting the Bologna data center, or the crazy summer when NOMADS FTP took early retirement (too soon?).
\nWe now ingest these in real time and surface delays on our status and pipeline pages. We have backfilled the archive through early 2022.
\nUsing these, we asked yet another question: Can lead-group delays detect official advisories early?
\n![\"Lead-group](\"https://dynamical.org/assets/notes/advisory-overlay.png\")
Not conclusively, based on the archive we\nhave. We found 24 product/run matches with an opening agency advisory. Four\nalso experienced a lead-group delay, and those delays followed the agency\npost by roughly 2, 11, 96, and 196 minutes. That is too small a sample for a\nbroad conclusion, but it is enough to reject the claim that lead-group spreadf\nwas an earlier advisory detector in this period.
\nIf you can expose an inbound HTTP endpoint, webhooks are a good way to be notified of specific dissemination events or delay conditions. wxopticon POSTs a signed JSON body to your endpoint the moment a run crosses\na boundary you've subscribed to:
\n{\n \"event_id\": \"noaa-gfs/external-noaa-gfs-aws/2026-06-10T06:00Z/complete\",\n \"group_id\": \"noaa-gfs\",\n \"product_id\": \"external-noaa-gfs-aws\",\n \"product_label\": \"NOAA GFS forecast (AWS)\",\n \"init_time\": \"2026-06-10T06:00:00Z\",\n \"kind\": \"complete\",\n \"lead_group\": \"f384\",\n \"lead_group_label\": \"16d\",\n \"max_lead_hours\": 384,\n \"occurred_at\": \"2026-06-10T11:12:04Z\",\n \"latency_s\": 18724.0,\n \"timing\": \"on_time\"\n}Subscriptions are managed at\nstatus.dynamical.org/webhooks; access\nis currently allowlisted, so get in touch if\nyou'd like to try it. You can even attach a small sandboxed Python function that runs\nagainst the just-arrived dataset and shapes the payload or filters out deliveries you don't want.
\n![\"A](\"https://dynamical.org/assets/notes/wxopticon-slack.png\")
For those of you who hear the soft footfall of the IT team plodding imperceptibly, but threateningly, in the distance — coming closer, ever closer at the mention of webhooks, wxopticon publishes the same\nevents as a single JSON file you fetch on your own schedule, with no subscription or auth:
\nhttps://assets.dynamical.org/wxopticon/feed.json
\nIt's a product-keyed snapshot. Each product carries its most recent runs, and\nevery run nests the same discrete arrival events a webhook would deliver.\nActive agency advisories sit in a top-level list because one incident can span\nproducts or cycles; affected products carry references back to them.
\n{\n \"generated_at\": \"2026-06-10T09:45:00+00:00\",\n \"products\": {\n \"external-noaa-gfs-aws\": {\n \"label\": \"NOAA GFS (AWS)\",\n \"runs\": [\n { \"init_time\": \"2026-06-10T06:00:00+00:00\", \"status\": \"in_flight\",\n \"completion_pct\": 0.62, \"timing\": \"on_time\",\n \"events\": [\n { \"event_id\": \"noaa-gfs/external-noaa-gfs-aws/2026-06-10T06:00Z/progress/f240\",\n \"group_id\": \"noaa-gfs\", \"product_id\": \"external-noaa-gfs-aws\",\n \"product_label\": \"NOAA GFS forecast (AWS)\",\n \"kind\": \"progress\", \"lead_group\": \"f240\", \"lead_group_label\": \"10d\",\n \"occurred_at\": \"2026-06-10T09:41:03Z\", \"timing\": \"on_time\" }\n ] }\n ]\n }\n },\n \"advisories\": []\n}The nested events are identical to what a webhook carries (minus the\nsubscription id), so the client logic is the same: select the products you care\nabout, and dedupe on event_id across polls. The file is refreshed each five-minute cycle and\nserved with Cache-Control: max-age=5, stale-while-revalidate=10, so feel free to slam it.
We will continue to tune how "delayed" is determined. And as wxopticon now also ingests,\narchives, and cross-references official source advisories (an ECMWF\ndissemination delay, for example), we will continue to explore patterns.
\nwxopticon is a living but experimental piece of our infrastructure. If there's a source you'd like us to watch, or a boundary you wish you could subscribe to, let us know.
\n", "date_published": "2026-07-07T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-06-01/", "url": "https://dynamical.org/updates/2026-06-01/", "title": "AIFS ENS and validation reports", "content_html": "I was tempted, but alas Gefs (short for Gefsfry) did not make the cut for the name of my newborn son. He's here and everyone is healthy - so I'm getting back to it. Some catch-up:
\n![\"ECMWF](\"https://dynamical.org/assets/catalog-thumbnails/ecmwf-aifs-ens-forecast.jpg\")
The ensemble sibling to AIFS Single is now in the catalog. ECMWF AIFS ENS: 15-day global forecasts at 0.25°, 6-hourly steps, 51 members, updating every 6 hours. Archive starts 2025-07-02.
\n\nDid you know Alden has knuckle tats? They spell
\n ┌─┬─┬─┬─┐ ┌─┬─┬─┬─┐\n │R│I│G│O│ │R│O│U│S│\n └┬┴┬┴┬┴┬┘ └┬┴┬┴┬┴┬┘\n │ │ │ │ │ │ │ │\n══╧═╧═╧═╧══ ══╧═╧═╧═╧══\nTo live up to the knuckles, we are adding in-depth validation reports to each catalog entry. For each variable, you get three views: a null-fraction trace over time (so you can see any gaps in the archive at a glance), side-by-side spatial maps comparing our dataset against analysis at a sample timestamp, and time-series comparisons at two random geographic points. The spatial plots also include a value-distribution histogram so you can see at a glance whether the numbers look right.
\nFind it linked from each catalog page, or go directly — e.g. ECMWF AIFS ENS forecast validation.
\nWe are in the process of backfilling these for existing data products and we would LOVE your feedback!
\nThese are one layer of a broader validation process that runs automatically as each dataset updates. The goal is to give us (and you) enough visibility into the data that you can trust it before you build on it.
\nIf you've ever wondered why we started with the data products that make up our catalog, check out my blog post on the AWS Public Sector Blog: What if swapping your weather model was boring?
\nAWS Open Data has been an incredible partner as we've gotten dynamical.org up and running!
\nEnjoy the weather,
\nMM
\n", "date_published": "2026-06-01T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-04-24/", "url": "https://dynamical.org/updates/2026-04-24/", "title": "Time 2 Chunk (2.0), DWD ICON-EU, status for weather", "content_html": "Listen, my second child is arriving any minute now. This transmission will be brief. I haven't decided if I will name him Poseidon, Gefs, or Turbulence.
\nWe're all-in on Icechunk. Our existing Zarr v3 datasets will continue updating for 90 days, at which point we will sunset them. Consider them deprecated and update to use our Icechunk Zarrs by July 23, 2026.
\nThe new access pattern:
\nimport dynamical_catalog\n\n# Open a dataset as an xarray Dataset via its Icechunk repository\nds = dynamical_catalog.open(\"noaa-gfs-forecast\")To more easily ensure the right set of dependencies and to make an increasingly variable set of opening incantations more accessible, we made a python library to list and open datasets.
\nimport dynamical_catalog\n\n# List all available datasets\ndynamical_catalog.list()\n\n# Get the underlying Zarr store if you want even more control\nstore = dynamical_catalog.get_store(\"noaa-gfs-forecast\")\nds = xr.open_zarr(store)We are STAC-maxxers now. https://stac.dynamical.org/catalog.json contains ALL of the data used to create our documentation and drive the dynamical-catalog library. Point your robots at it for all the documentation they need to get going. STAC-as-skill.
\nDeutscher Wetterdienst's ICON-EU is now a cloud-optimized, live-updating Icechunk 2.0 Zarr. High-resolution 5-day forecasts over Europe, nested from DWD's global ICON model, at the 00/06/12/18 UTC cycles.
\nWhat makes this one extra fun: as far as we can tell, no other public archive of ICON-EU exists. DWD only keeps a short rolling window on their own servers, so we'll keep growing the archive from here.
\nHuge thanks to:
\nFind it at:
\n\nEnjoy the weather,
\nMM
\n", "date_published": "2026-04-24T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-04-17/", "url": "https://dynamical.org/updates/2026-04-17/", "title": "dynamical.org - icechunk 2.0 upgrade cometh", "content_html": "Short one today — a quick housekeeping note for anyone pulling from our icechunk data products.
\nIcechunk 2 is out! If you're using our datasets via Icechunk, please update your Python library before Thursday April 24:
\npip install icechunk --upgrade\nor
\nuv add icechunk --upgrade\nNote that icechunk 2 requires Python 3.12 or higher. If you're on an older runtime, now's the time to bump it.
\nWe'll be targeting the v2 format going forward.
\nEnjoy the weather,
\nMM
\n", "date_published": "2026-04-17T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-03-31/", "url": "https://dynamical.org/updates/2026-03-31/", "title": "Rain, radar, and reckless optimism.", "content_html": "Salutations, wxers. To balance out all the April 1 corporate tomfoolery I'm going to be less of a goofball today. Let's see if I can keep it STRICTLY BUSINESS.
\nLots to report since my last update. The dynamical(.org) core is a-running.
\n![\"ECMWF](\"https://dynamical.org/assets/catalog-thumbnails/ecmwf-aifs-single-forecast.jpg\")
The AI weather model from the European Centre has entered the catalog. ECMWF's AIFS Single — 15-day global forecasts at 0.25°, 6-hourly steps, live updating. This is the single-trace (deterministic-ish?) run of ECMWF's machine learning forecast system.
\n\nAnd of course, don't miss the release trailer: Youtube | LinkedIn
\n![\"NOAA](\"https://dynamical.org/assets/catalog-thumbnails/noaa-mrms-conus-analysis-hourly.jpg\")
New CONUS RADAR dataset! MRMS is Multi-Radar Multi-Sensor — NOAA's merged radar and gauge precipitation analysis over CONUS, hourly, back to late 2014. If you've ever wanted to answer "how much did it actually rain here, at this exact hour" without downloading 47 files from 3 different FTP servers... this is for you.
\n\nRelease trailer: Youtube | LinkedIn
\n![\"Requests](\"https://dynamical.org/assets/requests_2026-04-01.png\")
We passed the 100M monthly request mark. And then we passed the 200M monthly request mark. Which is either a sign that the data is useful or that three of you are running massive autoresearch.md deployments.
\nEither way, it's exciting and we appreciate everyone who's sent us what you're building!
\nWe're testing a simple Python library: dynamical-catalog. The idea is to make discovery and loading even simpler — browse the catalog, pick a dataset, and get an xarray Dataset or Store back without copy-pasting URLs. We'll handle picking the best Store backend. It's early days. If you try it, let us know what breaks.
\nEnjoy the weather,
\nMM
\n", "date_published": "2026-04-01T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-03-10/", "url": "https://dynamical.org/updates/2026-03-10/", "title": "Warmer temps, datasets heating up.", "content_html": "It's Spring and unlike hobbits (I'm reading LOTR), who would be thinking about gardens and second breakfast and whether Farmer Maggot's mushroom crop will come in early, we are staring at grib files. Just kidding, we are walking outside enjoying the warm weather ... while talking on the phone about grib files.
\nWe've been busy with a whole load of updates.
\nOur old noaa-gfs-analysis-hourly was our "Hello World" — 4 variables, static, Zarr v2.
It has been given a lovely retirement party and replaced with the new GFS analysis.
\n| \n | \n |
|---|---|
| Spatial domain | \nGlobal | \n
| Spatial resolution | \n0.25° | \n
| Time resolution | \n1 hour | \n
Four more sweet years. High Resolution Rapid Refresh has been retroactively retained through repeated re-retrieval - and now, re-released for a more uh... robust retrospective.
\nIt previously started in 2018 and now reaches back to 2014. The early years (HRRR v1/v2 era) have more missing source files — NaN values where data are unavailable — but that's the archaeology tax.
\nIcechunk note: we've updated to v0.2.0. Please point your code at noaa-hrrr-analysis/v0.2.0.icechunk/. We'll be stopping updates to v0.1.0 on 2026-03-19 and removing the archive after that.
By popular demand we've been adding variables to existing datasets. ECMWF, GEFS, and HRRR have all received new variables recently. We drew the line at will_the_sports_be_cancelled_2m and ideal_dog_walk_surface but almost everything else is in there. Check the catalog pages for the full list.
I tried (and tried [and tried]) to find a good [citation needed] open source of obs once ISD was decomissioned and GHCNh made me want to cry enough tears to register on the precip gauge. Alas.
\nThose tireless little weather stations at airports across the US are available as a GeoParquet dataset. 1940ish to present. This is our first non-model dataset and it's 100% experimental, but it updates every hour and it's what the scorecard is built on.
\nAs a first step towards cloud-optimized ICON-EU Zarrs, we've started archiving DWD's gribs on Source Coop — DWD only maintains a short rolling window on their servers, so someone had to hoard them. Data hoarding is our North Star!
\nBig thanks to Jack Kelly of Open Climate Fix for leading this effort and to the Patrick J McGovern Foundation for supporting the work.
\nEnjoy the weather
\nMM
\n", "date_published": "2026-03-10T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-01-23/", "url": "https://dynamical.org/updates/2026-01-23/", "title": "January cornucopia: AMS | HRRR analysis | more", "content_html": "What say the pantheon of forecasts? Shall AMS travel be a disaster?
\n\n\nAs the flakes that fall thick upon a winter's day, when Zeus is minded to snow and to display these his arrows to humankind - he lulls the wind to rest, and snows hour after hour till he has buried the tops of the high mountains, the headlands that jut into the sea, the grassy plains, and the tilled fields of men...
\n
"What Happens When Weather Data is Easy to Use?". Join Alden during the Environmental Data Access and Its Future: Open Data for Open Science Session on Monday, January 26 at 2:15 PM CT to explore how accessible weather data transforms research, operations, and decision-making across the atmospheric sciences community. Shoot him an email if you want to meet up and chat!
\nHRRR analysis (2018-09-16 to present) is now available in the catalog, including the Icechunk preview.
\n\n| \n | \n |
|---|---|
| Spatial domain | \nContinental United States | \n
| Spatial resolution | \n3 km | \n
| Time domain | \n2018-09-16 00:00:00 UTC to Present | \n
| Time resolution | \n1 hour | \n
![\"A](\"https://dynamical.org/assets/hrrr_analysis_precipitation_surface_example.png\")
Icechunk previews for our datasets are now listed on the Registry of Open Data on AWS! Check out the Icechunk example usage in our docs for IFS ENS, GEFS, GFS, and HRRR forecasts to get started. With the addition of GEFS forecast and analysis we now have an identical, live updating Icechunk Zarr for all live updating standard Zarr v3 datasets in our catalog!
\nThe "Union of Organizations Concerned That Earth Data Is Generally Too Hard To Work With" (still workshopping this title, but it has a good ring to it) just became more powerful - Earthmover launched their Marketplace. (Congrats team)
\nAnd the aforementioned dynamical.org Icechunk catalog is within!
\nShrouded in mystery and intrigue, the steering committee guides the helm from the shadows. To lift the veil is to challenge authority itself, and... hm? What's that? The meeting notes are public?! Oh.
\n\nShout out to all the contributors to cf_xarray which both a) now works great with our datasets and b) is part of our test suite to ensure all new datasets are CF compliant from the start. Thanks to our steering committee for guiding us towards more interoperable standards.
\nMM
\n", "date_published": "2026-01-23T00:00:00Z" } , { "id": "https://dynamical.org/updates/2026-01-08/", "url": "https://dynamical.org/updates/2026-01-08/", "title": "Preview of dynamical.org Icechunk Zarrs are now listed on the Registry of Open Data on AWS!", "content_html": "Check out the Icechunk example usage in our docs for IFS ENS, GFS, and HRRR forecasts to get started or browse these datasets on the Registry of Open Data on AWS.
\nWhy Icechunk?
\nWhen Icechunk 2 is released we will make two small breaking changes (and notify you about them here):
\nSpecial thanks to the AWS Open Data Sponsorship Program for supporting the storage and distribution of these datasets.
\n", "date_published": "2026-01-08T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-11-20/", "url": "https://dynamical.org/updates/2025-11-20/", "title": "🟢 ECMWF IFS ENS live and updating", "content_html": "Colder temps may be here (where I live, at least), but ensembles are so hot right now. Can we agree on that? No? Ha ha get it!?
\nAs of this week The European Centre's IFS ENS forecast is live and updating in our catalog. See the catalog for init times, spatiotemporal domain, and the variable list.
\n![\"ECMWF](\"https://dynamical.org/assets/ecmwf_ifs_ens_tmp.png\")
I suppose with that HRRR "release trailer" I've signed myself up for something completely unsustainable -- but until that time comes, when the catalog entries cometh faster than I can splice together sci fi video clips... -- ENS has a release trailer too.
\nIcechunk versions are available to select groups for testing (and will likely be our default in the future). If you want to try them out please respond.
\nI've actually been a little surprised at the lack of Icechunk inquiries. Are y'all even drinking the cutting edge data format koolaide?!
\nSending from an airship above the clouds. Enjoy the weather. End transmission.
\nMM
\n", "date_published": "2025-11-20T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-09-29/", "url": "https://dynamical.org/updates/2025-09-29/", "title": "Wake up, babe. A new zarr just dropped.", "content_html": "The last time I wrote you all, the summer sweltered, determinism was still alive (now it is dead, and only 11 AI weather papers had been published in Nature (now the total is up to 188,732).
\nIf I keep it up y'all are never going to take me seriously. But lo, some seriously new stuff!
\n![\"HRRR](\"https://assets.dynamical.org/media/hrrr_wind_speed_10m_2025-05-25T00_conus.webp\")
HRRR forecasts are live and updating! Check out the release trailer (lol) here.
\nGive it a try! It'll be like that time in high school when you played too much Elder Scrolls Oblivion during a long weekend, and when you finally stepped outside into nature you thought to yourself "woah, these graphics are incredible."
\nA massive thanks to Alden, Tony, Alex and Sam for getting this one out there!
\n![\"Screenshot](\"https://assets.buttondown.email/images/e40436c1-8757-411a-90f6-4d939577d0bd.png?w=320\")
We've also launched Earthchunk Icechunk versions of the GFS and HRRR forecasts. It's internal only until we sort out a few final details, but if you'd feel the need for speed and are prerelease-tolerant let us know and we can hook you up.
In case you missed it we have a podcast now! We so far have covered:
\n5 papers down, 188,727 to go! No new AI weather papers allowed.
\nHey, wouldn't you know it. Having stuff in this here catalog makes it easier to build things. I was tired of stats against reanalysis as "truth" so Alden and I hacked out a very simple scorecard. When I say simple, I mean simple. For now it's just RMSE for 2m temp and precip. Of course, a few weeks after this went live, ISD-lite was killed in favor of GHCN so I gotta flip that over to keep things live-updating! URGH.
\n![\"Screenshot](\"https://assets.buttondown.email/images/38062496-0e9f-4f28-8d72-5db0922d307d.png?w=300&fit=max\")
Enjoy the weather.
\nMM
\n", "date_published": "2025-09-29T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-07-02/", "url": "https://dynamical.org/updates/2025-07-02/", "title": "🟢 GFS is live", "content_html": "GFS is GEFS' less ambiguous older sibling. The archive is processed and the updater is chugging: 4 years of 16 day forecasts, hourly step, released 4 times a day. GFS might be wrong, but at least it's confident!
\nA note from Alden, Chef de cuisine
\n\n\nTonight, I've prepared for you a dataset I think you will enjoy. Check out the example notebook linked in the docs to make your own interactive animation of an atmospheric river with buttery smooth hourly resolution. The way atmospheric rivers move reminds me of how rivers move on land, only on a time scale of hours not years. Served as a semifreddo with a peach compote.
\n
![\"Visual](\"https://dynamical.org/assets/gfs_atmospheric_river_feb_2024_precipitable_water_atmosphere.webp\")
\n The dynamical.org steering committee met and talked about [1] Data and [2] Things About Data Beyond My Comprehension. Read the summary notes
\nDid you catch episode number two of weathering? This time we talked about Aardvark Weather. As Marta put it: "Where does the end begin? Where does the end end? And what the HELL is happening in the middle?"
\nCheck it out. Like (in your heart) and subscribe (to our obsessions).
\n", "date_published": "2025-07-02T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-06-14/", "url": "https://dynamical.org/updates/2025-06-14/", "title": "introducing a new pod; GFS forecasts incoming", "content_html": "Not even my proprietary UniverseCast™ Reality Ensemble Forecast⁺ could have predicted this. The most unlikely of scenarios: A podcast? Yes, a podcast. And it's called Weathering.
\nWe are trying to make sense of the current wave of progress occurring in weather forecasting.
\nEach episode, we focus on a paper or group of papers and try to take a intertextual approach -- clicking on mental hyperlinks to help us understand and put into context. The format is inspired by the only two podcasts I regularly listen to (Future of Coding and Jed Sundwall's Techs on Texts).
\nEpisode 001: The limits of predictability
\nOur maiden voyage dives into a fresh-off-the-press paper from the University of Washington: Testing the Limit of Atmospheric Predictability With a Machine Learning Weather Model.
Check it out, and let us know if there's a paper you think we should cover next.
\nIn normal dynamical.org news, operational GFS forecasts are landing in the catalog imminently. We previously did an analysis as our first demo, so this will be a nice upgrade to tie the bow on the GEFS/GFS family.
\nAfter that, the catalog will be expanding rapidly! Stay tuned.
\nHave a good weekend,
\nMM
⁺ Fictional? Or extremely limited access to drive scarcity? You decide.
\n", "date_published": "2025-06-14T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-04-29/", "url": "https://dynamical.org/updates/2025-04-29/", "title": "GEFS analysis is live!", "content_html": "Well, technically it's been live for a few days. Check out our new GEFS analysis, which is derived from the GEFS forecast catalog item.
\n![\"image.png\"](\"https://assets.buttondown.email/images/3f8a5ed8-29ca-4b60-88d3-690add9bfa80.png?w=960&fit=max\")
Both are live updating, and the analysis was designed to work easily with the forecast dataset. For example, you can develop your algorithm/model/sentient AI television weatherperson on the analysis, and then deploy it live with the forecast dataset. COOL!
\nStriving to create weather data experiences as satisfying as when you peel citrus in one contiguous piece. You know what I mean :)
\nEnjoy the weather,
\nMM
\n", "date_published": "2025-04-29T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-03-20/", "url": "https://dynamical.org/updates/2025-03-20/", "title": "1 Million Requests!", "content_html": "While modest on the scale of Computers, 1M is not bad for our baby’s first week in the world. The gigabytes are flying and we’ve received requests from 51 countries over the last few days. No single person has downloaded the whole dataset, so thank you but also WHAT ARE YOU WAITING FOR?!
\nA reminder that the GEFS dataset is Zarr v3, so don’t forget to update your Xarray>=2025.1.2 which includes v3 support!
\nWith GEFS complete, we are already working on our next slate of catalog additions. Our Phase 1 will include the following datasets:
\nForecasts
\nAnalyses
\nThree ways you, dear reader, can help:
\n[1] Financial support:
\nWe estimate that making Phase 1 operational & publicly accessible will run around $100,000 per year with the cost growing by approximately 30% year over year as the datasets get inevitably larger. Pennies compared to the value, in my humble opinion!
\nSo… we are seeking our first cohort of sponsors to make these forecasts and analyses a reality. If you belong to an organization that benefits from, and is able to support our work, we’d love to hear from you. Reply to this email to learn more about our plans for fiscal sponsorship.
\n[2] Publicize these datasets as they’re released! Tell your coworkers, post on whatever social media we are using these days, tell your parents over dinner, teach your toddler about scones and pancakes.
\n[3] Write code? Not quite yet but soon. We are testing to see if our open source code so far is sufficiently clear to have other folks pitch in and write reformatters for new datasets. If all goes well we may solicit help from your big brain.
\nHelp us out, and especially [1] so that there are NO BLOCKERS, ONLY WEATHER.
\n\nMM
\n", "date_published": "2025-03-20T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-03-12/", "url": "https://dynamical.org/updates/2025-03-12/", "title": "About GEFS-fing Time & Introducing Our Steering Committee", "content_html": "Weatherlings, the time has come for our first non-demo release. Weighing in at 115TB compressed and 815TB (!) uncompressed, GEFS ain’t messing around.
\nLast year, we launched dynamical.org into the world and watched as hundreds of weather data nerds emerged (sublimated?) from the woodwork. Our community has been growing steadily, united by a shared mission to make weather data Spark Joy.
\nI am thrilled to announce the latest addition to Dynamical’s catalog: NOAA's Global Ensemble Forecast System (GEFS) 35-day forecasts!
\nThe Specs:
\n| Parameter | \nDetails | \n
|---|---|
| Spatial Domain | \nGlobal | \n
| Spatial Resolution | \n0-240 hours: 0.25 degrees (~20km), 243-840 hours: 0.5 degrees (~40km) | \n
| Time Domain | \nForecasts initialized 2020-10-01 00:00:00 UTC to Present | \n
| Time Resolution | \nForecasts initialized every 24 hours | \n
| Forecast Domain | \nForecast lead time 0-840 hours (0-35 days) ahead | \n
| Forecast Resolution | \nForecast step 0-240 hours: 3-hourly, 243-840 hours: 6-hourly | \n
Best of all, it's operationally updated with minimal latency and available in the Zarr v3 format that plants crave:
\nimport xarray as xr\n\nds = xr.open_dataset(\"https://data.dynamical.org/noaa/gefs/forecast-35-day/latest.zarr?email=optional@email.com\")Note, added 2026-08-06: the data.dynamical.org URL above stops working after August 31, 2026. This dataset is still public and still updating — it is now an Icechunk repository, opened as dynamical_catalog.open("noaa-gefs-forecast-35-day"). See the migration guide for the supported access patterns.
We are already using this new dataset for R&D at Upstream Tech, and we can’t wait to hear how you use it! We've included documentation and example notebooks in the catalog entry to get you started. If you hit snags or have feedback, drop us a line at feedback@dynamical.org.
\nWe’ll be holding our first dynamical.org office hours Thursday Mar 20, 2025 at 10-11am PDT / 1-2pm EDT / 5-6pm UTC. Join this Google Meet to ask questions or give feedback.
\nWe're excited to announce the formation of the dynamical.org Steering Committee! This dream team will:
\nCurrent Members:
\nBy bringing together this collective expertise, we're ensuring that Dynamical.org remains relevant, impactful, and true to its mission.
\nShine on (lol),
\nMM
\n", "date_published": "2025-03-12T00:00:00Z" } , { "id": "https://dynamical.org/updates/2025-01-10/", "url": "https://dynamical.org/updates/2025-01-10/", "title": "Alden @ AMS; GEFS zarrs?!", "content_html": "CALLING ALL WEATHER DWEEBS! Bracing for a winter storm here in Nashville, where I moved to from Boston in order to escape winter storms. 3 to 7 inches they say - good thing I brought the XC skis!
\nFrom the look of it the vast majority of our subscribers will be at AMS. We are excited to see you there!
\nAlden will be presenting at the 105th Annual Meeting of the American Meteorological Society (AMS) along with our first fiscal sponsor at Hydro-Québec! Their talk, titled “Fast, Easy, Open: Sustaining Live Updating, Analysis Ready, Cloud Optimized Weather Data Archives,” will highlight the vision behind dynamical.org and the drive to create and sustain ARCO weather datasets.
\nIf you’re at AMS, find Alden and Upstream Tech's Phil Butcher.
\nWe’ve made some big strides toward releasing a complete archive and live-updating GEFS dataset to the catalog:
\n>>> ds.u100\n<xarray.DataArray 'u100' (init_time: 1, ensemble_member: 4, lead_time: 5,\n latitude: 721, longitude: 1440)> Size: 83MB\ndask.array<open_dataset-u100, shape=(1, 4, 5, 721, 1440), dtype=float32, chunksize=(1, 4, 5, 73, 72), chunktype=numpy.ndarray>\nCoordinates:\n * ensemble_member (ensemble_member) uint16 8B 0 1 2 3\n expected_forecast_length (init_time) timedelta64[ns] 8B 35 days\n ingested_forecast_length (init_time) timedelta64[ns] 8B 35 days\n * init_time (init_time) datetime64[ns] 8B 2025-01-08\n * latitude (latitude) float64 6kB 90.0 89.75 ... -89.75 -90.0\n * lead_time (lead_time) timedelta64[ns] 40B 0 days 00:00:00...\n * longitude (longitude) float64 12kB -180.0 -179.8 ... 179.8\n valid_time (init_time, lead_time) datetime64[ns] 40B 2025-...\nAttributes:\n long_name: 100 metre U wind component\n short_name: u100\n standard_name: eastward_wind\n step_type: instant\n units: m/sWoah nelly, check out those attributes!
\nWe’re testing out operational updates in our compute cluster THIS WEEK. Once that’s complete, we’ll be ready to push live, real-time updates to the catalog. So close, we can almost taste the churros/pancakes/scones (ha ha ha get it).
\nFinally - we'll be launching our steering committee this quarter. Thanks to all those who responded with interest in helping us achieve our mission.
\nCatch you in the clouds.
\nMM
\n", "date_published": "2025-01-10T00:00:00Z" } , { "id": "https://dynamical.org/updates/2024-09-26/", "url": "https://dynamical.org/updates/2024-09-26/", "title": "September update", "content_html": "Ahoy weatherheads! It’s been a busy few months for us as we shift gears from dreaming to building.
\n![\"GEFS](\"https://dynamical.org/assets/gefs-dlwrf-img-only.png\")
\nGEFS ensemble average downward long-wave radiation flux.A hearty thank you to everyone who filled out our GFS survey. Common themes included the value of live updating and forecast data, and a clear preference for reading data with python + xarray. That aligned with our expectations (wonderful) and is reflected in our roadmap above. We did see more desire for long history (re)analysis data than we had anticipated -- we'll be moving up a true reanalysis dataset in priority thanks to this feedback. Reply to this email if there's a specific reanalysis you'd like!
\nWe are officially launching our steering committee, which will be comprised of industry users and experts in the technical and scientific domain. The committee will meet quarterly to evaluate and influence the catalog and open-source roadmap and advise potential collaborations and partnerships.
\nIf you’re interested in joining or nominating someone else, please respond to this email or email marshall@dynamical.org!
\n\n\nMy souls, how the wind did scream along! And every second or two there’d come a glare that lit up the white-caps for a half a mile around, and you’d see the islands looking dusty through the rain, and the trees thrashing around in the wind; then comes a h-whack!—bum! bum! bumble-umble-um-bum-bum-bum-bum—and the thunder would go rumbling and grumbling away, and quit—and then rip comes another flash and another sockdolager.
\n
I’m reading Adventures of Huckleberry Finn, which has some stellar meteorological descriptions. Should we petition NOAA to make their detailed forecasts use “h-wack!-bum! bum!” and “sockdolager?” I think yes.
\nWe enjoyed Dr. Jack Kelly’s post on the vision this community shares, where he outlined a complimentary route to solving major pain points. Thanks for the dynamical.org shoutouts, Jack! Definitely check it out here: https://openclimatefix.org/post/lazy-loading-making-it-easier-to-access-vast-datasets-of-weather-satellite-data
\nWe were thrilled to see our pals at Brightband announce themselves to the world. The field of AI meteorological forecasts is progressing at quite the clip - three cheers for more open source models! We also enjoyed Zeus AI's cool work synthesizing multiple, sparse data sources into a single gap filled analysis state.
\nEnjoy the weather!
\n", "date_published": "2024-09-26T00:00:00Z" } , { "id": "https://dynamical.org/updates/2024-07-09/", "url": "https://dynamical.org/updates/2024-07-09/", "title": "Let there be zarr: GFS hourly analysis is live", "content_html": "Last week, we announced dynamical.org to the world. The response was unexpected*: in just seven days, over a hundred nerds registered themselves as testers with many more following along for the ride.
\nApparently, we are not alone in our frustration with yesterday's weather data. Together we can create a future of well-organized data about our most beloved chaotic system!
\nToday, we are excited to announce the first entry into dynamical’s catalog: an hourly GFS analysis from 2015-01-15 to present.
\nTesters! Fire up your rigs - in the catalog entry above, we've included documentation and an example notebook. If you run into any issues, email us at feedback@dynamical.org. As our very first release, we’re expecting there will be lots of additions and refinement based on community input.
\nOur pals at Source Cooperative are generously providing storage for this first catalog entry. Their team is wonderful and their mission is badass - check them and the broader Radiant Earth organization out.
\nEnjoy the weather^!
\nMM
\n* or expected, depending on how long you've been working with yesterday's weather data
\n^ courtesy of Alex M.
\n", "date_published": "2024-07-09T00:00:00Z" } , { "id": "https://dynamical.org/updates/2024-06-25/", "url": "https://dynamical.org/updates/2024-06-25/", "title": "Introducing dynamical.org", "content_html": "Introducing dynamical!
\nThank you for being the first to follow our journey making weather and climate data more accessible. We are humbled by the response. Our mission clearly strikes a chord - weather data today is a pain to work with.
\nOver one hundred people (!?) signed up to test the first versions of the data we endeavor to release. And that’s without having spilled the beans on the best parts 😉
\nSo what’s the plan? First, check out Alden’s fantastic introduction to dynamical.org (video link). Second, read about how we are designing dynamical to be a long-term steward and rock-solid foundation on which to build.
\nWe are aiming to release our first public test next week in collaboration with Source Cooperative: a GFS analysis with a long historical record. If you are interested in testing this out and haven’t already told us, reply to this email. We plan to fast-follow with documentation and a data catalog with our roadmap.
\nEnjoy the weather!
\nMM + AKS
\nP.S. Please forward this email to folks working with weather data and consider amplifying our announcement on linkedin / fedi!
\n", "date_published": "2024-06-25T00:00:00Z" } ] }