The Big Geodata Newsletter provides a quick monthly update on the recent news and developments in the big geodata domain. We try to keep it concise, informative and interesting. If you want to be informed about the developments in the rapidly changing landscape of big geodata please subscribe:
In this issue you will find information on NASA's TiTiler-CMR, enabling on-demand visualization of EO data; TerraMesh, a multimodal EO dataset for training large-scale foundation models; comparisons between Zarr and Cloud-Optimized GeoTIFFs for cloud-native geospatial workflows; OSMlanduse, a high-resolution land-use dataset of the EU combining OpenStreetMap and Sentinel-2 data; and how citizen science combined with MODIS satellite data is revealing large-scale patterns in bird population trends.
In this issue you will find information on managing large-scale geospatial data using the new Microsoft Planetary Computer Pro, the new fully open-source release of cuPyNumeric, accelerating Apache Parquet scans on Apache Spark using GPUs, Obstore - a solution for high-throughput I/O in cloud object storage, and OpenspaceGlobal, which maps urban open spaces across 169 megacities.
In this issue you will find information on Geospatial Reasoning by Google Research for integrating generative AI with geospatial foundation models, the Sedona STAC Reader for streamlining large-scale geospatial data analysis, the advantages of tensor-based models over tables for scientific big data, NVIDIA Earth-2’s approach to AI-driven flood risk assessment, and the LUIcube dataset tracking global land-use intensity from 1992 to 2020.
In this issue you will find information on RaQuet specification for storing and querying raster data efficiently, discover TerraTorch for fine tuning of geospatial foundation models, learn about new Python Zarr 3.0 insights and performances, Geoparquet Downloader QGIS plugin providing faster access to geospatial data, and GEDTM30 - a novel global digital terrain model.
In this issue, explore GeoJupyter for interactive geospatial computing within Jupyter, discover Amadeus for simplifying large-scale environmental data analysis in R, and learn about NCZarr, which connects netCDF and Zarr for scalable, cloud-native scientific data management. You’ll also find insights into GeoCore, a powerful and scalable framework for geospatial machine learning, and EarthView, a large-scale remote sensing dataset designed to advance self-supervised learning in Earth observation applications.
In this issue you will find information on GeoArrow and GeoParquet for efficient analysis of large geospatial datasets, explore advancements in Zarr Python 3.0 for scalable data storage, uncover insights on IBM's Geospatial TensorLakehouse for enhancing geospatial AI analysis, and learn about cutting-edge solutions like NVIDIA Earth-2 for solar irradiance prediction and the hybrid quantum-classical convolutional neural network (QC-CNN) for EO data.
In this issue you will find information on GenCast that predicts weather conditions with state-of-the-art accuracy, VirtualiZarr to create virtual Zarr stores using xarray syntax, SLURM-style job arrays on the Cloud with Coiled, TorchSpatial - a python package for spatial representation learning and geo-aware model development, and Population Dynamics Foundation Model by Google Research.
In this issue you will find information on XDGGS for planetary-scale data cube computations with Discrete Global Grid Systems, Arkouda for large-scale geocomputing by using Pangeo stack, DMR++ for easy access to HDF4/5 data on the Cloud without reformatting, accelerated data analytics workflows with RAPIDS cuDF, and Vector Data Cubes for geospatial insights with spatiotemporal vector data.
In this issue you will find information on Icechunk for transactional cloud-native data storage, Marimo for reactive notebooks for dynamic code cell updates, Coiled’s call for community input on a geospatial benchmark suite, STAC GeoParquet for efficient cloud-native geospatial data handling, and Fields of the World - a multinational dataset for agricultural boundary segmentation!
In this issue you will find information on how millions of Dask nodes are managed in production, cloud-native access to NetCDF datasets by using Kerchunk, insights into EuroCrops - an open-source dataset for European crop analysis, and advancements in geospatial foundation models for image analysis, particularly enhancing NASA-IBM Prithvi's domain adaptability!
In this issue you will find information on 10th anniversary of PDOK and its new services, recent developments in efficient creation of multi-scale Zarr pyramids to boost big data storage and access, global urban green space dataset covering more than 1000 cities, and the GeoAI challenges to accelerate the implementation and monitoring of SDGs.
In this issue you will find information on recent foundational models leveraging EO datasets, new meta-learning frameworks, a global 30 m land-cover product and on an interesting challenge on crop yield prediction!
In this issue you will find information on HyperCoast, a new Python package for hyperspectral data; Coiled’s benchmarking of DataFrame technologies; news on the retirement of Microsoft’s Planetary Computer Hub; and on SIRCLE and SWAG, two new models that tackle the challenges of processing petabyte-scale EO data.
In this issue you will find information on Fiboa - a project standardising agriculture field boundary data, Sentinel-2 Super-Resolution model, Cubed - an alternative backend for Xarray, and CPU and GPU optimizations for LiDAR data processing.
In this issue you will find information on new ML-ready dataset standards Croissant and MajorTOM, GPU accelerated data analytics with NVIDIA RAPIDS cuDF, and a new analysis ready platform for Earth Video Cubes.
In this issue you will find information on PMTiles - a novel archive format for tiled data visualization, DiffusionSAT - a large generative model trained on high-resolution remote sensing datasets, the One Billion Row Challenge, and a MOOC on Cubes & Clouds.
In this issue you will find information on utilizing GDAL with AWS EMR-Serverless, GridMesa: an adaptive grid approximation model to handle large spatial data, and on creating EO data cubes with Cubo and XEE.
In this issue you will find information on a new method to visualise in-memory raster data using Leafmap, the Google-Microsoft combined buildings footprints dataset, cloud-optimised Geo-Zarr format, and a state-of-the-art global high-resolution canopy height model.
We wish all readers a very Happy New Year! In this issue you will find information on the new Spatial Extension for DuckDB, recent query optimization efforts for Dask Expressions, the Copernicus Data Space Ecosystem and state-of-the-art global database of 2 million training points to map landcover change! We also introduced you to Indupriya Mydur, who recently joined as a student assistant with CRIB.