NASA-IBM-Lunar-Foundation-Model

A multimodal, multi-resolution foundation model for lunar remote sensing.

A ViT-B encoder–decoder trained from scratch on SomBench — ~2 million co-registered lunar tile bundles spanning 11 modalities at two spatial scales (LROC NAC ~1 m/px and LROC WAC ~100 m/px). It adapts the TerraMind masked-token recipe to the Moon with two extensions:

1. Acquisition geometry as explicit context — per-tile illumination angles, solar-frame anchors, and tile footprint are sequence-tokenized encoder inputs. Lunar surface appearance is governed more by illumination geometry than by intrinsic surface variation, so this hands the model the dominant confound instead of making it recover a quantity already recorded for every tile.
2. Joint mixed-resolution pretraining — NAC- and WAC-anchored tiles train in a single mixed-batch loop at native resolution, so one set of weights covers both resolution families across a 100× scale gap.

FlexiViT patch embedding lets the checkpoint be fine-tuned at other patch sizes without retraining the backbone; modality-wise tokenization lets you drop or add modalities at fine-tuning time.

📄 Full method, ablations, and per-baseline results: paper · 💻 Fine-tuning code: GitHub

Modality-wise tokenization (left/center-left) feeds a TerraMind-style any-to-any masked-token objective (center-right); the pretrained encoder then supports multimodal generation, multimodal fine-tuning, and FlexiViT fine-tuning at new patch sizes (right).

Model at a glance

Repository contents

backbone/         pretrained checkpoint and config to be used for finetuning/generation
tokenizers/       checkpoints and configs for all tokenizers

Intended use

A reusable representation for lunar remote sensing: fine-tune or LoRA-adapt the encoder for detection, segmentation, and dense regression on LROC WAC/NAC imagery and co-registered terrain products, with modality subsets and patch grids that differ from pretraining. Any-to-any multimodal generation is supported as a qualitative probe of learned cross-modal structure.

Out of scope. Not a scientific-grade generative product — generated fields are not calibrated predictions and are no substitute for instruments, stereo photogrammetry, or geodetic solutions. The model maintains no geodetic reference frame: it recovers local structure but not absolute values (DTM shape is right at a shifted elevation offset; generated lat/lon can be off by tens of degrees). Not validated for operational decisions such as landing-site certification or hazard clearance. Ice-prospectivity outputs regress a knowledge-driven fuzzy-overlay prospectivity map, not measured ice. Not evaluated beyond the Moon or on products absent from SomBench.

Modalities

Dense modalities are patch-embedded and tokenized as images. A sample is either NAC- or WAC-centered — the two families are never mixed within a sample.

Aspect is stored as a sine–cosine pair to avoid angular wrap-around.

Two sequence-like modalities carry scalar context, binned and stringified before text tokenization (e.g. with 0.25° longitude bins, a tile-center longitude of 87.49° becomes C_LON=87.25-->87.50):

• Optical metadata (8 fields) — solar incidence, emission, phase and azimuth angles; sub-solar and tile-center coordinates; ground sampling distance. Available at inference whenever the image is, so no label leakage.
• Static-map context (28 fields, tile-footprint averages) — Diviner thermophysics (TREG, TBOL, ROCK_ABUND, HPAR, DICE), LOLA products (ROUGHNESS, PSR, AVG_ILLUM, ALBEDO), Mini-RF radar (MINIRF_*), Kaguya mineralogy (MI_*, SW_FE, SP_MINER), LROC WAC (nr643, TIO2, WAC_*), GRAIL (GRAVITY), and Lunar Prospector (HYDROGEN). Coverage varies by instrument footprint, from near-global to polar-only (~6,900 polar tiles per track).

Usage

Fine-tuning runs through TerraTorch, configured from YAML. The companion repo adds the SomBench benchmark datamodules, backbone wrappers, support for registering new input modalities into the pretrained backbone, and FlexiViT patch-embedding interpolation.

git clone https://github.com/NASA-IMPACT/NASA-IBM-Lunar-Foundation-Model && cd $_
pip install -e .
terratorch fit --config configs/finetune/ice_pro