- DINOv3: checkpoint each of 24 transformer blocks (recompute on backward)
- DGTRS-CLIP: checkpoint each of 12 transformer blocks
- Enables batch_size=24 on RTX 4090 (was 8 without checkpointing)
- Peak VRAM: 20.3 GB at bs=24 (was OOM at bs=16 before)
- ~20-30% slower per step, but 3x more in-batch negatives (23 vs 7)
- Enabled by default (gradient_checkpointing=True in config)
- Update README with VRAM benchmarks and checkpointing docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Single DINOv3 WEB for both drone and satellite branches (shared_encoder=True default)
- One set of MONA adapters instead of two: 7M trainable vs 14M
- Total params: 438M (was 748M), trainable: 10.6M (was 17.6M)
- Asymmetric mode still available via shared_encoder=False
- Add gradient accumulation (grad_accum_steps, --grad-accum CLI flag)
- Update model summary in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New config field grad_accum_steps (default=1, no change in behavior)
- Loss scaled by 1/accum, optimizer step every N micro-batches
- Scheduler counts optimizer steps (not micro-batches)
- CLI flag --grad-accum for override
- Document gradient accumulation and in-batch negatives in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that dataset stores only positive drone-satellite pairs,
negatives are formed automatically via InfoNCE similarity matrix
within each batch (B-1 in-batch negatives per query).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Show MONA adapters (7M) and LoRA (147K) in branch diagrams
- Add retrieval/loss block with temperature and CE weights
- Add diagnostics pipeline block (per-batch CSV, Grad-CAM, profiler, grad norms)
- Add gtauav_image_heavy.gin to structure
- Split CSV row in diagnostics table into per-batch and per-epoch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gitea 1.25 renders math blocks via ```math fence reliably.
Replaced $$...$$ with ```math blocks, inline math kept as backtick code.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through
Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3
Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values
11.1M trainable / 734M total (1.51%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.
CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
