PROBLEM: GTA-UAV has overlapping satellite crops (partial IoU).
Standard InfoNCE with diagonal targets treated valid matches as negatives.
R@K checked only diagonal — missed valid matches, artificially low recall.
FIXES:
1. WeightedInfoNCE loss (src/losses/weighted_infonce.py):
- Per-sample adaptive label smoothing from positive_weights (IoU)
- Higher weight → sharper target, lower → softer (semi-positive tolerance)
- Based on Game4Loc reference implementation
2. Multi-match R@K evaluation:
- Uses dataset.get_all_valid_sat_names() to get ALL valid matches per query
- R@K counts hit if ANY valid satellite is in top-K (not just diagonal)
- AP computed as MRR over first valid match
3. Dataset returns positive_weight per sample:
- Sampled satellite weight passed to loss for adaptive smoothing
- All valid satellite candidates exposed for evaluation
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Asymmetric DINOv3: WEB (drone) + SAT (satellite) with separate MONA
- MONA on all 24 blocks per encoder (was last 12)
- Remove projection, native 1024-dim retrieval space (was 512)
- Total: 748M params, 17.6M trainable (2.35%)
Hard negative memory bank:
- MoCo-style FIFO queue of 4096 detached gallery embeddings
- Each batch: B in-batch + Q queue negatives in InfoNCE
- Queue updated after each forward pass
Training config:
- batch_size=8, grad_accum=8, effective_batch=64
- eval_every=1 (eval + train recall every epoch)
- Max bs=24 with grad checkpointing on RTX 4090
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Document AP (MRR) metric computed on both train and val
- Add output CSVs table (train.csv, val.csv, train_recall.csv, train_batches.csv)
- Add plots table (train_metrics, val_metrics with AP panel, overview)
- Update diagnostics table with recall CSV and plots
- Note overfitting detection via train vs val comparison
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Compute AP (Mean Reciprocal Rank) in _evaluate() for both q→g and g→q
- AP saved in val.csv and train_recall.csv alongside R@K
- New AP plot panel in val_metrics.png (train vs val, both directions)
- Log AP in console output for train-recall and val epochs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- _evaluate() now computes per-batch loss when loss_fn is provided
- Val loss and train recall loss saved in val.csv and train_recall.csv
- Overview plot shows train vs val loss curves side by side
- Helps detect overfitting: val loss diverging from train loss
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Evaluate R@K on train set (subset matching test size) alongside val
- New train_recall.csv with per-epoch train R@1/R@5/R@10
- Plot train vs val recall on same chart (solid=val, dashed=train)
- Helps detect overfitting: train R@1 up + val R@1 flat = overfit
- Train eval uses clean transforms (no augmentation)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
epoch_000_train.csv and epoch_000_val.csv removed — all data is in
train.csv and val.csv which persist across resume.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On resume, the same epoch may be re-run. Now log_train/log_val remove
existing entries for the same epoch before appending, preventing
duplicate rows in train.csv/val.csv.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On init, load existing train.csv/val.csv so that epoch-level metrics
and plots include the full training history after checkpoint resume.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Access model.image_encoder instead of model.drone_encoder/sat_encoder
when shared_encoder=True. Caused AttributeError at eval epoch 4.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switch MONA from fp32 to bfloat16 (same exponent range as fp32, no underflow)
- fp16 causes NaN: gamma=1e-6 falls into subnormal range (min normal ~6.1e-5)
- bf16 min normal ~1.2e-38, so 1e-6 is safe
- RTX 4090 supports bf16 natively
- Document bf16 vs fp16 vs fp32 comparison in README
- Update model summary: 3.5M MONA (last 12 blocks), 5.6M total trainable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- MONA fp16 causes NaN (gamma=1e-6 underflows in fp16 min subnormal ~6e-8)
- Revert MONA forward to fp32 with autocast(enabled=False), cast output back
- Fix loss CSV: save raw_loss before backward() (tensor consumed after backward)
- Verified: loss=3.78, no NaN, bs=48 peak=21.4 GB
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DINOv3: checkpoint each of 24 transformer blocks (recompute on backward)
- DGTRS-CLIP: checkpoint each of 12 transformer blocks
- Enables batch_size=24 on RTX 4090 (was 8 without checkpointing)
- Peak VRAM: 20.3 GB at bs=24 (was OOM at bs=16 before)
- ~20-30% slower per step, but 3x more in-batch negatives (23 vs 7)
- Enabled by default (gradient_checkpointing=True in config)
- Update README with VRAM benchmarks and checkpointing docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Single DINOv3 WEB for both drone and satellite branches (shared_encoder=True default)
- One set of MONA adapters instead of two: 7M trainable vs 14M
- Total params: 438M (was 748M), trainable: 10.6M (was 17.6M)
- Asymmetric mode still available via shared_encoder=False
- Add gradient accumulation (grad_accum_steps, --grad-accum CLI flag)
- Update model summary in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New config field grad_accum_steps (default=1, no change in behavior)
- Loss scaled by 1/accum, optimizer step every N micro-batches
- Scheduler counts optimizer steps (not micro-batches)
- CLI flag --grad-accum for override
- Document gradient accumulation and in-batch negatives in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that dataset stores only positive drone-satellite pairs,
negatives are formed automatically via InfoNCE similarity matrix
within each batch (B-1 in-batch negatives per query).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Show MONA adapters (7M) and LoRA (147K) in branch diagrams
- Add retrieval/loss block with temperature and CE weights
- Add diagnostics pipeline block (per-batch CSV, Grad-CAM, profiler, grad norms)
- Add gtauav_image_heavy.gin to structure
- Split CSV row in diagnostics table into per-batch and per-epoch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_batches.csv and epoch_N_batches.csv now update after every batch
instead of flushing at epoch end. Uses file append mode for efficiency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use gin.configurable(module=...) to prevent __main__ vs module name clash
- Remove `import src.training.train_gtauav` from gin files (already loaded)
- Use short selector names (TrainConfigGTAUAV) in all gin configs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New: src/training/plot_metrics.py
- train_metrics.png: loss, temperature, gates, lr
- val_metrics.png: R@K q→g and g→q
- overview.png: combined loss + R@1 + gates/tau
Auto-generates plots in {output_dir}/logs/ after each validation epoch.
Also callable standalone: python -m src.training.plot_metrics --log-dir ...
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gitea 1.25 renders math blocks via ```math fence reliably.
Replaced $$...$$ with ```math blocks, inline math kept as backtick code.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through
Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3
Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values
11.1M trainable / 734M total (1.51%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.
CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: GradScaler scales gradients by ~65536 in fp16, causing
logit_scale.exp() gradient to overflow. The learnable temperature
and similarity logits must stay in fp32.
Fix: model forward runs inside autocast(fp16), but loss computation
(similarity @ temperature + cross_entropy) runs outside in fp32.
Also: clamp logit_scale in logit-space before exp() and force
similarity computation to fp32.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AsymmetricEncoder.save_checkpoint(): saves model_state + metadata
- AsymmetricEncoder.load_checkpoint(): rebuilds model with frozen backbones,
then loads trainable weights from checkpoint
- --resume flag restores optimizer, loss (learnable tau), and scheduler state
- Training continues from the saved epoch
Usage:
python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
