Six critical fixes to the caption-test training/eval stack:
1. **IndentationError blocker** (train_gtauav.py:765-766)
Unparseable file — train-recall LOGGER.info block was orphaned outside
its `if eval_every` guard. Wrapped in `if train_recall:` so val eval
and Grad-CAM only run on eval epochs.
2. **Full satellite gallery in `_evaluate`**
Old code assembled gallery from DataLoader batches (one random sat per
drone), producing an incomplete gallery of size ≈ N_query instead of
N_unique_sat. Metrics were inflated because retrieval was against a
subset that always contained the target.
New `GTAUAVSatGallery` / `GTAUAVDroneQuery` iterate all unique tiles
and queries independently; full-gallery multi-match R@K + MRR.
3. **Per-sample caption mask** (`AsymmetricEncoder._fuse_with_mask`)
Mixed batches (some samples have captions, some don't) previously
encoded empty strings through DGTRS and mixed the noise output into
every sample via scalar gate. New `encode_query`/`encode_gallery` use
`torch.where` to fall back to pure image features for empty-caption
samples. Training `forward()` routes through the same helper so
training and eval share code.
4. **Symmetric InfoNCE as primary loss** (multi_infonce.InfoNCELoss)
Switched gin default from `WeightedInfoNCELoss` (adaptive label
smoothing — not the Game4Loc soft-IoU target it claimed) to the
existing symmetric InfoNCE with q2g=0.6/g2q=0.4 weighting. Loss type
now selectable via `cfg.loss_type ∈ {"symmetric", "weighted"}`.
5. **MutuallyExclusiveSampler** (new file)
BatchSampler that greedily packs drones whose `sat_candidates` sets
are pairwise disjoint within a batch. Eliminates false negatives from
the semi-positive graph without needing soft-label losses.
At bs=8 keeps 100% of 24,891 train entries; at bs=64 keeps 92.6%.
`set_epoch()` for reproducibility + different batches per epoch.
6. **Temperature clamp [0.01, 0.1]** (both loss modules)
Old tau_max=0.5 allowed the logit distribution to collapse into a
near-uniform softmax. Tightened to the CLIP-standard range.
Also:
- Added `scripts/smoke_eval.py` / `scripts/smoke_train.py` for fast
regression checks (eval in ~2 min, 2 train steps in ~1 min on RTX 4090).
- CLAUDE.md updated to reflect the new pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through
Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3
Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values
11.1M trainable / 734M total (1.51%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.
CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- UAV-VisLoc processed at 512x512 (for segmentation/depth/normals)
- Dataset verified: 6744 drone, 74807 crops, median match 25.9m
- Known issue: 6 drones in route 06 outside satellite coverage
- Resize to model input size (224/256) in dataloader
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
