The mutex-only run still collapsed at epoch 1 — same pattern as the DSS
run. Val loss locks to log(8) ≈ 2.08 (uniform over the in-batch sat),
train loss grows monotonically (4.80 → 5.56), train R@1 drops 7.79% →
0.34%. Mode collapse, not sampler-induced.
The smoking gun is the queue: WeightedInfoNCELoss (the OLD-run loss)
silently ignored `queue_negatives`, so the OLD-run effective task was
in-batch-only contrast against 8 negatives. Switching to InfoNCELoss
made the queue active — 4096 stale embeddings without a momentum encoder
to keep them consistent with the live model. With the trimmed adapter
surface (MONA in last 12/24 blocks → 3.5M trainable), the model can't
reconcile fresh representations against stale negatives and collapses.
Disable the queue entirely (`neg_bank_size = 0`). Matches OLD's effective
setup — same 8 in-batch negatives, but with the new SymmetricInfoNCE +
mutex sampler + tau clamp 0.1 + per-sample mask + full-gallery eval.
Output → `out/gtauav/baseline_inbatch` (separate from the failed mutex run).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previous baseline run collapsed to ~random retrieval (R@1 0.6% at epoch 8,
train loss trending up 4.2 → 4.5). Hypothesis: at bs=8, DSS packs
visually-identical drones into a batch where InfoNCE asks the model to
discriminate them, and the hard-mining queue amplifies that hardness —
together they prevent convergence from a near-random start.
Override the new baseline config to run with the simpler regime:
sampler_type = "mutex" (disable DSS, keep only the no-false-negative guarantee)
hard_mining_k = 0 (use full queue as uniform negatives, no per-query top-K)
Fresh `out/gtauav/baseline_mutex` output dir so results stay separate from
the failed run's mixed logs.
Other architecture changes (shared DINOv3 WEB, MONA in last 12 blocks,
grad_accum=8) kept — verify they work with simple sampling before
layering DSS/mining back on.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
