Three related architecture changes, driven by a cost/simplicity trade-off:
1. **Shared encoder**: one DINOv3 LVD-1689M (WEB) processes both drone
and satellite images. Previously asymmetric — separate WEB (drone) and
SAT-493M (satellite) encoders. Saves ~303M frozen params and halves
VRAM for the image tower. Expected to lose some satellite-domain
inductive bias; MONA adapters pick up the slack.
2. **MONA in last 12/24 blocks**: adapters injected only in the top half
of the ViT. The lowest 12 blocks keep their pretrained features
untouched. Trainable MONA count drops from 14.0M (48 adapters × 2
encoders) to 3.5M (24 adapters × 1 encoder).
3. **No DINO_SAT**: `nn_models/DINO_SAT` is no longer loaded by the
default config. It stays on disk and the path param is kept for
backward compat with asymmetric checkpoints.
Parameter counts (with text fusion + LoRA + gates):
Before: 17.6M trainable / 733M total (2.35%)
After: 7.06M trainable / 434M total (1.63%)
Also fixes a pre-existing resume bug: checkpoints now record
`shared_encoder`, `baseline_mode`, `mona_bottleneck`, `mona_last_n_blocks`
so `AsymmetricEncoder.load_checkpoint` can rebuild the right architecture.
Old checkpoints still load (missing keys fall back to asymmetric defaults
via `ckpt.get(..., <default>)`).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Three upgrades to the DynamicSimilaritySampler infrastructure:
1. **GPU kNN** (`dss_knn_device="cuda"`, default):
Moves the per-seed similarity matmul to the GPU. At 25K train items
this cuts per-epoch sampler generation from 17s to 1.6s — a 10.8x
speedup. Negligible VRAM (100MB for the [N, 1024] embedding tensor).
2. **LSH index** (`src/datasets/lsh_index.py`, opt-in via `dss_use_lsh=True`):
Random-projection cosine-LSH with H tables of B bits each. When enabled,
the sampler narrows the candidate pool per seed via hash-bucket lookup
before exact refinement. At 25K it's a wash (pool already fits in VRAM)
but provides a scaling path for 100K+ where the N² similarity matrix
would stop fitting. Default off.
3. **Embedding cache** (`src/datasets/embedding_cache.py`, `dss_cache_dir` config):
Disk-backed cache for drone query embeddings, keyed by epoch. Skips
re-embedding on --resume and lets ablations replay from a snapshot.
Atomic writes via `.tmp` → `.replace`.
Measured on 25K train entries, 1024-dim random embeddings:
CPU kNN: 17.44s
GPU kNN: 1.62s (10.8x)
GPU + LSH: 1.42s (LSH candidate pool 0.05% of N)
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Batches assembled from visually-similar drone queries pressure the model to
learn finer discriminative features. Random mutex batches average ~0.26
pairwise cosine similarity in query embedding space; DSS batches average
~0.71 — confirming the lookalikes grouping works as intended.
Algorithm per batch:
1. Pick a random seed drone from the remaining pool.
2. Rank the entire remaining pool by cosine similarity to the seed.
3. Walk the ranking in descending order; add items whose sat_candidates
don't collide with the batch's already-claimed set.
4. Drop the seed if no valid batch can be assembled (rare mutex deadlock).
Inherits MutuallyExclusiveSampler semantics — no false negatives. Degrades
gracefully to mutex-only when no embeddings are set (warmup epochs, or if
`sampler_type="mutex"` is chosen).
Integration in `train_gtauav.py`:
- New `_embed_drone_queries` helper: model.encode_query forwarded over
GTAUAVDroneQuery, returns [N, D] CPU tensor. ~13s per 1024 queries on
a 4090 → ~5 min for the full 25K train set.
- Epoch loop re-embeds every `dss_reembed_every` epochs after a `dss_warmup_epochs`
warmup (first epochs use mutex-only since untrained embeddings aren't
informative for kNN).
- Config: `sampler_type` ∈ {"mutex", "dss"}. Default flipped to "dss".
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
New `hard_mining_k` parameter on InfoNCELoss. When >0 and queue is non-empty,
each query row keeps only its K highest-similarity queue entries (via
`torch.topk`) as negatives, instead of the full queue. Fully vectorized —
no Python loop, no extra forward pass.
Rationale: the memory bank grows to 4096 detached gallery embeddings, but
most are easy negatives that contribute almost nothing to the gradient.
Hard mining focuses compute on the small subset that actually shapes the
decision boundary. +2-3% R@1 in similar contrastive setups.
Edge cases:
- K=0: mining disabled, full queue used (original behavior).
- K >= queue size: falls back to full queue (e.g. warmup when queue is small).
- Queue empty: in-batch only, no changes.
Default in `gtauav_balanced.gin`: K=512 (1/8 of queue). Smoke-train updated
to exercise the full memory-bank + mining path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Six critical fixes to the caption-test training/eval stack:
1. **IndentationError blocker** (train_gtauav.py:765-766)
Unparseable file — train-recall LOGGER.info block was orphaned outside
its `if eval_every` guard. Wrapped in `if train_recall:` so val eval
and Grad-CAM only run on eval epochs.
2. **Full satellite gallery in `_evaluate`**
Old code assembled gallery from DataLoader batches (one random sat per
drone), producing an incomplete gallery of size ≈ N_query instead of
N_unique_sat. Metrics were inflated because retrieval was against a
subset that always contained the target.
New `GTAUAVSatGallery` / `GTAUAVDroneQuery` iterate all unique tiles
and queries independently; full-gallery multi-match R@K + MRR.
3. **Per-sample caption mask** (`AsymmetricEncoder._fuse_with_mask`)
Mixed batches (some samples have captions, some don't) previously
encoded empty strings through DGTRS and mixed the noise output into
every sample via scalar gate. New `encode_query`/`encode_gallery` use
`torch.where` to fall back to pure image features for empty-caption
samples. Training `forward()` routes through the same helper so
training and eval share code.
4. **Symmetric InfoNCE as primary loss** (multi_infonce.InfoNCELoss)
Switched gin default from `WeightedInfoNCELoss` (adaptive label
smoothing — not the Game4Loc soft-IoU target it claimed) to the
existing symmetric InfoNCE with q2g=0.6/g2q=0.4 weighting. Loss type
now selectable via `cfg.loss_type ∈ {"symmetric", "weighted"}`.
5. **MutuallyExclusiveSampler** (new file)
BatchSampler that greedily packs drones whose `sat_candidates` sets
are pairwise disjoint within a batch. Eliminates false negatives from
the semi-positive graph without needing soft-label losses.
At bs=8 keeps 100% of 24,891 train entries; at bs=64 keeps 92.6%.
`set_epoch()` for reproducibility + different batches per epoch.
6. **Temperature clamp [0.01, 0.1]** (both loss modules)
Old tau_max=0.5 allowed the logit distribution to collapse into a
near-uniform softmax. Tightened to the CLIP-standard range.
Also:
- Added `scripts/smoke_eval.py` / `scripts/smoke_train.py` for fast
regression checks (eval in ~2 min, 2 train steps in ~1 min on RTX 4090).
- CLAUDE.md updated to reflect the new pipeline.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through
Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3
Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values
11.1M trainable / 734M total (1.51%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.
CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- UAV-VisLoc processed at 512x512 (for segmentation/depth/normals)
- Dataset verified: 6744 drone, 74807 crops, median match 25.9m
- Known issue: 6 drones in route 06 outside satellite coverage
- Resize to model input size (224/256) in dataloader
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
