Fix GTA-UAV eval + training pipeline: full gallery, mutex sampler, per-sample mask

Six critical fixes to the caption-test training/eval stack:

1. **IndentationError blocker** (train_gtauav.py:765-766)
   Unparseable file — train-recall LOGGER.info block was orphaned outside
   its `if eval_every` guard. Wrapped in `if train_recall:` so val eval
   and Grad-CAM only run on eval epochs.

2. **Full satellite gallery in `_evaluate`**
   Old code assembled gallery from DataLoader batches (one random sat per
   drone), producing an incomplete gallery of size ≈ N_query instead of
   N_unique_sat. Metrics were inflated because retrieval was against a
   subset that always contained the target.
   New `GTAUAVSatGallery` / `GTAUAVDroneQuery` iterate all unique tiles
   and queries independently; full-gallery multi-match R@K + MRR.

3. **Per-sample caption mask** (`AsymmetricEncoder._fuse_with_mask`)
   Mixed batches (some samples have captions, some don't) previously
   encoded empty strings through DGTRS and mixed the noise output into
   every sample via scalar gate. New `encode_query`/`encode_gallery` use
   `torch.where` to fall back to pure image features for empty-caption
   samples. Training `forward()` routes through the same helper so
   training and eval share code.

4. **Symmetric InfoNCE as primary loss** (multi_infonce.InfoNCELoss)
   Switched gin default from `WeightedInfoNCELoss` (adaptive label
   smoothing — not the Game4Loc soft-IoU target it claimed) to the
   existing symmetric InfoNCE with q2g=0.6/g2q=0.4 weighting. Loss type
   now selectable via `cfg.loss_type ∈ {"symmetric", "weighted"}`.

5. **MutuallyExclusiveSampler** (new file)
   BatchSampler that greedily packs drones whose `sat_candidates` sets
   are pairwise disjoint within a batch. Eliminates false negatives from
   the semi-positive graph without needing soft-label losses.
   At bs=8 keeps 100% of 24,891 train entries; at bs=64 keeps 92.6%.
   `set_epoch()` for reproducibility + different batches per epoch.

6. **Temperature clamp [0.01, 0.1]** (both loss modules)
   Old tau_max=0.5 allowed the logit distribution to collapse into a
   near-uniform softmax. Tightened to the CLIP-standard range.

Also:
- Added `scripts/smoke_eval.py` / `scripts/smoke_train.py` for fast
  regression checks (eval in ~2 min, 2 train steps in ~1 min on RTX 4090).
- CLAUDE.md updated to reflect the new pipeline.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-24 15:58:27 +03:00
parent ce7892926f
commit a499fcfd65
10 changed files with 640 additions and 141 deletions

View File

@@ -30,9 +30,13 @@ TextFusionMLP shared между query и gallery (одинаковый форм
Для sat images без captions: s_txt=None → g = s_img (gate passthrough)
LOSS: L = 0.6·CE(q̂·ĝᵀ/τ, targets) + 0.4·CE(ĝ·q̂ᵀ/τ, targets)
τ = 1/exp(logit_scale), learnable, clamped [0.01, 0.5], init=0.07
τ = 1/exp(logit_scale), learnable, clamped [0.01, 0.1], init=0.07
label_smoothing=0.1
BATCH SAMPLING: MutuallyExclusiveSampler — в одном батче нет двух drone'ов
с пересекающимися sat_candidates (исключает false negatives, которые
иначе появляются из-за multi-positive структуры GTA-UAV).
BASELINE: σ(α_q)=σ(α_g)=1.0, text disabled, DGTRS not loaded
```
@@ -42,7 +46,7 @@ BASELINE: σ(α_q)=σ(α_g)=1.0, text disabled, DGTRS not loaded
- **L3 fingerprint:** P3 — уникальные landmarks для matching (20-50 tok)
- **Fusion:** z_text = MLP([z₁; z₂; z₃]) — concat 3×768 → Linear(2304,1024) → GELU → Linear(1024,1024)
- **Shared MLP** между query и gallery ветками (одинаковый формат captions)
- **Satellite captions:** 6,546 из 14,640 sat images имеют captions. Для остальных gate passthrough (g = s_img)
- **Satellite captions:** 6,546 из 14,640 sat images имеют captions. Для остальных gate passthrough (g = s_img)**per-sample mask** в `_fuse_with_mask` возвращает чистые image features для samples без caption (без шума от пустых строк)
### Text encoder: DGTRS-CLIP (official architecture)
- Код: `src/models/dgtrs/` — из github.com/MitsuiChen14/DGTRS (Apache-2.0)
@@ -94,10 +98,14 @@ Eval: Resize(256) + CenterCrop(256) + ImageNet normalization.
|------|-----------|
| `src/models/dgtrs/model.py` | Официальная архитектура DGTRS-CLIP text encoder (Apache-2.0) |
| `src/models/dgtrs/simple_tokenizer.py` | BPE tokenizer (248 tokens, vocab 49408) |
| `src/models/asymmetric_encoder.py` | DINOv3ViT + TextFusionMLP + AsymmetricEncoder + GatedFusion |
| `src/datasets/gtauav_dataset.py` | GTA-UAV-LR loader + L1/L2/L3 caption parsing из VLM JSON |
| `src/losses/multi_infonce.py` | InfoNCE с learnable temperature (fp32), clamp [0.01, 0.5] |
| `src/training/train_gtauav.py` | Training loop с gin, W&B/TB, AMP, per-group LR, warmup, --resume |
| `src/models/asymmetric_encoder.py` | DINOv3ViT + TextFusionMLP + AsymmetricEncoder + GatedFusion + encode_query/encode_gallery (per-sample caption mask) |
| `src/datasets/gtauav_dataset.py` | GTA-UAV-LR loader + L1/L2/L3 captions + GTAUAVSatGallery/GTAUAVDroneQuery (full retrieval eval) |
| `src/datasets/mutually_exclusive_sampler.py` | BatchSampler: drone'ы в батче не делят sat_candidates (no false negatives) |
| `src/losses/multi_infonce.py` | **Primary:** SymmetricInfoNCE + MoCo queue, learnable τ clamp [0.01, 0.1], weights q2g=0.6 g2q=0.4 |
| `src/losses/weighted_infonce.py` | Alternative: per-sample adaptive label smoothing (активируется `loss_type="weighted"`) |
| `src/losses/hard_negatives.py` | NegativeMemoryBank (MoCo-style FIFO queue 4096 × 1024) |
| `src/training/train_gtauav.py` | Training loop: full-gallery `_evaluate`, mutex sampler wiring, loss_type switch |
| `scripts/smoke_eval.py` / `scripts/smoke_train.py` | Регрессионные smoke-тесты для eval и train pipeline |
| `src/training/trackers.py` | Unified experiment tracker: W&B + TensorBoard + CSV |
| `src/training/grad_monitor.py` | Gradient norm monitoring per param group |
| `src/training/gradcam.py` | Grad-CAM visualization для DINOv3 encoders |
@@ -203,7 +211,9 @@ Meta-файл `meta/seg_filter.json`: исключение изображени
- 10 epochs, batch 64, AMP, image 256x256
- **Optimizer:** AdamW, per-group LR: proj=1e-4, text=1e-5 (10x lower)
- **Scheduler:** linear warmup (2 epochs) + cosine annealing (per-step)
- **Loss:** InfoNCE с learnable temperature (CLIP logit_scale), init=0.07, clamp [0.01, 0.5]
- **Loss:** SymmetricInfoNCE (q2g=0.6, g2q=0.4) с learnable τ (init=0.07, clamp [0.01, 0.1])
- **Batch sampler:** MutuallyExclusiveSampler — batches disjoint по sat_candidates (на bs=8 сохраняет 100% entries)
- **Eval:** full satellite gallery (~2684 unique tiles для test_20) с multi-match R@K (учитывает все positive/semi-positive)
- **Augmentations:**
- Drone: RandomResizedCrop(0.7-1.0), HFlip, Rotation(15°), ColorJitter, Grayscale(5%), GaussianBlur
- Satellite: RandomResizedCrop(0.7-1.0), HFlip, ColorJitter, Grayscale(5%)