Rewrite: GatedFusion architecture + UAV-GeoLoc dataset

Architecture v2: - Query branch: drone + text -> GatedFusion -> proj -> query_emb - Gallery branch: satellite -> proj -> gallery_emb - Single InfoNCE loss (asymmetric 0.6/0.4) - GatedFusion: learnable gated addition (sigma(alpha)*img + (1-sigma(alpha))*text) - Baseline mode: gate=1.0 (text ignored) Dataset: - UAV-GeoLoc loader with template captions from path metadata - 27 terrain types with predefined features - Random positive crop sampling per epoch Configs: balanced (gate=0.7), baseline (no text), text_heavy (gate=0.3) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-04-17 17:13:00 +03:00
parent 2ce4017ebd
commit abb3337f8d
12 changed files with 1077 additions and 781 deletions
--- a/README.md
+++ b/README.md
@@ -1,33 +1,42 @@
-# Caption Quality Test on UAV-VisLoc
+# Caption Quality Test for Cross-View Geo-Localization

-Validate generated text captions by measuring retrieval R@1 lift on UAV-VisLoc.
-Uses GeoRSCLIP ViT-B/32 dual encoder with multi-term InfoNCE.
+Validate whether generated text captions improve retrieval R@1 in cross-view
+geo-localization (drone-to-satellite). Uses GeoRSCLIP ViT-B/32 dual encoder
+with GatedFusion on the query branch.

-Full analysis: `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md`
+## Architecture
+
+```
+Query:   drone_img + caption -> GatedFusion -> proj -> query_emb
+Gallery: sat_img -> proj -> gallery_emb
+Loss:    InfoNCE(query, gallery)
+```
+
+Baseline: fusion gate = 1.0 (text ignored).

 ## Structure

 ```
 caption_test/
 ├── conf/
-│   ├── balanced.gin          # Primary test: λ=(1.0, 0.3, 0.3, 0.1)
-│   ├── baseline_no_text.gin  # Reference: λ=(1.0, 0, 0, 0)
-│   └── text_heavy.gin        # Stress test: λ=(0.5, 0.5, 0.5, 0.2)
+│   ├── balanced.gin          # Primary: gate init 0.7 (30% text)
+│   ├── baseline_no_text.gin  # Reference: gate = 1.0 (no text)
+│   └── text_heavy.gin        # Stress: gate init 0.3 (70% text)
 ├── scripts/
-│   ├── generate_captions.py  # Offline caption generation (template/VLM/hybrid)
-│   └── compare_runs.py       # Δ R@1 comparison report builder
+│   ├── generate_captions.py  # Offline caption generation
+│   └── compare_runs.py       # Delta R@1 comparison report
 ├── src/
 │   ├── datasets/
-│   │   └── visloc_with_captions.py
+│   │   └── visloc_with_captions.py  # UAV-GeoLoc loader + template captions
 │   ├── models/
-│   │   └── dual_encoder.py   # GeoRSCLIP wrapper, projection heads
+│   │   └── dual_encoder.py   # GeoRSCLIP + GatedFusion + projection heads
 │   ├── losses/
-│   │   └── multi_infonce.py  # 4-term InfoNCE + curriculum + cosine τ
+│   │   └── multi_infonce.py  # InfoNCE with cosine temperature
 │   ├── training/
-│   │   └── train.py          # Main loop
+│   │   └── train.py          # Main training loop
 │   └── eval/
-│       └── evaluate.py       # R@K metrics, Δ R@1 helper
-└── data/                     # (user-provided) VisLoc pairs + captions
+│       └── evaluate.py       # R@K metrics, Delta R@1
+└── checkpoints/              # RS5M_ViT-B-32.pt (user-provided)
 ```

 ## Prerequisites
@@ -40,99 +49,66 @@ Pillow
 numpy
 ```

-GeoRSCLIP ViT-B/32 checkpoint: download `RS5M_ViT-B-32.pt` from
+GeoRSCLIP checkpoint: download `RS5M_ViT-B-32.pt` from
 `github.com/om-ai-lab/RS5M` and place under `checkpoints/`.

 ## Workflow

-### 1. Generate captions
+### 1. Train baseline (no text)

 ```bash
-python -m scripts.generate_captions \
-    --image_root data/visloc/images \
-    --pairs_csv data/visloc/pairs.csv \
-    --output data/visloc_train.json \
-    --strategy hybrid \
-    --vlm_refine_ratio 0.1
-```
-
-Replace `_placeholder_vlm_caption` in `scripts/generate_captions.py` with real
-Qwen2.5-VL or InternVL2 inference before running on production data.
-
-### 2. Train three variants (in parallel or sequentially)
-
-```bash
-# Baseline (no captions)
 python -m src.training.train --config conf/baseline_no_text.gin
+```

-# Balanced (primary, with captions)
+### 2. Train with captions
+
+```bash
 python -m src.training.train --config conf/balanced.gin
-
-# Text-heavy (stress test)
-python -m src.training.train --config conf/text_heavy.gin
 ```

-### 3. Evaluate each on test split
-
-```python
-from src.eval.evaluate import run_evaluation_from_checkpoint
-
-run_evaluation_from_checkpoint(
-    checkpoint_path="out/caption_test/balanced/ckpt_epoch029.pt",
-    test_manifest="data/visloc_test.json",
-    image_root="data/visloc/images",
-    output_path="out/caption_test/balanced/eval_report.json",
-)
-```
-
-### 4. Compare and get verdict
+### 3. Compare and get verdict

 ```bash
 python -m scripts.compare_runs \
    --baseline_report out/caption_test/baseline_no_text/eval_report.json \
-    --full_report    out/caption_test/balanced/eval_report.json \
-    --output         out/caption_test/comparison.md
+    --full_report out/caption_test/balanced/eval_report.json \
+    --output out/caption_test/comparison.md
 ```

-## Decision rule (from `compare_runs.py`)
+## Decision rule

-| Δ R@1 (drone→sat) | Verdict |
+| Delta R@1 (query->gallery) | Verdict |
 |---|---|
-| ≥ +3% | ✅ PASS — captions informative, proceed to World-UAV |
-| +1% to +3% | ⚠️ MARGINAL — add VLM refinement, re-run |
-| 0 to +1% | ❌ WEAK — redesign caption pipeline |
-| < 0 | ❌❌ HARMFUL — critical bug |
+| >= +3% | PASS -- captions informative, proceed to production |
+| +1% to +3% | MARGINAL -- add VLM refinement, re-run |
+| 0 to +1% | WEAK -- redesign caption pipeline |
+| < 0 | HARMFUL -- critical bug |

 ## Expected runtime (RTX 4090, 24 GB)

 | Phase | Time |
 |---|---|
-| Caption generation (6K pairs, hybrid) | ~1 h |
-| Single training run (30 epochs, batch 128) | ~2–3 h |
-| Full test (3 variants × 3 seeds = 9 runs) | ~30 h |
-| Evaluation + comparison | ~30 min |
+| Single training run (10 epochs, batch 128, 206K queries) | ~15-30 min |
+| Full test (baseline + balanced + text_heavy) | ~1-1.5 h |
+| Evaluation | ~2-5 min per run |

-## Notes on code style
+## Dataset

-Follows NADEZHDA code style:
- `from __future__ import annotations` everywhere.
- Type hints on all signatures.
- Google-style docstrings.
- `@gin.configurable` on top-level classes.
- Atomic checkpoint saves (`_atomic_save` helper).
- No emojis in code, English-only code comments.
+UAV-GeoLoc Terrain split (from `/mnt/data1tb/cvgl_datasets/UAV-GeoLoc/`):
+- Train: 206,108 queries, 94,709 DB crops (140 scenes)
+- Val: 62,368 queries, 26,597 DB crops (40 scenes)
+- Test: 33,472 queries, 11,684 DB crops (20 scenes)

-## Relation to NADEZHDA main pipeline
+Template captions generated automatically from path metadata:
+```
+"Aerial view at 100m facing northwest over volcanic terrain near KilaueaVolcano.
+ Plan-view features: lava flows, crater edges, volcanic rock."
+```

-This is an **isolated experimental track**: completely separate from the main
-Student/Teacher training under `code_nadezhda/`. Once captions pass the
-Δ R@1 ≥ +3% gate here, the hybrid caption generation strategy is applied to
-World-UAV 927K, and those captions feed into E1 Teacher training with
-Multi-FiLM conditioning (see `АНАЛИЗ_fusion_для_NADEZHDA.md`).
+## Code style

-## Files referenced
-
- `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md` — full experimental design
- `2_hypotesis/АНАЛИЗ_text_encoder_для_NADEZHDA.md` — why GeoRSCLIP
- `2_hypotesis/АНАЛИЗ_fusion_для_NADEZHDA.md` — where captions land in Teacher
- `2_hypotesis/ROADMAP_E0_E9_unified.md` — phase E_caption (parallel to E0)
+- `from __future__ import annotations` everywhere
+- Type hints on all signatures
+- Google-style docstrings
+- `@gin.configurable` on top-level classes
+- No emojis in code, English-only comments