Pikaliov
2ce4017ebd
Self-contained experimental track validating generated text captions
via retrieval R@1 lift on UAV-VisLoc.
Architecture: GeoRSCLIP ViT-B/32 dual encoder, 512-dim shared space.
Loss: 4-term InfoNCE (img-img + sat-cap + drone-cap + cap-cap)
with cosine temperature decay, PALW-like curriculum.
Metric: delta R@1 (with text - without text) >= +3% => PASS.
Gin-configured (balanced / baseline_no_text / text_heavy variants).
Follows NADEZHDA code style.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
139 lines
4.4 KiB
Markdown
139 lines
4.4 KiB
Markdown
# Caption Quality Test on UAV-VisLoc
|
||
|
||
Validate generated text captions by measuring retrieval R@1 lift on UAV-VisLoc.
|
||
Uses GeoRSCLIP ViT-B/32 dual encoder with multi-term InfoNCE.
|
||
|
||
Full analysis: `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md`
|
||
|
||
## Structure
|
||
|
||
```
|
||
caption_test/
|
||
├── conf/
|
||
│ ├── balanced.gin # Primary test: λ=(1.0, 0.3, 0.3, 0.1)
|
||
│ ├── baseline_no_text.gin # Reference: λ=(1.0, 0, 0, 0)
|
||
│ └── text_heavy.gin # Stress test: λ=(0.5, 0.5, 0.5, 0.2)
|
||
├── scripts/
|
||
│ ├── generate_captions.py # Offline caption generation (template/VLM/hybrid)
|
||
│ └── compare_runs.py # Δ R@1 comparison report builder
|
||
├── src/
|
||
│ ├── datasets/
|
||
│ │ └── visloc_with_captions.py
|
||
│ ├── models/
|
||
│ │ └── dual_encoder.py # GeoRSCLIP wrapper, projection heads
|
||
│ ├── losses/
|
||
│ │ └── multi_infonce.py # 4-term InfoNCE + curriculum + cosine τ
|
||
│ ├── training/
|
||
│ │ └── train.py # Main loop
|
||
│ └── eval/
|
||
│ └── evaluate.py # R@K metrics, Δ R@1 helper
|
||
└── data/ # (user-provided) VisLoc pairs + captions
|
||
```
|
||
|
||
## Prerequisites
|
||
|
||
```
|
||
torch>=2.0
|
||
open_clip_torch
|
||
gin-config
|
||
Pillow
|
||
numpy
|
||
```
|
||
|
||
GeoRSCLIP ViT-B/32 checkpoint: download `RS5M_ViT-B-32.pt` from
|
||
`github.com/om-ai-lab/RS5M` and place under `checkpoints/`.
|
||
|
||
## Workflow
|
||
|
||
### 1. Generate captions
|
||
|
||
```bash
|
||
python -m scripts.generate_captions \
|
||
--image_root data/visloc/images \
|
||
--pairs_csv data/visloc/pairs.csv \
|
||
--output data/visloc_train.json \
|
||
--strategy hybrid \
|
||
--vlm_refine_ratio 0.1
|
||
```
|
||
|
||
Replace `_placeholder_vlm_caption` in `scripts/generate_captions.py` with real
|
||
Qwen2.5-VL or InternVL2 inference before running on production data.
|
||
|
||
### 2. Train three variants (in parallel or sequentially)
|
||
|
||
```bash
|
||
# Baseline (no captions)
|
||
python -m src.training.train --config conf/baseline_no_text.gin
|
||
|
||
# Balanced (primary, with captions)
|
||
python -m src.training.train --config conf/balanced.gin
|
||
|
||
# Text-heavy (stress test)
|
||
python -m src.training.train --config conf/text_heavy.gin
|
||
```
|
||
|
||
### 3. Evaluate each on test split
|
||
|
||
```python
|
||
from src.eval.evaluate import run_evaluation_from_checkpoint
|
||
|
||
run_evaluation_from_checkpoint(
|
||
checkpoint_path="out/caption_test/balanced/ckpt_epoch029.pt",
|
||
test_manifest="data/visloc_test.json",
|
||
image_root="data/visloc/images",
|
||
output_path="out/caption_test/balanced/eval_report.json",
|
||
)
|
||
```
|
||
|
||
### 4. Compare and get verdict
|
||
|
||
```bash
|
||
python -m scripts.compare_runs \
|
||
--baseline_report out/caption_test/baseline_no_text/eval_report.json \
|
||
--full_report out/caption_test/balanced/eval_report.json \
|
||
--output out/caption_test/comparison.md
|
||
```
|
||
|
||
## Decision rule (from `compare_runs.py`)
|
||
|
||
| Δ R@1 (drone→sat) | Verdict |
|
||
|---|---|
|
||
| ≥ +3% | ✅ PASS — captions informative, proceed to World-UAV |
|
||
| +1% to +3% | ⚠️ MARGINAL — add VLM refinement, re-run |
|
||
| 0 to +1% | ❌ WEAK — redesign caption pipeline |
|
||
| < 0 | ❌❌ HARMFUL — critical bug |
|
||
|
||
## Expected runtime (RTX 4090, 24 GB)
|
||
|
||
| Phase | Time |
|
||
|---|---|
|
||
| Caption generation (6K pairs, hybrid) | ~1 h |
|
||
| Single training run (30 epochs, batch 128) | ~2–3 h |
|
||
| Full test (3 variants × 3 seeds = 9 runs) | ~30 h |
|
||
| Evaluation + comparison | ~30 min |
|
||
|
||
## Notes on code style
|
||
|
||
Follows NADEZHDA code style:
|
||
- `from __future__ import annotations` everywhere.
|
||
- Type hints on all signatures.
|
||
- Google-style docstrings.
|
||
- `@gin.configurable` on top-level classes.
|
||
- Atomic checkpoint saves (`_atomic_save` helper).
|
||
- No emojis in code, English-only code comments.
|
||
|
||
## Relation to NADEZHDA main pipeline
|
||
|
||
This is an **isolated experimental track**: completely separate from the main
|
||
Student/Teacher training under `code_nadezhda/`. Once captions pass the
|
||
Δ R@1 ≥ +3% gate here, the hybrid caption generation strategy is applied to
|
||
World-UAV 927K, and those captions feed into E1 Teacher training with
|
||
Multi-FiLM conditioning (see `АНАЛИЗ_fusion_для_NADEZHDA.md`).
|
||
|
||
## Files referenced
|
||
|
||
- `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md` — full experimental design
|
||
- `2_hypotesis/АНАЛИЗ_text_encoder_для_NADEZHDA.md` — why GeoRSCLIP
|
||
- `2_hypotesis/АНАЛИЗ_fusion_для_NADEZHDA.md` — where captions land in Teacher
|
||
- `2_hypotesis/ROADMAP_E0_E9_unified.md` — phase E_caption (parallel to E0)
|