caption-test/README.md

Caption Quality Test on UAV-VisLoc

Validate generated text captions by measuring retrieval R@1 lift on UAV-VisLoc. Uses GeoRSCLIP ViT-B/32 dual encoder with multi-term InfoNCE.

Full analysis: 2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md

Structure

caption_test/
├── conf/
│   ├── balanced.gin          # Primary test: λ=(1.0, 0.3, 0.3, 0.1)
│   ├── baseline_no_text.gin  # Reference: λ=(1.0, 0, 0, 0)
│   └── text_heavy.gin        # Stress test: λ=(0.5, 0.5, 0.5, 0.2)
├── scripts/
│   ├── generate_captions.py  # Offline caption generation (template/VLM/hybrid)
│   └── compare_runs.py       # Δ R@1 comparison report builder
├── src/
│   ├── datasets/
│   │   └── visloc_with_captions.py
│   ├── models/
│   │   └── dual_encoder.py   # GeoRSCLIP wrapper, projection heads
│   ├── losses/
│   │   └── multi_infonce.py  # 4-term InfoNCE + curriculum + cosine τ
│   ├── training/
│   │   └── train.py          # Main loop
│   └── eval/
│       └── evaluate.py       # R@K metrics, Δ R@1 helper
└── data/                     # (user-provided) VisLoc pairs + captions

Prerequisites

torch>=2.0
open_clip_torch
gin-config
Pillow
numpy

GeoRSCLIP ViT-B/32 checkpoint: download RS5M_ViT-B-32.pt from github.com/om-ai-lab/RS5M and place under checkpoints/.

Workflow

1. Generate captions

python -m scripts.generate_captions \
    --image_root data/visloc/images \
    --pairs_csv data/visloc/pairs.csv \
    --output data/visloc_train.json \
    --strategy hybrid \
    --vlm_refine_ratio 0.1

Replace _placeholder_vlm_caption in scripts/generate_captions.py with real Qwen2.5-VL or InternVL2 inference before running on production data.

2. Train three variants (in parallel or sequentially)

# Baseline (no captions)
python -m src.training.train --config conf/baseline_no_text.gin

# Balanced (primary, with captions)
python -m src.training.train --config conf/balanced.gin

# Text-heavy (stress test)
python -m src.training.train --config conf/text_heavy.gin

3. Evaluate each on test split

from src.eval.evaluate import run_evaluation_from_checkpoint

run_evaluation_from_checkpoint(
    checkpoint_path="out/caption_test/balanced/ckpt_epoch029.pt",
    test_manifest="data/visloc_test.json",
    image_root="data/visloc/images",
    output_path="out/caption_test/balanced/eval_report.json",
)

4. Compare and get verdict

python -m scripts.compare_runs \
    --baseline_report out/caption_test/baseline_no_text/eval_report.json \
    --full_report    out/caption_test/balanced/eval_report.json \
    --output         out/caption_test/comparison.md

Decision rule (from compare_runs.py)

Δ R@1 (drone→sat)	Verdict
≥ +3%	✅ PASS — captions informative, proceed to World-UAV
+1% to +3%	⚠️ MARGINAL — add VLM refinement, re-run
0 to +1%	❌ WEAK — redesign caption pipeline
< 0	❌❌ HARMFUL — critical bug

Expected runtime (RTX 4090, 24 GB)

Phase	Time
Caption generation (6K pairs, hybrid)	~1 h
Single training run (30 epochs, batch 128)	~2–3 h
Full test (3 variants × 3 seeds = 9 runs)	~30 h
Evaluation + comparison	~30 min

Notes on code style

Follows NADEZHDA code style:

• from __future__ import annotations everywhere.
• Type hints on all signatures.
• Google-style docstrings.
• @gin.configurable on top-level classes.
• Atomic checkpoint saves (_atomic_save helper).
• No emojis in code, English-only code comments.

Relation to NADEZHDA main pipeline

This is an isolated experimental track: completely separate from the main Student/Teacher training under code_nadezhda/. Once captions pass the Δ R@1 ≥ +3% gate here, the hybrid caption generation strategy is applied to World-UAV 927K, and those captions feed into E1 Teacher training with Multi-FiLM conditioning (see АНАЛИЗ_fusion_для_NADEZHDA.md).

Files referenced

• 2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md — full experimental design
• 2_hypotesis/АНАЛИЗ_text_encoder_для_NADEZHDA.md — why GeoRSCLIP
• 2_hypotesis/АНАЛИЗ_fusion_для_NADEZHDA.md — where captions land in Teacher
• 2_hypotesis/ROADMAP_E0_E9_unified.md — phase E_caption (parallel to E0)