Initial commit: caption quality test on UAV-VisLoc
Self-contained experimental track validating generated text captions
via retrieval R@1 lift on UAV-VisLoc.
Architecture: GeoRSCLIP ViT-B/32 dual encoder, 512-dim shared space.
Loss: 4-term InfoNCE (img-img + sat-cap + drone-cap + cap-cap)
with cosine temperature decay, PALW-like curriculum.
Metric: delta R@1 (with text - without text) >= +3% => PASS.
Gin-configured (balanced / baseline_no_text / text_heavy variants).
Follows NADEZHDA code style.
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
138
README.md
Normal file
138
README.md
Normal file
@@ -0,0 +1,138 @@
|
||||
# Caption Quality Test on UAV-VisLoc
|
||||
|
||||
Validate generated text captions by measuring retrieval R@1 lift on UAV-VisLoc.
|
||||
Uses GeoRSCLIP ViT-B/32 dual encoder with multi-term InfoNCE.
|
||||
|
||||
Full analysis: `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md`
|
||||
|
||||
## Structure
|
||||
|
||||
```
|
||||
caption_test/
|
||||
├── conf/
|
||||
│ ├── balanced.gin # Primary test: λ=(1.0, 0.3, 0.3, 0.1)
|
||||
│ ├── baseline_no_text.gin # Reference: λ=(1.0, 0, 0, 0)
|
||||
│ └── text_heavy.gin # Stress test: λ=(0.5, 0.5, 0.5, 0.2)
|
||||
├── scripts/
|
||||
│ ├── generate_captions.py # Offline caption generation (template/VLM/hybrid)
|
||||
│ └── compare_runs.py # Δ R@1 comparison report builder
|
||||
├── src/
|
||||
│ ├── datasets/
|
||||
│ │ └── visloc_with_captions.py
|
||||
│ ├── models/
|
||||
│ │ └── dual_encoder.py # GeoRSCLIP wrapper, projection heads
|
||||
│ ├── losses/
|
||||
│ │ └── multi_infonce.py # 4-term InfoNCE + curriculum + cosine τ
|
||||
│ ├── training/
|
||||
│ │ └── train.py # Main loop
|
||||
│ └── eval/
|
||||
│ └── evaluate.py # R@K metrics, Δ R@1 helper
|
||||
└── data/ # (user-provided) VisLoc pairs + captions
|
||||
```
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```
|
||||
torch>=2.0
|
||||
open_clip_torch
|
||||
gin-config
|
||||
Pillow
|
||||
numpy
|
||||
```
|
||||
|
||||
GeoRSCLIP ViT-B/32 checkpoint: download `RS5M_ViT-B-32.pt` from
|
||||
`github.com/om-ai-lab/RS5M` and place under `checkpoints/`.
|
||||
|
||||
## Workflow
|
||||
|
||||
### 1. Generate captions
|
||||
|
||||
```bash
|
||||
python -m scripts.generate_captions \
|
||||
--image_root data/visloc/images \
|
||||
--pairs_csv data/visloc/pairs.csv \
|
||||
--output data/visloc_train.json \
|
||||
--strategy hybrid \
|
||||
--vlm_refine_ratio 0.1
|
||||
```
|
||||
|
||||
Replace `_placeholder_vlm_caption` in `scripts/generate_captions.py` with real
|
||||
Qwen2.5-VL or InternVL2 inference before running on production data.
|
||||
|
||||
### 2. Train three variants (in parallel or sequentially)
|
||||
|
||||
```bash
|
||||
# Baseline (no captions)
|
||||
python -m src.training.train --config conf/baseline_no_text.gin
|
||||
|
||||
# Balanced (primary, with captions)
|
||||
python -m src.training.train --config conf/balanced.gin
|
||||
|
||||
# Text-heavy (stress test)
|
||||
python -m src.training.train --config conf/text_heavy.gin
|
||||
```
|
||||
|
||||
### 3. Evaluate each on test split
|
||||
|
||||
```python
|
||||
from src.eval.evaluate import run_evaluation_from_checkpoint
|
||||
|
||||
run_evaluation_from_checkpoint(
|
||||
checkpoint_path="out/caption_test/balanced/ckpt_epoch029.pt",
|
||||
test_manifest="data/visloc_test.json",
|
||||
image_root="data/visloc/images",
|
||||
output_path="out/caption_test/balanced/eval_report.json",
|
||||
)
|
||||
```
|
||||
|
||||
### 4. Compare and get verdict
|
||||
|
||||
```bash
|
||||
python -m scripts.compare_runs \
|
||||
--baseline_report out/caption_test/baseline_no_text/eval_report.json \
|
||||
--full_report out/caption_test/balanced/eval_report.json \
|
||||
--output out/caption_test/comparison.md
|
||||
```
|
||||
|
||||
## Decision rule (from `compare_runs.py`)
|
||||
|
||||
| Δ R@1 (drone→sat) | Verdict |
|
||||
|---|---|
|
||||
| ≥ +3% | ✅ PASS — captions informative, proceed to World-UAV |
|
||||
| +1% to +3% | ⚠️ MARGINAL — add VLM refinement, re-run |
|
||||
| 0 to +1% | ❌ WEAK — redesign caption pipeline |
|
||||
| < 0 | ❌❌ HARMFUL — critical bug |
|
||||
|
||||
## Expected runtime (RTX 4090, 24 GB)
|
||||
|
||||
| Phase | Time |
|
||||
|---|---|
|
||||
| Caption generation (6K pairs, hybrid) | ~1 h |
|
||||
| Single training run (30 epochs, batch 128) | ~2–3 h |
|
||||
| Full test (3 variants × 3 seeds = 9 runs) | ~30 h |
|
||||
| Evaluation + comparison | ~30 min |
|
||||
|
||||
## Notes on code style
|
||||
|
||||
Follows NADEZHDA code style:
|
||||
- `from __future__ import annotations` everywhere.
|
||||
- Type hints on all signatures.
|
||||
- Google-style docstrings.
|
||||
- `@gin.configurable` on top-level classes.
|
||||
- Atomic checkpoint saves (`_atomic_save` helper).
|
||||
- No emojis in code, English-only code comments.
|
||||
|
||||
## Relation to NADEZHDA main pipeline
|
||||
|
||||
This is an **isolated experimental track**: completely separate from the main
|
||||
Student/Teacher training under `code_nadezhda/`. Once captions pass the
|
||||
Δ R@1 ≥ +3% gate here, the hybrid caption generation strategy is applied to
|
||||
World-UAV 927K, and those captions feed into E1 Teacher training with
|
||||
Multi-FiLM conditioning (see `АНАЛИЗ_fusion_для_NADEZHDA.md`).
|
||||
|
||||
## Files referenced
|
||||
|
||||
- `2_hypotesis/АНАЛИЗ_caption_quality_test_VisLoc.md` — full experimental design
|
||||
- `2_hypotesis/АНАЛИЗ_text_encoder_для_NADEZHDA.md` — why GeoRSCLIP
|
||||
- `2_hypotesis/АНАЛИЗ_fusion_для_NADEZHDA.md` — where captions land in Teacher
|
||||
- `2_hypotesis/ROADMAP_E0_E9_unified.md` — phase E_caption (parallel to E0)
|
||||
Reference in New Issue
Block a user