Add ML diagnostics tooling (W&B, TensorBoard, Grad-CAM, profiler) and gin configs

- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 20:30:50 +03:00
parent 83ce04150d
commit 29a09349e7
11 changed files with 1098 additions and 57 deletions

View File

@@ -218,10 +218,13 @@ Gallery: sat_img -> GeoRSCLIP -> gallery
```
caption-test/
├── conf/ # Gin configs (v2)
│ ├── balanced.gin
│ ├── baseline_no_text.gin
── text_heavy.gin
├── conf/ # Gin configs
│ ├── gtauav_balanced.gin # GTA-UAV with text (10 epochs, v3)
│ ├── gtauav_baseline.gin # GTA-UAV baseline, no text (v3)
── gtauav_text_heavy.gin # GTA-UAV text-heavy gate=0.3 (v3)
│ ├── balanced.gin # UAV-GeoLoc with text (v2)
│ ├── baseline_no_text.gin # UAV-GeoLoc baseline (v2)
│ └── text_heavy.gin # UAV-GeoLoc text-heavy (v2)
├── nn_models/ # Pre-trained checkpoints (v3, gitignored)
│ ├── DINO_WEB/ # DINOv3 ViT-L/16 LVD-1689M (.pth)
│ ├── DINO_SAT/ # DINOv3 ViT-L/16 SAT-493M (.safetensors)
@@ -250,7 +253,12 @@ caption-test/
│ │ └── multi_infonce.py # InfoNCE with learnable temperature
│ ├── training/
│ │ ├── train_gtauav.py # Training loop GTA-UAV (v3)
│ │ ── train.py # Training loop UAV-GeoLoc (v2)
│ │ ── train.py # Training loop UAV-GeoLoc (v2)
│ │ ├── trackers.py # Unified tracker: W&B + TensorBoard
│ │ ├── grad_monitor.py # Gradient norm monitoring per group
│ │ ├── gradcam.py # Grad-CAM visualization for DINOv3
│ │ ├── profiling.py # PyTorch Profiler + torchinfo summary
│ │ └── plot_metrics.py # Seaborn/matplotlib metric plots
│ └── eval/
│ └── evaluate.py # R@K metrics, Delta R@1
└── checkpoints/ # GeoRSCLIP RS5M_ViT-B-32.pt (v2)
@@ -268,6 +276,17 @@ regex
gin-config
Pillow
numpy
pandas
matplotlib
seaborn
```
### Optional (for extended diagnostics)
```
wandb # Weights & Biases experiment tracking
torchinfo # Model summary tables
tensorboard # TensorBoard logging (included with torch)
```
## Workflow (V3 — GTA-UAV)
@@ -279,26 +298,52 @@ python -m scripts.make_split --output-dir meta
python -m scripts.filter_segmentation --output meta/seg_filter.json
```
### 2. Train baseline (no text)
### 2. Train with gin configs (recommended)
```bash
# Baseline (no text, 10 epochs)
python -m src.training.train_gtauav --config conf/gtauav_baseline.gin \
--filter-meta meta/seg_filter.json
# With captions (L1/L2/L3, 10 epochs)
python -m src.training.train_gtauav --config conf/gtauav_balanced.gin \
--filter-meta meta/seg_filter.json
# Text-heavy (gate=0.3, 70% text weight)
python -m src.training.train_gtauav --config conf/gtauav_text_heavy.gin \
--filter-meta meta/seg_filter.json
```
### 3. Train without gin (CLI-only)
```bash
python -m src.training.train_gtauav --baseline --filter-meta meta/seg_filter.json
```
### 3. Train with captions (L1/L2/L3)
```bash
python -m src.training.train_gtauav --filter-meta meta/seg_filter.json
```
### 4. Resume from checkpoint
### 4. Enable diagnostics
```bash
# W&B + Grad-CAM + PyTorch Profiler
python -m src.training.train_gtauav --config conf/gtauav_balanced.gin \
--filter-meta meta/seg_filter.json --wandb --gradcam --profile
# Gin parameter overrides from CLI
python -m src.training.train_gtauav --config conf/gtauav_balanced.gin \
--filter-meta meta/seg_filter.json \
--gin-param 'TrainConfigGTAUAV.batch_size=16' 'TrainConfigGTAUAV.epochs=20'
```
CLI flags (`--wandb`, `--gradcam`, `--profile`, `--epochs`, etc.) take priority over gin config.
### 5. Resume from checkpoint
```bash
python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt \
--filter-meta meta/seg_filter.json
```
### 5. Compare and get verdict
### 6. Compare and get verdict
```bash
python -m scripts.compare_runs \
@@ -307,6 +352,24 @@ python -m scripts.compare_runs \
--output out/gtauav/comparison.md
```
### 7. View TensorBoard
```bash
tensorboard --logdir out/gtauav/with_text/tb_logs
```
## Diagnostics & Visualization
| Tool | Flag | Output | Description |
|------|------|--------|-------------|
| **TensorBoard** | `--use-tb` (default on) | `{out}/tb_logs/` | Scalars, histograms, images |
| **W&B** | `--wandb` | cloud | Full experiment tracking, Grad-CAM images |
| **Grad-CAM** | `--gradcam` | `{out}/gradcam/` | DINOv3 attention heatmaps (drone + satellite) |
| **PyTorch Profiler** | `--profile` | `{out}/profiler/` | Chrome trace, CUDA timeline, memory |
| **torchinfo** | auto | `{out}/model_summary.txt` | Layer-by-layer parameter table |
| **Gradient norms** | `--log-grad-norms` (default on) | TB/W&B | Per-group: MONA, LoRA, MLP, gates, tau |
| **CSV + plots** | auto | `{out}/logs/` | train.csv, val.csv, PNG plots every epoch |
## Decision rule
| Delta R@1 (drone→satellite) | Verdict |