- Evaluate R@K on train set (subset matching test size) alongside val
- New train_recall.csv with per-epoch train R@1/R@5/R@10
- Plot train vs val recall on same chart (solid=val, dashed=train)
- Helps detect overfitting: train R@1 up + val R@1 flat = overfit
- Train eval uses clean transforms (no augmentation)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
epoch_000_train.csv and epoch_000_val.csv removed — all data is in
train.csv and val.csv which persist across resume.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On resume, the same epoch may be re-run. Now log_train/log_val remove
existing entries for the same epoch before appending, preventing
duplicate rows in train.csv/val.csv.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
On init, load existing train.csv/val.csv so that epoch-level metrics
and plots include the full training history after checkpoint resume.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Access model.image_encoder instead of model.drone_encoder/sat_encoder
when shared_encoder=True. Caused AttributeError at eval epoch 4.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Switch MONA from fp32 to bfloat16 (same exponent range as fp32, no underflow)
- fp16 causes NaN: gamma=1e-6 falls into subnormal range (min normal ~6.1e-5)
- bf16 min normal ~1.2e-38, so 1e-6 is safe
- RTX 4090 supports bf16 natively
- Document bf16 vs fp16 vs fp32 comparison in README
- Update model summary: 3.5M MONA (last 12 blocks), 5.6M total trainable
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- MONA fp16 causes NaN (gamma=1e-6 underflows in fp16 min subnormal ~6e-8)
- Revert MONA forward to fp32 with autocast(enabled=False), cast output back
- Fix loss CSV: save raw_loss before backward() (tensor consumed after backward)
- Verified: loss=3.78, no NaN, bs=48 peak=21.4 GB
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- DINOv3: checkpoint each of 24 transformer blocks (recompute on backward)
- DGTRS-CLIP: checkpoint each of 12 transformer blocks
- Enables batch_size=24 on RTX 4090 (was 8 without checkpointing)
- Peak VRAM: 20.3 GB at bs=24 (was OOM at bs=16 before)
- ~20-30% slower per step, but 3x more in-batch negatives (23 vs 7)
- Enabled by default (gradient_checkpointing=True in config)
- Update README with VRAM benchmarks and checkpointing docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Single DINOv3 WEB for both drone and satellite branches (shared_encoder=True default)
- One set of MONA adapters instead of two: 7M trainable vs 14M
- Total params: 438M (was 748M), trainable: 10.6M (was 17.6M)
- Asymmetric mode still available via shared_encoder=False
- Add gradient accumulation (grad_accum_steps, --grad-accum CLI flag)
- Update model summary in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- New config field grad_accum_steps (default=1, no change in behavior)
- Loss scaled by 1/accum, optimizer step every N micro-batches
- Scheduler counts optimizer steps (not micro-batches)
- CLI flag --grad-accum for override
- Document gradient accumulation and in-batch negatives in README
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Explain that dataset stores only positive drone-satellite pairs,
negatives are formed automatically via InfoNCE similarity matrix
within each batch (B-1 in-batch negatives per query).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Show MONA adapters (7M) and LoRA (147K) in branch diagrams
- Add retrieval/loss block with temperature and CE weights
- Add diagnostics pipeline block (per-batch CSV, Grad-CAM, profiler, grad norms)
- Add gtauav_image_heavy.gin to structure
- Split CSV row in diagnostics table into per-batch and per-epoch
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
train_batches.csv and epoch_N_batches.csv now update after every batch
instead of flushing at epoch end. Uses file append mode for efficiency.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Use gin.configurable(module=...) to prevent __main__ vs module name clash
- Remove `import src.training.train_gtauav` from gin files (already loaded)
- Use short selector names (TrainConfigGTAUAV) in all gin configs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- Add unified experiment tracker (W&B + TensorBoard) with graceful fallback
- Add gradient norm monitoring per param group (MONA, LoRA, MLP, gates, tau)
- Add Grad-CAM visualization for DINOv3 drone/satellite encoders
- Add PyTorch Profiler wrapper + torchinfo model summary
- Add gin-config support to train_gtauav.py with CLI overrides
- Add v3 gin configs: gtauav_balanced, gtauav_baseline, gtauav_text_heavy, gtauav_image_heavy
- Generate metric plots every epoch (not just on eval)
- Set default epochs to 10
- Update README and CLAUDE.md with new tooling and usage docs
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
New: src/training/plot_metrics.py
- train_metrics.png: loss, temperature, gates, lr
- val_metrics.png: R@K q→g and g→q
- overview.png: combined loss + R@1 + gates/tau
Auto-generates plots in {output_dir}/logs/ after each validation epoch.
Also callable standalone: python -m src.training.plot_metrics --log-dir ...
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Gitea 1.25 renders math blocks via ```math fence reliably.
Replaced $$...$$ with ```math blocks, inline math kept as backtick code.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through
Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3
Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values
11.1M trainable / 734M total (1.51%)
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.
CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Root cause: GradScaler scales gradients by ~65536 in fp16, causing
logit_scale.exp() gradient to overflow. The learnable temperature
and similarity logits must stay in fp32.
Fix: model forward runs inside autocast(fp16), but loss computation
(similarity @ temperature + cross_entropy) runs outside in fp32.
Also: clamp logit_scale in logit-space before exp() and force
similarity computation to fp32.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- AsymmetricEncoder.save_checkpoint(): saves model_state + metadata
- AsymmetricEncoder.load_checkpoint(): rebuilds model with frozen backbones,
then loads trainable weights from checkpoint
- --resume flag restores optimizer, loss (learnable tau), and scheduler state
- Training continues from the saved epoch
Usage:
python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- UAV-VisLoc processed at 512x512 (for segmentation/depth/normals)
- Dataset verified: 6744 drone, 74807 crops, median match 25.9m
- Known issue: 6 drones in route 06 outside satellite coverage
- Resize to model input size (224/256) in dataloader
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
