Pikaliov/caption-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
16 Commits 4 Branches 0 Tags
219bb779ebc5228ac653511ffe626e8c940e32a5
Commit Graph

16 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
pikaliov
219bb779eb Update docs: full architecture with tensor shapes, formulas, optimizer details 
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.

CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:47:38 +03:00
pikaliov
bcb01bcb6d Fix NaN: compute loss in fp32 outside AMP autocast 
Root cause: GradScaler scales gradients by ~65536 in fp16, causing
logit_scale.exp() gradient to overflow. The learnable temperature
and similarity logits must stay in fp32.

Fix: model forward runs inside autocast(fp16), but loss computation
(similarity @ temperature + cross_entropy) runs outside in fp32.

Also: clamp logit_scale in logit-space before exp() and force
similarity computation to fp32.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:42:29 +03:00
pikaliov
f41a0f27fe Fix epoch display: show 1/10 instead of 0/9 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:37:05 +03:00
pikaliov
a47dd6308e Add architecture diagram to README, update docs for DGTRS text encoder 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:36:24 +03:00
pikaliov
433fa40ed6 Replace broken LRSCLIPTextEncoder with official DGTRS architecture 
Root cause of NaN: our open_clip wrapper had 3 bugs:
1. Positional embeddings summed for all positions instead of masked
   (official: mask1 for pos 0-19, mask2 for pos 20-247)
2. open_clip uses batch-first transformer, DGTRS uses sequence-first
   (LND format with nn.MultiheadAttention)
3. open_clip tokenizer truncates to 77 tokens, DGTRS needs 248

Fix: copied official DGTRS text encoder architecture from
github.com/MitsuiChen14/DGTRS (Apache-2.0):
- src/models/dgtrs/model.py: DGTRSTextEncoder, build_model,
  load_dgtrs_text_encoder, tokenize_dgtrs
- src/models/dgtrs/simple_tokenizer.py: BPE tokenizer (248 tokens)
- src/models/dgtrs/bpe_simple_vocab_16e6.txt.gz: vocabulary

Removed: LRSCLIPTextEncoder class, open_clip dependency for text encoding

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:35:41 +03:00
pikaliov
a214320d81 Suppress open_clip 'no pretrained weights' warning for LRSCLIP init 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:26:59 +03:00
pikaliov
fa32b2e67f Suppress spurious lr_scheduler.step() warning from PyTorch 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:24:54 +03:00
pikaliov
517b87d3d8 Fix scheduler warning: use last_epoch instead of step() loop on resume 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:23:17 +03:00
pikaliov
905b9867c8 Add 80/20 random split (replaces cross-area 46/54 split) 
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
  shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:19:37 +03:00
pikaliov
44bce3096c Add model save/load and --resume for training continuation 
- AsymmetricEncoder.save_checkpoint(): saves model_state + metadata
- AsymmetricEncoder.load_checkpoint(): rebuilds model with frozen backbones,
  then loads trainable weights from checkpoint
- --resume flag restores optimizer, loss (learnable tau), and scheduler state
- Training continues from the saved epoch

Usage:
  python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:14:54 +03:00
pikaliov
3014a5def8 Update docs: training improvements (learnable temp, augmentations, warmup) 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:08:06 +03:00
pikaliov
998d52cb57 Improve training: learnable temperature, per-group LR, warmup, augmentations 
Loss:
- Learnable temperature (CLIP-style logit_scale) with clamp [0.01, 0.5]
- Replaces fixed cosine schedule (still available via --no-learnable-temp)
- Default tau_init=0.07

Optimizer:
- Per-group LR: projections 1e-4, text encoder 1e-5 (10x lower)
- Learnable temperature included in projection param group

Scheduler:
- Linear warmup (2 epochs default) + cosine annealing
- Per-step scheduling (not per-epoch)

Augmentations (separate drone/satellite):
- Drone: RandomResizedCrop(0.7-1.0), HFlip, Rotation(15), ColorJitter,
  RandomGrayscale(0.05), GaussianBlur
- Satellite: RandomResizedCrop(0.7-1.0), HFlip, ColorJitter, RandomGrayscale
- Eval: clean Resize+CenterCrop (no augmentation)

Dataset: supports separate drone_transform/sat_transform args

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:07:17 +03:00
pikaliov
6ad9c4d149 Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder 
V3 architecture for CVGL caption validation on GTA-UAV-LR dataset:
- AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen)
  + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze)
- L1/L2/L3 hierarchical captions from VLM-generated descriptions
- TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion
- Segmentation filter: exclude images with >=90% background+water
- 10.9M trainable / 733M total params, 256x256 input
- coloredlogs + tqdm + emoji for training UX
- Baseline mode (--baseline): image-only, no text encoder loaded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 17:54:27 +03:00
pikaliov
5da791801c Update docs: target-size 512, dataset verification results 
- UAV-VisLoc processed at 512x512 (for segmentation/depth/normals)
- Dataset verified: 6744 drone, 74807 crops, median match 25.9m
- Known issue: 6 drones in route 06 outside satellite coverage
- Resize to model input size (224/256) in dataloader

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-18 02:41:31 +03:00
pikaliov
abb3337f8d Rewrite: GatedFusion architecture + UAV-GeoLoc dataset 
Architecture v2:
- Query branch: drone + text -> GatedFusion -> proj -> query_emb
- Gallery branch: satellite -> proj -> gallery_emb
- Single InfoNCE loss (asymmetric 0.6/0.4)
- GatedFusion: learnable gated addition (sigma(alpha)*img + (1-sigma(alpha))*text)
- Baseline mode: gate=1.0 (text ignored)

Dataset:
- UAV-GeoLoc loader with template captions from path metadata
- 27 terrain types with predefined features
- Random positive crop sampling per epoch

Configs: balanced (gate=0.7), baseline (no text), text_heavy (gate=0.3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-17 17:13:00 +03:00
Pikaliov
2ce4017ebd Initial commit: caption quality test on UAV-VisLoc 
Self-contained experimental track validating generated text captions
via retrieval R@1 lift on UAV-VisLoc.

Architecture: GeoRSCLIP ViT-B/32 dual encoder, 512-dim shared space.
Loss: 4-term InfoNCE (img-img + sat-cap + drone-cap + cap-cap)
      with cosine temperature decay, PALW-like curriculum.
Metric: delta R@1 (with text - without text) >= +3% => PASS.

Gin-configured (balanced / baseline_no_text / text_heavy variants).
Follows NADEZHDA code style.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-17 00:04:46 +03:00