Pikaliov/caption-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
37 Commits 4 Branches 0 Tags
471bc6444a8492152eb6d24173b405dc69bbab55
Commit Graph

14 Commits

Author SHA1 Message Date
pikaliov
471bc6444a Restore conv7x7 in MONA (full 3x3+5x5+7x7 multi-scale) 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 21:45:27 +03:00
pikaliov
11fc348ecd Add projection 1024→512, revert MONA bottleneck to 64 
- Add image_projection: Linear(1024→512) after DINOv3 CLS output
- Retrieval space: 512-dim (was 1024-dim native DINOv3)
- TextFusionMLP: 3×768→512→512 (was →1024→1024)
- GatedFusion operates in 512-dim
- MONA bottleneck restored to 64 (works at native 1024 inside DINOv3)
- Trainable: 8.9M (projection 0.5M + MONA 6.8M + LoRA 0.1K + MLP 1.5M)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 21:44:10 +03:00
pikaliov
d279a6e745 Optimize MONA: fp16, remove conv7x7, bottleneck 64→32 
- Remove forced fp32 cast in MONA forward (runs in AMP fp16 now)
- Remove conv7x7 from MonaOp (keep 3x3 + 5x5 only)
- Reduce default bottleneck from 64 to 32
- MONA params: 3.5M (was 7.0M, -50%)
- Total trainable: 7.0M (was 10.5M)
- Peak VRAM at bs=24: 18.6 GB (was 20.3 GB before fp16)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 21:40:23 +03:00
pikaliov
6b7bcae198 Add gradient checkpointing for DINOv3 and DGTRS-CLIP (bs 8→24) 
- DINOv3: checkpoint each of 24 transformer blocks (recompute on backward)
- DGTRS-CLIP: checkpoint each of 12 transformer blocks
- Enables batch_size=24 on RTX 4090 (was 8 without checkpointing)
- Peak VRAM: 20.3 GB at bs=24 (was OOM at bs=16 before)
- ~20-30% slower per step, but 3x more in-batch negatives (23 vs 7)
- Enabled by default (gradient_checkpointing=True in config)
- Update README with VRAM benchmarks and checkpointing docs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 21:34:31 +03:00
pikaliov
da2d2ea90e Switch to shared DINOv3 WEB encoder (saves ~4-5 GB VRAM) 
- Single DINOv3 WEB for both drone and satellite branches (shared_encoder=True default)
- One set of MONA adapters instead of two: 7M trainable vs 14M
- Total params: 438M (was 748M), trainable: 10.6M (was 17.6M)
- Asymmetric mode still available via shared_encoder=False
- Add gradient accumulation (grad_accum_steps, --grad-accum CLI flag)
- Update model summary in README

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 21:25:46 +03:00
pikaliov
a39f8a9655 Add MONA adapters for DINOv3 + LoRA for DGTRS-CLIP text encoder 
MONA (Multi-Cognitive One-Shot Nested Adaptation, CVPR 2025):
- 2 adapters per DINOv3 block (after MSA, after MLP) × 24 layers × 2 encoders
- MonaOp: parallel DWConv 3×3/5×5/7×7 + 1×1 projector
- ScaledLayerNorm + down(1024→64) + MonaOp + GELU + up(64→1024)
- 7M params per encoder, 14M total for drone+sat

LoRA (Low-Rank Adaptation):
- Q and V projections in all 12 DGTRS-CLIP transformer blocks
- rank=4, ~147K params total
- Replaces partial unfreeze (was ~7.6M for last block)

Both adapters run in fp32 (torch.amp.autocast disabled) to avoid
AMP gradient overflow. Frozen backbone layers still run in fp16.

Total trainable: 17.6M / 748M (2.35%)
  MONA (2×DINOv3): 14.0M
  LoRA (DGTRS):    147K
  TextFusionMLP:   3.4M
  Gates + logit:   3

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 19:24:01 +03:00
pikaliov
0c41c1f017 Remove projections (1024 native), add satellite text, dual GatedFusion 
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
  DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through

Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3

Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values

11.1M trainable / 734M total (1.51%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 19:01:30 +03:00
pikaliov
433fa40ed6 Replace broken LRSCLIPTextEncoder with official DGTRS architecture 
Root cause of NaN: our open_clip wrapper had 3 bugs:
1. Positional embeddings summed for all positions instead of masked
   (official: mask1 for pos 0-19, mask2 for pos 20-247)
2. open_clip uses batch-first transformer, DGTRS uses sequence-first
   (LND format with nn.MultiheadAttention)
3. open_clip tokenizer truncates to 77 tokens, DGTRS needs 248

Fix: copied official DGTRS text encoder architecture from
github.com/MitsuiChen14/DGTRS (Apache-2.0):
- src/models/dgtrs/model.py: DGTRSTextEncoder, build_model,
  load_dgtrs_text_encoder, tokenize_dgtrs
- src/models/dgtrs/simple_tokenizer.py: BPE tokenizer (248 tokens)
- src/models/dgtrs/bpe_simple_vocab_16e6.txt.gz: vocabulary

Removed: LRSCLIPTextEncoder class, open_clip dependency for text encoding

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:35:41 +03:00
pikaliov
a214320d81 Suppress open_clip 'no pretrained weights' warning for LRSCLIP init 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:26:59 +03:00
pikaliov
44bce3096c Add model save/load and --resume for training continuation 
- AsymmetricEncoder.save_checkpoint(): saves model_state + metadata
- AsymmetricEncoder.load_checkpoint(): rebuilds model with frozen backbones,
  then loads trainable weights from checkpoint
- --resume flag restores optimizer, loss (learnable tau), and scheduler state
- Training continues from the saved epoch

Usage:
  python -m src.training.train_gtauav --resume out/gtauav/with_text/ckpt_epoch004.pt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:14:54 +03:00
pikaliov
998d52cb57 Improve training: learnable temperature, per-group LR, warmup, augmentations 
Loss:
- Learnable temperature (CLIP-style logit_scale) with clamp [0.01, 0.5]
- Replaces fixed cosine schedule (still available via --no-learnable-temp)
- Default tau_init=0.07

Optimizer:
- Per-group LR: projections 1e-4, text encoder 1e-5 (10x lower)
- Learnable temperature included in projection param group

Scheduler:
- Linear warmup (2 epochs default) + cosine annealing
- Per-step scheduling (not per-epoch)

Augmentations (separate drone/satellite):
- Drone: RandomResizedCrop(0.7-1.0), HFlip, Rotation(15), ColorJitter,
  RandomGrayscale(0.05), GaussianBlur
- Satellite: RandomResizedCrop(0.7-1.0), HFlip, ColorJitter, RandomGrayscale
- Eval: clean Resize+CenterCrop (no augmentation)

Dataset: supports separate drone_transform/sat_transform args

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:07:17 +03:00
pikaliov
6ad9c4d149 Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder 
V3 architecture for CVGL caption validation on GTA-UAV-LR dataset:
- AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen)
  + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze)
- L1/L2/L3 hierarchical captions from VLM-generated descriptions
- TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion
- Segmentation filter: exclude images with >=90% background+water
- 10.9M trainable / 733M total params, 256x256 input
- coloredlogs + tqdm + emoji for training UX
- Baseline mode (--baseline): image-only, no text encoder loaded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 17:54:27 +03:00
pikaliov
abb3337f8d Rewrite: GatedFusion architecture + UAV-GeoLoc dataset 
Architecture v2:
- Query branch: drone + text -> GatedFusion -> proj -> query_emb
- Gallery branch: satellite -> proj -> gallery_emb
- Single InfoNCE loss (asymmetric 0.6/0.4)
- GatedFusion: learnable gated addition (sigma(alpha)*img + (1-sigma(alpha))*text)
- Baseline mode: gate=1.0 (text ignored)

Dataset:
- UAV-GeoLoc loader with template captions from path metadata
- 27 terrain types with predefined features
- Random positive crop sampling per epoch

Configs: balanced (gate=0.7), baseline (no text), text_heavy (gate=0.3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-17 17:13:00 +03:00
Pikaliov
2ce4017ebd Initial commit: caption quality test on UAV-VisLoc 
Self-contained experimental track validating generated text captions
via retrieval R@1 lift on UAV-VisLoc.

Architecture: GeoRSCLIP ViT-B/32 dual encoder, 512-dim shared space.
Loss: 4-term InfoNCE (img-img + sat-cap + drone-cap + cap-cap)
      with cosine temperature decay, PALW-like curriculum.
Metric: delta R@1 (with text - without text) >= +3% => PASS.

Gin-configured (balanced / baseline_no_text / text_heavy variants).
Follows NADEZHDA code style.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
 2026-04-17 00:04:46 +03:00