Pikaliov/caption-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
19 Commits 4 Branches 0 Tags
082f82f138868c1ba867b8c12755aaa870df4ba7
Commit Graph

8 Commits

Author SHA1 Message Date
pikaliov
082f82f138 Update docs: MONA/LoRA architecture, LaTeX formulas, param summary 
README: LaTeX math formulas for text fusion, gated fusion, MONA adapter,
LoRA, and InfoNCE loss. Added adaptation methods table (MONA + LoRA).
Updated model summary to 17.6M/748M (2.35%).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 19:26:01 +03:00
pikaliov
0c41c1f017 Remove projections (1024 native), add satellite text, dual GatedFusion 
Architecture changes:
- Removed proj_drone/proj_sat (1024→512): retrieval space is now
  DINOv3 native 1024-dim, no information loss from projection
- TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches
- Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP
- Two separate GatedFusion gates: α_q (query) and α_g (gallery)
- For sat images without captions (~57%): gate passes image features through

Dataset changes:
- GTAUAVDataset now loads satellite captions from caption index
- collate_gtauav_batch includes sat_caption_l1/l2/l3

Training loop:
- Passes satellite captions to model forward
- Logs both gate_q and gate_g values

11.1M trainable / 734M total (1.51%)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 19:01:30 +03:00
pikaliov
219bb779eb Update docs: full architecture with tensor shapes, formulas, optimizer details 
README: architecture diagram with tensor dimensions, L1/L2/L3 text hierarchy
description, text fusion formula, InfoNCE loss formula with learnable
temperature, metrics table, optimizer/scheduler details with per-group LR,
augmentation table, model parameter summary.

CLAUDE.md: updated to DGTRS-CLIP (official architecture), loss formula,
optimizer/scheduler details, text encoder architecture notes.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:47:38 +03:00
pikaliov
905b9867c8 Add 80/20 random split (replaces cross-area 46/54 split) 
- scripts/make_split.py: merges cross-area train+test (33,708 pairs),
  shuffles with seed=42, splits 80/20
- meta/train_80.json (26,966) + meta/test_20.json (6,742)
- After seg filter: 24,891 train / 6,252 test
- Default paths in train_gtauav.py updated to use new split

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:19:37 +03:00
pikaliov
3014a5def8 Update docs: training improvements (learnable temp, augmentations, warmup) 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 18:08:06 +03:00
pikaliov
6ad9c4d149 Add GTA-UAV experiment: asymmetric DINOv3 + LRSCLIP text encoder 
V3 architecture for CVGL caption validation on GTA-UAV-LR dataset:
- AsymmetricEncoder: DINOv3 ViT-L/16 (LVD drone + SAT satellite, frozen)
  + LRSCLIP/DGTRS-CLIP ViT-L-14 text encoder (248 tok, partial unfreeze)
- L1/L2/L3 hierarchical captions from VLM-generated descriptions
- TextFusionMLP (concat 3x768 -> MLP -> 512) + GatedFusion
- Segmentation filter: exclude images with >=90% background+water
- 10.9M trainable / 733M total params, 256x256 input
- coloredlogs + tqdm + emoji for training UX
- Baseline mode (--baseline): image-only, no text encoder loaded

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-21 17:54:27 +03:00
pikaliov
5da791801c Update docs: target-size 512, dataset verification results 
- UAV-VisLoc processed at 512x512 (for segmentation/depth/normals)
- Dataset verified: 6744 drone, 74807 crops, median match 25.9m
- Known issue: 6 drones in route 06 outside satellite coverage
- Resize to model input size (224/256) in dataloader

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-18 02:41:31 +03:00
pikaliov
abb3337f8d Rewrite: GatedFusion architecture + UAV-GeoLoc dataset 
Architecture v2:
- Query branch: drone + text -> GatedFusion -> proj -> query_emb
- Gallery branch: satellite -> proj -> gallery_emb
- Single InfoNCE loss (asymmetric 0.6/0.4)
- GatedFusion: learnable gated addition (sigma(alpha)*img + (1-sigma(alpha))*text)
- Baseline mode: gate=1.0 (text ignored)

Dataset:
- UAV-GeoLoc loader with template captions from path metadata
- 27 terrain types with predefined features
- Random positive crop sampling per epoch

Configs: balanced (gate=0.7), baseline (no text), text_heavy (gate=0.3)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-04-17 17:13:00 +03:00