Pikaliov/caption-test
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add architecture diagram to README, update docs for DGTRS text encoder

Browse Source 
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
pikaliov
2026-04-21 18:36:24 +03:00
parent 433fa40ed6
commit a47dd6308e
1 changed files with 29 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

32
README.md
View File

@@ -10,9 +10,35 @@ geo-localization (drone-to-satellite).
Asymmetric architecture with domain-specific image encoders and hierarchical text. Asymmetric architecture with domain-specific image encoders and hierarchical text.
``` ```
Query: drone_img (DINOv3 LVD) + L1/L2/L3 captions (LRSCLIP) -> GatedFusion -> query ┌─────────────────────────── QUERY BRANCH ───────────────────────────┐
Gallery: sat_img (DINOv3 SAT) -> gallery │ │
Loss: InfoNCE(query, gallery) │ drone_img ──► DINOv3 ViT-L/16 LVD ──► CLS [1024] │
│ (frozen, 303M) │ │
│ proj_drone [1024→512] │
│ │ │
│ L1 (overview) ──► DGTRS-CLIP ──► [768] ─┐ │
│ L2 (full desc) ──► DGTRS-CLIP ──► [768] ─┼─ concat [2304] │
│ L3 (fingerprint) ──► DGTRS-CLIP ──► [768] ─┘ │ │
│ (248 tok) MLP [2304→768→512] │
│ │ │
│ GatedFusion(img_512, text_512) │
│ │ │
│ L2-norm ──► query [512] │
└─────────────────────────────────────────────────────────────────────┘
┌─────────────────────────── GALLERY BRANCH ─────────────────────────┐
│ │
│ sat_img ──► DINOv3 ViT-L/16 SAT ──► CLS [1024] │
│ (frozen, 303M) │ │
│ proj_sat [1024→512] │
│ │ │
│ L2-norm ──► gallery [512] │
└─────────────────────────────────────────────────────────────────────┘
LOSS: InfoNCE(query, gallery) — symmetric, learnable temperature
weights: 0.6 × q→g + 0.4 × g→q
BASELINE: gate = 1.0 (text branch disabled, no DGTRS loaded)
``` ```
**Models:** **Models:**