Remove projections (1024 native), add satellite text, dual GatedFusion
Architecture changes: - Removed proj_drone/proj_sat (1024→512): retrieval space is now DINOv3 native 1024-dim, no information loss from projection - TextFusionMLP: 2304→1024→1024 (was 2304→768→512), shared between branches - Gallery branch now uses satellite captions (L1/L2/L3) via shared TextFusionMLP - Two separate GatedFusion gates: α_q (query) and α_g (gallery) - For sat images without captions (~57%): gate passes image features through Dataset changes: - GTAUAVDataset now loads satellite captions from caption index - collate_gtauav_batch includes sat_caption_l1/l2/l3 Training loop: - Passes satellite captions to model forward - Logs both gate_q and gate_g values 11.1M trainable / 734M total (1.51%) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -145,7 +145,8 @@ class InfoNCELoss(nn.Module):
|
||||
weight_b2a=self.weight_g2q,
|
||||
)
|
||||
|
||||
gate = embeddings.get("gate", 1.0)
|
||||
gate_q = embeddings.get("gate_q", embeddings.get("gate", 1.0))
|
||||
gate_g = embeddings.get("gate_g", 1.0)
|
||||
|
||||
if isinstance(tau, float):
|
||||
tau_out = torch.tensor(tau, device=loss.device)
|
||||
@@ -155,5 +156,6 @@ class InfoNCELoss(nn.Module):
|
||||
return {
|
||||
"total": loss,
|
||||
"temperature": tau_out,
|
||||
"gate": torch.tensor(gate, device=loss.device),
|
||||
"gate_q": torch.tensor(gate_q, device=loss.device),
|
||||
"gate_g": torch.tensor(gate_g, device=loss.device),
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user