Fix README formulas: use ```math code fence for Gitea rendering
Gitea 1.25 renders math blocks via ```math fence reliably. Replaced $$...$$ with ```math blocks, inline math kept as backtick code. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
60
README.md
60
README.md
@@ -65,23 +65,31 @@ adapted with **LoRA** (rank=4 on Q/V in all 12 blocks).
|
||||
|
||||
### Text fusion (shared MLP for both branches)
|
||||
|
||||
$$\mathbf{z}_{\text{text}} = \text{MLP}\bigl([\mathbf{z}_1 \;;\; \mathbf{z}_2 \;;\; \mathbf{z}_3]\bigr)$$
|
||||
```math
|
||||
\mathbf{z}_{\text{text}} = \text{MLP}\bigl([\mathbf{z}_1 \;;\; \mathbf{z}_2 \;;\; \mathbf{z}_3]\bigr)
|
||||
```
|
||||
|
||||
where $[\mathbf{z}_1 ; \mathbf{z}_2 ; \mathbf{z}_3] \in \mathbb{R}^{B \times 2304}$ is the concatenation of three 768-dim DGTRS-CLIP embeddings, and
|
||||
where `[z₁ ; z₂ ; z₃] ∈ ℝ^{B×2304}` is the concatenation of three 768-dim DGTRS-CLIP embeddings, and
|
||||
|
||||
$$\text{MLP}: \text{Linear}(2304, 1024) \to \text{GELU} \to \text{Linear}(1024, 1024), \quad \mathbf{z}_{\text{text}} \in \mathbb{R}^{B \times 1024}$$
|
||||
```math
|
||||
\text{MLP}: \text{Linear}(2304, 1024) \to \text{GELU} \to \text{Linear}(1024, 1024), \quad \mathbf{z}_{\text{text}} \in \mathbb{R}^{B \times 1024}
|
||||
```
|
||||
|
||||
### Gated fusion (separate gates for query and gallery)
|
||||
|
||||
$$\mathbf{q} = \sigma(\alpha_q) \cdot \mathbf{d}_{\text{img}} + \bigl(1 - \sigma(\alpha_q)\bigr) \cdot \mathbf{d}_{\text{txt}} \qquad \text{(query branch)}$$
|
||||
```math
|
||||
\mathbf{q} = \sigma(\alpha_q) \cdot \mathbf{d}_{\text{img}} + \bigl(1 - \sigma(\alpha_q)\bigr) \cdot \mathbf{d}_{\text{txt}} \qquad \text{(query branch)}
|
||||
```
|
||||
|
||||
$$\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma(\alpha_g)\bigr) \cdot \mathbf{s}_{\text{txt}} \qquad \text{(gallery branch)}$$
|
||||
```math
|
||||
\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma(\alpha_g)\bigr) \cdot \mathbf{s}_{\text{txt}} \qquad \text{(gallery branch)}
|
||||
```
|
||||
|
||||
- $\alpha_q, \alpha_g$ — separate learnable scalars in logit-space, init $\sigma(\alpha) \approx 0.7$
|
||||
- $\sigma$ — sigmoid function
|
||||
- $\mathbf{d}_{\text{img}}, \mathbf{s}_{\text{img}} \in \mathbb{R}^{B \times 1024}$ — DINOv3+MONA image embeddings
|
||||
- $\mathbf{d}_{\text{txt}}, \mathbf{s}_{\text{txt}} \in \mathbb{R}^{B \times 1024}$ — fused text embeddings
|
||||
- For satellite images without captions: $\mathbf{s}_{\text{txt}} = \text{None} \Rightarrow \mathbf{g} = \mathbf{s}_{\text{img}}$
|
||||
- `α_q, α_g` — separate learnable scalars in logit-space, init `σ(α) ≈ 0.7`
|
||||
- `σ` — sigmoid function
|
||||
- `d_img, s_img ∈ ℝ^{B×1024}` — DINOv3+MONA image embeddings
|
||||
- `d_txt, s_txt ∈ ℝ^{B×1024}` — fused text embeddings
|
||||
- For satellite images without captions: `s_txt = None → g = s_img`
|
||||
|
||||
### Adaptation methods
|
||||
|
||||
@@ -92,30 +100,40 @@ $$\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma
|
||||
|
||||
**MONA adapter** (per block):
|
||||
|
||||
$$\mathbf{x} \leftarrow \mathbf{x} + \text{Up}_{64 \to 1024}\!\Bigl(\text{GELU}\bigl(\text{MonaOp}\bigl(\text{Down}_{1024 \to 64}(\hat{\mathbf{x}})\bigr)\bigr)\Bigr)$$
|
||||
```math
|
||||
\mathbf{x} \leftarrow \mathbf{x} + \text{Up}_{64 \to 1024}\!\Bigl(\text{GELU}\bigl(\text{MonaOp}\bigl(\text{Down}_{1024 \to 64}(\hat{\mathbf{x}})\bigr)\bigr)\Bigr)
|
||||
```
|
||||
|
||||
where $\hat{\mathbf{x}} = \gamma \cdot \text{LN}(\mathbf{x}) + \gamma_x \cdot \mathbf{x}$ (scaled LayerNorm, $\gamma$ init $10^{-6}$, $\gamma_x$ init $1$)
|
||||
where `x̂ = γ · LN(x) + γₓ · x` (scaled LayerNorm, `γ` init `10⁻⁶`, `γₓ` init `1`)
|
||||
|
||||
$$\text{MonaOp}(\mathbf{x}) = \frac{\text{DWConv}_{3 \times 3}(\mathbf{x}) + \text{DWConv}_{5 \times 5}(\mathbf{x}) + \text{DWConv}_{7 \times 7}(\mathbf{x})}{3} + \mathbf{x}$$
|
||||
```math
|
||||
\text{MonaOp}(\mathbf{x}) = \frac{\text{DWConv}_{3 \times 3}(\mathbf{x}) + \text{DWConv}_{5 \times 5}(\mathbf{x}) + \text{DWConv}_{7 \times 7}(\mathbf{x})}{3} + \mathbf{x}
|
||||
```
|
||||
|
||||
**LoRA** (per attention layer):
|
||||
|
||||
$$\mathbf{Q}' = \mathbf{Q} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_Q^T \mathbf{B}_Q^T, \qquad \mathbf{V}' = \mathbf{V} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_V^T \mathbf{B}_V^T$$
|
||||
```math
|
||||
\mathbf{Q}' = \mathbf{Q} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_Q^T \mathbf{B}_Q^T, \qquad \mathbf{V}' = \mathbf{V} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_V^T \mathbf{B}_V^T
|
||||
```
|
||||
|
||||
where $\mathbf{A} \in \mathbb{R}^{r \times d}$, $\mathbf{B} \in \mathbb{R}^{d \times r}$, $r = 4$
|
||||
where `A ∈ ℝ^{r×d}`, `B ∈ ℝ^{d×r}`, `r = 4`
|
||||
|
||||
### Loss function
|
||||
|
||||
Symmetric InfoNCE with learnable temperature (CLIP-style `logit_scale`):
|
||||
|
||||
$$\mathcal{L} = w_{q \to g} \cdot \mathcal{L}_{q \to g} + w_{g \to q} \cdot \mathcal{L}_{g \to q}$$
|
||||
```math
|
||||
\mathcal{L} = w_{q \to g} \cdot \mathcal{L}_{q \to g} + w_{g \to q} \cdot \mathcal{L}_{g \to q}
|
||||
```
|
||||
|
||||
$$\mathcal{L}_{q \to g} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{q}} \cdot \hat{\mathbf{g}}^T}{\tau},\; \text{targets}\right), \qquad \mathcal{L}_{g \to q} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{g}} \cdot \hat{\mathbf{q}}^T}{\tau},\; \text{targets}\right)$$
|
||||
```math
|
||||
\mathcal{L}_{q \to g} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{q}} \cdot \hat{\mathbf{g}}^T}{\tau},\; \text{targets}\right), \qquad \mathcal{L}_{g \to q} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{g}} \cdot \hat{\mathbf{q}}^T}{\tau},\; \text{targets}\right)
|
||||
```
|
||||
|
||||
- $\tau = 1 / \exp(\text{logit\_scale})$ — learnable scalar, clamped $\tau \in [0.01, 0.5]$, init $\tau_0 = 0.07$
|
||||
- $w_{q \to g} = 0.6$, $w_{g \to q} = 0.4$
|
||||
- $\text{targets} = [0, 1, 2, \ldots, B-1]$ — positives on diagonal
|
||||
- label smoothing $= 0.1$
|
||||
- `τ = 1 / exp(logit_scale)` — learnable scalar, clamped `τ ∈ [0.01, 0.5]`, init `τ₀ = 0.07`
|
||||
- `w_{q→g} = 0.6`, `w_{g→q} = 0.4`
|
||||
- `targets = [0, 1, 2, ..., B−1]` — positives on diagonal
|
||||
- label smoothing `= 0.1`
|
||||
|
||||
Loss and adapters run in **fp32** (AMP autocast disabled) to prevent gradient overflow.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user