From 4c862b6ec774b0d1d33b3e1fe44d9de651bdd7be Mon Sep 17 00:00:00 2001 From: pikaliov Date: Tue, 21 Apr 2026 19:28:08 +0300 Subject: [PATCH] Fix README formulas: use ```math code fence for Gitea rendering Gitea 1.25 renders math blocks via ```math fence reliably. Replaced $$...$$ with ```math blocks, inline math kept as backtick code. Co-Authored-By: Claude Opus 4.6 (1M context) --- README.md | 60 ++++++++++++++++++++++++++++++++++++------------------- 1 file changed, 39 insertions(+), 21 deletions(-) diff --git a/README.md b/README.md index c2e5a9b..a6e0965 100644 --- a/README.md +++ b/README.md @@ -65,23 +65,31 @@ adapted with **LoRA** (rank=4 on Q/V in all 12 blocks). ### Text fusion (shared MLP for both branches) -$$\mathbf{z}_{\text{text}} = \text{MLP}\bigl([\mathbf{z}_1 \;;\; \mathbf{z}_2 \;;\; \mathbf{z}_3]\bigr)$$ +```math +\mathbf{z}_{\text{text}} = \text{MLP}\bigl([\mathbf{z}_1 \;;\; \mathbf{z}_2 \;;\; \mathbf{z}_3]\bigr) +``` -where $[\mathbf{z}_1 ; \mathbf{z}_2 ; \mathbf{z}_3] \in \mathbb{R}^{B \times 2304}$ is the concatenation of three 768-dim DGTRS-CLIP embeddings, and +where `[z₁ ; z₂ ; z₃] ∈ ℝ^{B×2304}` is the concatenation of three 768-dim DGTRS-CLIP embeddings, and -$$\text{MLP}: \text{Linear}(2304, 1024) \to \text{GELU} \to \text{Linear}(1024, 1024), \quad \mathbf{z}_{\text{text}} \in \mathbb{R}^{B \times 1024}$$ +```math +\text{MLP}: \text{Linear}(2304, 1024) \to \text{GELU} \to \text{Linear}(1024, 1024), \quad \mathbf{z}_{\text{text}} \in \mathbb{R}^{B \times 1024} +``` ### Gated fusion (separate gates for query and gallery) -$$\mathbf{q} = \sigma(\alpha_q) \cdot \mathbf{d}_{\text{img}} + \bigl(1 - \sigma(\alpha_q)\bigr) \cdot \mathbf{d}_{\text{txt}} \qquad \text{(query branch)}$$ +```math +\mathbf{q} = \sigma(\alpha_q) \cdot \mathbf{d}_{\text{img}} + \bigl(1 - \sigma(\alpha_q)\bigr) \cdot \mathbf{d}_{\text{txt}} \qquad \text{(query branch)} +``` -$$\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma(\alpha_g)\bigr) \cdot \mathbf{s}_{\text{txt}} \qquad \text{(gallery branch)}$$ +```math +\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma(\alpha_g)\bigr) \cdot \mathbf{s}_{\text{txt}} \qquad \text{(gallery branch)} +``` -- $\alpha_q, \alpha_g$ — separate learnable scalars in logit-space, init $\sigma(\alpha) \approx 0.7$ -- $\sigma$ — sigmoid function -- $\mathbf{d}_{\text{img}}, \mathbf{s}_{\text{img}} \in \mathbb{R}^{B \times 1024}$ — DINOv3+MONA image embeddings -- $\mathbf{d}_{\text{txt}}, \mathbf{s}_{\text{txt}} \in \mathbb{R}^{B \times 1024}$ — fused text embeddings -- For satellite images without captions: $\mathbf{s}_{\text{txt}} = \text{None} \Rightarrow \mathbf{g} = \mathbf{s}_{\text{img}}$ +- `α_q, α_g` — separate learnable scalars in logit-space, init `σ(α) ≈ 0.7` +- `σ` — sigmoid function +- `d_img, s_img ∈ ℝ^{B×1024}` — DINOv3+MONA image embeddings +- `d_txt, s_txt ∈ ℝ^{B×1024}` — fused text embeddings +- For satellite images without captions: `s_txt = None → g = s_img` ### Adaptation methods @@ -92,30 +100,40 @@ $$\mathbf{g} = \sigma(\alpha_g) \cdot \mathbf{s}_{\text{img}} + \bigl(1 - \sigma **MONA adapter** (per block): -$$\mathbf{x} \leftarrow \mathbf{x} + \text{Up}_{64 \to 1024}\!\Bigl(\text{GELU}\bigl(\text{MonaOp}\bigl(\text{Down}_{1024 \to 64}(\hat{\mathbf{x}})\bigr)\bigr)\Bigr)$$ +```math +\mathbf{x} \leftarrow \mathbf{x} + \text{Up}_{64 \to 1024}\!\Bigl(\text{GELU}\bigl(\text{MonaOp}\bigl(\text{Down}_{1024 \to 64}(\hat{\mathbf{x}})\bigr)\bigr)\Bigr) +``` -where $\hat{\mathbf{x}} = \gamma \cdot \text{LN}(\mathbf{x}) + \gamma_x \cdot \mathbf{x}$ (scaled LayerNorm, $\gamma$ init $10^{-6}$, $\gamma_x$ init $1$) +where `x̂ = γ · LN(x) + γₓ · x` (scaled LayerNorm, `γ` init `10⁻⁶`, `γₓ` init `1`) -$$\text{MonaOp}(\mathbf{x}) = \frac{\text{DWConv}_{3 \times 3}(\mathbf{x}) + \text{DWConv}_{5 \times 5}(\mathbf{x}) + \text{DWConv}_{7 \times 7}(\mathbf{x})}{3} + \mathbf{x}$$ +```math +\text{MonaOp}(\mathbf{x}) = \frac{\text{DWConv}_{3 \times 3}(\mathbf{x}) + \text{DWConv}_{5 \times 5}(\mathbf{x}) + \text{DWConv}_{7 \times 7}(\mathbf{x})}{3} + \mathbf{x} +``` **LoRA** (per attention layer): -$$\mathbf{Q}' = \mathbf{Q} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_Q^T \mathbf{B}_Q^T, \qquad \mathbf{V}' = \mathbf{V} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_V^T \mathbf{B}_V^T$$ +```math +\mathbf{Q}' = \mathbf{Q} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_Q^T \mathbf{B}_Q^T, \qquad \mathbf{V}' = \mathbf{V} + \frac{\alpha}{r} \cdot \mathbf{x} \mathbf{A}_V^T \mathbf{B}_V^T +``` -where $\mathbf{A} \in \mathbb{R}^{r \times d}$, $\mathbf{B} \in \mathbb{R}^{d \times r}$, $r = 4$ +where `A ∈ ℝ^{r×d}`, `B ∈ ℝ^{d×r}`, `r = 4` ### Loss function Symmetric InfoNCE with learnable temperature (CLIP-style `logit_scale`): -$$\mathcal{L} = w_{q \to g} \cdot \mathcal{L}_{q \to g} + w_{g \to q} \cdot \mathcal{L}_{g \to q}$$ +```math +\mathcal{L} = w_{q \to g} \cdot \mathcal{L}_{q \to g} + w_{g \to q} \cdot \mathcal{L}_{g \to q} +``` -$$\mathcal{L}_{q \to g} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{q}} \cdot \hat{\mathbf{g}}^T}{\tau},\; \text{targets}\right), \qquad \mathcal{L}_{g \to q} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{g}} \cdot \hat{\mathbf{q}}^T}{\tau},\; \text{targets}\right)$$ +```math +\mathcal{L}_{q \to g} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{q}} \cdot \hat{\mathbf{g}}^T}{\tau},\; \text{targets}\right), \qquad \mathcal{L}_{g \to q} = \text{CrossEntropy}\!\left(\frac{\hat{\mathbf{g}} \cdot \hat{\mathbf{q}}^T}{\tau},\; \text{targets}\right) +``` -- $\tau = 1 / \exp(\text{logit\_scale})$ — learnable scalar, clamped $\tau \in [0.01, 0.5]$, init $\tau_0 = 0.07$ -- $w_{q \to g} = 0.6$, $w_{g \to q} = 0.4$ -- $\text{targets} = [0, 1, 2, \ldots, B-1]$ — positives on diagonal -- label smoothing $= 0.1$ +- `τ = 1 / exp(logit_scale)` — learnable scalar, clamped `τ ∈ [0.01, 0.5]`, init `τ₀ = 0.07` +- `w_{q→g} = 0.6`, `w_{g→q} = 0.4` +- `targets = [0, 1, 2, ..., B−1]` — positives on diagonal +- label smoothing `= 0.1` Loss and adapters run in **fp32** (AMP autocast disabled) to prevent gradient overflow.