Update dataset path (SSD_2_TB)

This commit is contained in:
pikaliov
2026-05-08 09:52:15 +03:00
parent 455ae2e99f
commit 7b3acc633a
25 changed files with 59 additions and 59 deletions

View File

@@ -175,20 +175,20 @@ Eval: Resize(256) + CenterCrop(256) + ImageNet normalization.
## Датасет: GTA-UAV-LR
- **RGB:** `/home/servml/Документы/datasets/GTA-UAV-LR/`
- **RGB:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR/`
- Drone: 33,763 PNG (512x384), altitudes 100-600m
- Satellite: 14,640 PNG (256x256 RGBA)
- Pairs: `cross-area-drone2sate-{train,test}.json` (primary split)
- Metadata: `*_drone_meta.csv` (height, yaw, roll, pitch)
- Origin: GTA V simulation (Los Santos)
- **Captions:** `/home/servml/Документы/datasets/GTA-UAV-LR-captions/`
- **Captions:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions/`
- Drone: 33,411 JSON (32,635 multi-paragraph P1/P2/P3 + 776 short water-only)
- Satellite: 6,546 JSON (все multi-paragraph)
- Формат: 3 абзаца (P1 Inventory + P2 Spatial + P3 Fingerprint)
- Token counts: ~430 output tokens per caption
- **Segmentation:** `/home/servml/Документы/datasets/GTA-UAV-LR-aug/`
- **Segmentation:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/`
- 48,403 images, 17 классов (background, building, road, vegetation, water, ...)
- Modalities: segm/, depth/, edge/, chm/, safetensors/
- Query: 512x512, DB: 256x256
@@ -331,7 +331,7 @@ python -m scripts.compare_runs \
### UAV-VisLoc Prepare
- **Путь:** `/home/servml/Документы/code/Yaroslav/UAV-VisLoc-prepare/scripts/prepare_dataset.py`
- **Статус:** выполнен (2026-04-17), данные в `/home/servml/Документы/datasets/UAV_VisLoc_processed/` (25 GB)
- **Статус:** выполнен (2026-04-17), данные в `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/UAV_VisLoc_processed/` (25 GB)
- **Задача:** нарезка satellite кропов 512x512, stride 256 + resize drone -> 512x512
- **Подробности:** см. ниже
@@ -376,7 +376,7 @@ Binary masks — natural FiLM gates. Modality dropout: text 0.3, CHM 0.5, seg 0.
## Датасеты (справочник)
### UAV-VisLoc
- **Путь:** `/home/servml/Документы/datasets/UAV_VisLoc_dataset/`
- **Путь:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/UAV_VisLoc_dataset/`
- **Структура:** 11 маршрутов (папки `01`-`11`), каждая содержит:
- `drone/` — drone-снимки (`XX_NNNN.JPG`)
- `satelliteXX.tif` — спутниковая карта
@@ -403,8 +403,8 @@ Binary masks — natural FiLM gates. Modality dropout: text 0.3, CHM 0.5, seg 0.
### Запуск
```bash
python scripts/prepare_dataset.py \
--src /home/servml/Документы/datasets/UAV_VisLoc_dataset \
--dst /home/servml/Документы/datasets/UAV_VisLoc_processed \
--src /media/servml/SSD_2_2TB/datasets/cvgl_datasets/UAV_VisLoc_dataset \
--dst /media/servml/SSD_2_2TB/datasets/cvgl_datasets/UAV_VisLoc_processed \
--crop-size 512 --stride 256 --target-size 512
```

View File

@@ -2,8 +2,8 @@
**Дата анализа:** 2026-04-21
**Метод:** Эмпирический анализ данных на диске + статья arXiv:2409.16925 + GitHub-репозиторий авторов
**Путь к данным:** `/home/servml/Документы/datasets/GTA-UAV-LR/`
**Путь к аугментациям:** `/home/servml/Документы/datasets/GTA-UAV-LR-aug/`
**Путь к данным:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR/`
**Путь к аугментациям:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/`
---
@@ -284,7 +284,7 @@
## 8. АУГМЕНТИРОВАННЫЙ НАБОР (GTA-UAV-LR-aug)
**Путь:** `/home/servml/Документы/datasets/GTA-UAV-LR-aug/`
**Путь:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/`
**Объём:** ~71 GB
### 8.1. Сгенерированные модальности

View File

@@ -377,9 +377,9 @@ that list it as a valid candidate. `n_scored_g2q` is reported in metrics for tra
### V3 — GTA-UAV + DINOv3 + DGTRS-CLIP (active)
**Dataset:** GTA-UAV-LR (33K drone + 14K satellite, GTA V synthetic)
- RGB: `/home/servml/Документы/datasets/GTA-UAV-LR/`
- Captions: `/home/servml/Документы/datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM)
- Segmentation: `/home/servml/Документы/datasets/GTA-UAV-LR-aug/` (17 classes)
- RGB: `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR/`
- Captions: `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions/` (40K JSON, 3-paragraph VLM)
- Segmentation: `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/` (17 classes)
- Seg filter: 37,498 passed / 10,905 excluded (>=90% background+water)
- Split: 80/20 random (26,966 train / 6,742 test → 24,891/6,252 after seg filter)

View File

@@ -2,7 +2,7 @@
**Дата анализа:** 2026-04-17
**Метод:** Эмпирический анализ данных на диске + статья arXiv:2405.11936 + GitHub-репозиторий авторов
**Путь к данным:** `/home/servml/Документы/datasets/UAV_VisLoc_dataset/`
**Путь к данным:** `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/UAV_VisLoc_dataset/`
---

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -1,8 +1,8 @@
# Pipeline: GTA-UAV-LR with text captions, servml workstation paths.
PipelineConfig.train_json = 'meta/train_80.json'
PipelineConfig.test_json = 'meta/test_20.json'
PipelineConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/home/servml/Документы/datasets/GTA-UAV-LR-captions'
PipelineConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PipelineConfig.caption_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions'
PipelineConfig.filter_meta = 'meta/seg_filter.json'
PipelineConfig.epochs = 10

View File

@@ -2,8 +2,8 @@
# scripts/filter_segmentation.py. Independent from training pipeline.
# Inputs.
PreprocessConfig.rgb_root = '/home/servml/Документы/datasets/GTA-UAV-LR'
PreprocessConfig.segm_root = '/home/servml/Документы/datasets/GTA-UAV-LR-aug/segm'
PreprocessConfig.rgb_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR'
PreprocessConfig.segm_root = '/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/segm'
# make_split.py — 80/20 split with seed=42.
PreprocessConfig.split_ratio = 0.8

View File

@@ -174,8 +174,8 @@ TrainConfigGTAUAV.output_dir = "out/gtauav/balanced_asym"
```python
# src/training/train_gtauav.py (module level)
_RGB_ROOT = "/home/servml/Документы/datasets/GTA-UAV-LR"
_CAPTION_ROOT = "/home/servml/Документы/datasets/GTA-UAV-LR-captions"
_RGB_ROOT = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR"
_CAPTION_ROOT = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions"
_TRAIN_JSON = "meta/train_80.json"
_TEST_JSON = "meta/test_20.json"
_DINO_WEB = "nn_models/DINO_WEB/dinov3-vitl16-pretrain-lvd1689m.pth"
@@ -183,17 +183,17 @@ _DINO_SAT = "nn_models/DINO_SAT/model.safetensors"
_LRSCLIP = "nn_models/LRSCLIP/DGTRS-CLIP-ViT-L-14.pt"
# src/datasets/gtauav_dataset.py (module level) — ДУБЛЬ:
_RGB_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR")
_CAPTION_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR-captions")
_RGB_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR")
_CAPTION_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions")
# scripts/make_split.py (module level):
_RGB_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR")
_RGB_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR")
# scripts/filter_segmentation.py (module level):
SEGM_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR-aug/segm")
SEGM_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/segm")
```
**Главная проблема — не сам факт хардкода**, а то, что один и тот же путь `/home/servml/Документы/datasets/GTA-UAV-LR` дублируется в **трёх** местах: train_gtauav.py + gtauav_dataset.py + make_split.py. Если переехать на другую машину — нужно править 3 файла; если забыть один — silent breakage.
**Главная проблема — не сам факт хардкода**, а то, что один и тот же путь `/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR` дублируется в **трёх** местах: train_gtauav.py + gtauav_dataset.py + make_split.py. Если переехать на другую машину — нужно править 3 файла; если забыть один — silent breakage.
**Масштаб:** 4 файла, ~10 module-level констант пути.

View File

@@ -25,7 +25,7 @@ from tqdm import tqdm
LOGGER = logging.getLogger("caption_test.filter_seg")
SEGM_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR-aug/segm")
SEGM_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/segm")
EXCLUDE_CLASSES = {0, 4} # background, water
DEFAULT_THRESHOLD = 0.90

View File

@@ -19,7 +19,7 @@ import coloredlogs
LOGGER = logging.getLogger("caption_test.make_split")
_RGB_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR")
_RGB_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR")
def main() -> None:

View File

@@ -19,8 +19,8 @@ class PipelineConfig:
# Data inputs.
train_json: str = "meta/train_80.json",
test_json: str = "meta/test_20.json",
rgb_root: str = "/home/servml/Документы/datasets/GTA-UAV-LR",
caption_root: str = "/home/servml/Документы/datasets/GTA-UAV-LR-captions",
rgb_root: str = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR",
caption_root: str = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions",
filter_meta: str | None = None,
# Training schedule.
epochs: int = 10,

View File

@@ -18,8 +18,8 @@ class PreprocessConfig:
def __init__(
self,
# Inputs.
rgb_root: str = "/home/servml/Документы/datasets/GTA-UAV-LR",
segm_root: str = "/home/servml/Документы/datasets/GTA-UAV-LR-aug/segm",
rgb_root: str = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR",
segm_root: str = "/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-aug/segm",
# make_split.py params.
split_ratio: float = 0.8,
split_seed: int = 42,

View File

@@ -30,8 +30,8 @@ LOGGER = logging.getLogger("caption_test.gtauav_dataset")
coloredlogs.install(level="INFO", logger=LOGGER, fmt="%(asctime)s %(name)s %(levelname)s %(message)s")
# Default paths.
_RGB_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR")
_CAPTION_ROOT = Path("/home/servml/Документы/datasets/GTA-UAV-LR-captions")
_RGB_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR")
_CAPTION_ROOT = Path("/media/servml/SSD_2_2TB/datasets/cvgl_datasets/GTA-UAV-LR-captions")
_EMPTY_CAPTION = ""
# Regex to split P1/P2/P3 sections.