184 lines
11 KiB
Markdown
184 lines
11 KiB
Markdown
# Шаг 4b — Что изменилось
|
|
|
|
---
|
|
|
|
## 1. Новая точка входа — `src/main.py` + добавлена debug-конфигурация `Main entry point` в `launch.json`
|
|
|
|
**Пример запуска отладки:**
|
|
```bash
|
|
python src/training/train_gtauav.py --config conf/gtauav_balanced.gin
|
|
```
|
|
|
|
## 2. Тесты
|
|
|
|
Добавлены `tests\conftest.py & test_trainer.py`
|
|
|
|
### Тест загрузки конфигураций в трейнер
|
|
Файл `test_trainer.py`
|
|
|
|
**Пример запуска отладки:**
|
|
```bash
|
|
python -m pytest tests/test_trainer.py
|
|
```
|
|
|
|
**Лог:**
|
|
|
|
```bash
|
|
=========================================== test session starts ===========================================
|
|
platform linux -- Python 3.10.19, pytest-9.0.3, pluggy-1.6.0
|
|
rootdir: /home/servml/Документы/code/Yaroslav/caption-test
|
|
collected 21 items
|
|
|
|
tests/test_trainer.py . ✅ _SUPPORTED_BACKBONES must be a frozenset (immutable, hashable).
|
|
. ✅ Exactly dinov3 and stripnet are supported in the current refactor.
|
|
. ✅ ModelsConfig union mirrors _SUPPORTED_BACKBONES (dinov3 | stripnet).
|
|
.
|
|
Supported backbones pass _validate_backbone silently.
|
|
✅ dinov3
|
|
. ✅ stripnet
|
|
.
|
|
Unsupported backbones (incl. sofia) raise NotImplementedError, not ImportError.
|
|
✅ sofia_v1
|
|
. ✅ sofia_v71
|
|
. ✅ mistral_42b
|
|
. ✅ empty
|
|
.
|
|
Trainer(...) instantiates from every real preset's loaded cfgs.
|
|
✅ gtauav_balanced
|
|
. ✅ gtauav_balanced_asym
|
|
. ✅ gtauav_baseline
|
|
. ✅ gtauav_baseline_asym
|
|
. ✅ gtauav_image_heavy
|
|
. ✅ gtauav_text_heavy
|
|
. ✅ gtauav_balanced_stripnet
|
|
. ✅ gtauav_balanced_stripnet_unfrozen
|
|
. ✅ gtauav_baseline_stripnet
|
|
. ✅ gtauav_baseline_stripnet_unfrozen
|
|
. ✅ All runtime fields are None / 0 / [] before .train() is called.
|
|
. ✅ Trainer.train() takes only `self` — main.py calls trainer.train().
|
|
|
|
[100%]
|
|
|
|
=========================================== 21 passed in 2.19s ===========================================
|
|
```
|
|
|
|
## 3. Декомпозиция вызовов в `trainer_new.py`
|
|
|
|
**Теперь вызовы выглядят такой цепочкой:**
|
|
```python
|
|
def train(self) -> None:
|
|
"""Full pipeline: setup → build → train → evaluate → cleanup."""
|
|
self._validate_backbone()
|
|
clear_vram()
|
|
set_seed(self.pipeline_cfg.seed)
|
|
self._setup_output_dir()
|
|
self._setup_tracker()
|
|
self._build_model()
|
|
self._configure_gradient_checkpointing()
|
|
self._log_model_summary()
|
|
self._build_loss()
|
|
self._build_neg_bank()
|
|
self._build_data_loaders()
|
|
self._build_optimizer_and_scheduler()
|
|
self._restore_from_resume()
|
|
self._setup_profiler()
|
|
|
|
try:
|
|
self._train_loop()
|
|
self._final_evaluation()
|
|
finally:
|
|
self._cleanup()
|
|
```
|
|
|
|
**Лог:**
|
|
|
|
### Тест вызовов до блока `try: self._train_loop()`
|
|
✅ Загрузка весов
|
|
✅ Сборка пайплайна
|
|
✅ Загрузка моделей
|
|
✅ Загрузка данных
|
|
|
|
```bash
|
|
2026-05-07 09:40:53 caption_test.trainer INFO ⚙️ Validate backbone
|
|
2026-05-07 09:40:53 caption_test.trainer INFO ⚙️ Setup out dir
|
|
2026-05-07 09:40:53 caption_test.trainer INFO ⚙️ Setup tracker...
|
|
2026-05-07 09:40:53 caption_test.trackers WARNING tensorboard not installed, skipping TB tracking
|
|
2026-05-07 09:41:24 caption_test.trainer INFO ⚙️ Build loss...
|
|
2026-05-07 09:41:52 caption_test.trainer INFO Building model — with text (L1/L2/L3), shared DINOv3 WEB
|
|
2026-05-07 09:42:24 caption_test.trainer INFO ⚙️ Build DINOv3 model...
|
|
2026-05-07 09:50:48 caption_test.model INFO 🧊 Loading DINOv3 from dinov3-vitl16-pretrain-lvd1689m.pth
|
|
2026-05-07 09:50:48 caption_test.model INFO 🧊 Loading DINOv3 from dinov3-vitl16-pretrain-lvd1689m.pth
|
|
2026-05-07 09:54:48 caption_test.model INFO 🧊 DINOv3 loaded: 303,129,600 params
|
|
2026-05-07 09:54:48 caption_test.model INFO 🧊 DINOv3 loaded: 303,129,600 params
|
|
2026-05-07 09:55:03 caption_test.adapters INFO 🔧 MONA injected: 24 adapters (blocks 12-23 of 24), 3,502,080 trainable params (bottleneck=64)
|
|
2026-05-07 09:55:03 caption_test.adapters INFO 🔧 MONA injected: 24 adapters (blocks 12-23 of 24), 3,502,080 trainable params (bottleneck=64)
|
|
2026-05-07 09:55:05 caption_test.model INFO Shared encoder mode: single DINOv3 WEB for drone + satellite
|
|
2026-05-07 09:55:05 caption_test.model INFO Shared encoder mode: single DINOv3 WEB for drone + satellite
|
|
2026-05-07 09:55:38 caption_test.dgtrs INFO 📝 Loading DGTRS-CLIP text encoder from DGTRS-CLIP-ViT-L-14.pt
|
|
2026-05-07 09:55:38 caption_test.dgtrs INFO 📝 Loading DGTRS-CLIP text encoder from DGTRS-CLIP-ViT-L-14.pt
|
|
2026-05-07 09:55:39 caption_test.dgtrs INFO 📝 DGTRS text encoder loaded: 123,972,096 params, context=248 tokens
|
|
2026-05-07 09:55:39 caption_test.dgtrs INFO 📝 DGTRS text encoder loaded: 123,972,096 params, context=248 tokens
|
|
2026-05-07 10:03:01 caption_test.adapters INFO 🔧 LoRA injected: 12 blocks, rank=4, 147,456 trainable params
|
|
2026-05-07 10:03:01 caption_test.adapters INFO 🔧 LoRA injected: 12 blocks, rank=4, 147,456 trainable params
|
|
2026-05-07 10:04:01 caption_test.trainer INFO embed_dim=1024
|
|
2026-05-07 10:04:22 caption_test.trainer INFO ⚙️ Configure gradient checkpointing...
|
|
2026-05-07 10:04:42 caption_test.trainer INFO ✅ Gradient checkpointing enabled (DINOv3 + DGTRS)
|
|
2026-05-07 10:04:51 caption_test.trainer INFO trainable=7,059,458 (1.63%) total=434,161,154
|
|
2026-05-07 10:04:51 caption_test.profiler INFO torchinfo not installed, using basic parameter count
|
|
2026-05-07 10:04:51 caption_test.profiler INFO Model summary:
|
|
Total parameters: 434,161,154
|
|
Trainable parameters: 7,059,458 (1.63%)
|
|
[trainable] image_encoder.layer.12.mona_attn.gamma: [1024] (1,024)
|
|
... (Вывод списка параметров)
|
|
...
|
|
2026-05-07 10:05:00 caption_test.trainer INFO ⚙️ Build loss...
|
|
2026-05-07 10:06:06 caption_test.trainer INFO Loss: SymmetricInfoNCE Temperature: learnable (init=0.070) q2g=0.60 g2q=0.40
|
|
2026-05-07 10:06:07 caption_test.trainer INFO ⚙️ Build negative bank...
|
|
2026-05-07 10:06:13 caption_test.trainer INFO ⚙️ Build dataloaders...
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:13 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:19 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:19 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO ✅ Loaded 24891 pairs from meta/train_80.json
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO ✅ Loaded 24891 pairs from meta/train_80.json
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:20 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO ✅ Loaded 6252 pairs from meta/test_20.json
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO ✅ Loaded 6252 pairs from meta/test_20.json
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 🔻 Filter loaded: 10905 excluded images
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📐 Altitude index: 33708 drones (from 2 CSV)
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:22 caption_test.gtauav_dataset INFO 📚 Loading caption index from /home/servml/Документы/datasets/GTA-UAV-LR-captions
|
|
2026-05-07 10:06:24 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:24 caption_test.gtauav_dataset INFO 📚 Caption index: 39957 entries
|
|
2026-05-07 10:06:25 caption_test.gtauav_dataset INFO ✅ Loaded 24891 pairs from meta/train_80.json
|
|
2026-05-07 10:06:25 caption_test.gtauav_dataset INFO ✅ Loaded 24891 pairs from meta/train_80.json
|
|
2026-05-07 10:06:25 caption_test.trainer INFO Sampler: MutuallyExclusive — no false negatives within a batch
|
|
2026-05-07 10:06:25 caption_test.trainer INFO train=24891 test=6252 batch=8 accum=8 effective_batch=64
|
|
2026-05-07 10:06:46 caption_test.trainer INFO ⚙️ Build optimizer & scheduler...
|
|
2026-05-07 10:06:46 caption_test.trainer INFO Optimizer: AdamW LR: proj=1e-04 text=1e-05 warmup=2 epochs
|
|
2026-05-07 10:07:33 caption_test.trainer INFO ⚙️ Restore from resume...
|
|
2026-05-07 10:07:37 caption_test.trainer INFO ⚙️ Setup profiler...
|
|
```
|
|
|
|
`src/main.py`:
|
|
- Читает имя пресета из `sys.argv[1]` (один позиционный аргумент)
|
|
- Резолвит корень проекта через `get_proj_dir()` (поиск по маркерам `pyproject.toml`/`.git`/`in/`)
|
|
- Формирует `path2cfg = f"{proj_dir}in/config_files/"` буквально по REQUIREMENTS_GIN_STYLE.md §5
|
|
- Вызывает `load_all_configs(path2cfg, preset_name)` — двухпроходная загрузка из `_common`-файлов и пресет-директории
|
|
- Передаёт 6 объектов конфига в `train(...)` именованными аргументами
|
|
|
|
|