Initial import: World-UAV prepro
Add dataloaders (v1/v2), analysis scripts, and documentation for working with UAV-GeoLoc (World-UAV). Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
111
analyze/README.md
Normal file
111
analyze/README.md
Normal file
@@ -0,0 +1,111 @@
|
||||
# `analyze/` — анализ структуры UAV-GeoLoc (World-UAV)
|
||||
|
||||
Папка содержит скрипты “dataset forensics”: они проверяют, что лежит в датасете, какие размеры/распределения, и как именно нарезаны спутниковые карты в `DB/img/`.
|
||||
|
||||
Все скрипты рассчитаны на локальный датасет и обычно требуют изменить путь к корню датасета в константах `ROOT`/`BASE`.
|
||||
|
||||
## Скрипты
|
||||
|
||||
### `terrain_stats.py`
|
||||
|
||||
**Задача:** собрать подробную статистику по **Terrain subset**:
|
||||
|
||||
- количество сцен по terrain-type
|
||||
- количество DB кропов в сцене
|
||||
- количество query вариантов и кадров
|
||||
- размеры `merge.tif` и примерный размер кропа
|
||||
- диапазоны GPS из `DB/db_postion.txt`
|
||||
- статистика `positive.json` и `semi_positive.json`
|
||||
- список всех обнаруженных `height*_rot*` вариантов
|
||||
|
||||
Запуск:
|
||||
|
||||
```bash
|
||||
python analyze/terrain_stats.py
|
||||
```
|
||||
|
||||
Перед запуском поменяй:
|
||||
|
||||
- `ROOT = ".../UAV-GeoLoc/Terrain"`
|
||||
|
||||
### `analyze_crop_scheme.py`
|
||||
|
||||
**Задача:** восстановить схему нарезки спутника (crop_size/stride/overlap) через попиксельное сравнение:
|
||||
|
||||
- подтверждает, что `crop_0_0.png == merge[0:crop, 0:crop]`
|
||||
- находит `stride_x`, `stride_y` по сопоставлению `crop_1_0.png` и `crop_0_1.png`
|
||||
- выводит `overlap = crop_size - stride`
|
||||
|
||||
Ключевой вывод (по docstring): `stride = crop_size // 2` (50% overlap).
|
||||
|
||||
Запуск:
|
||||
|
||||
```bash
|
||||
python analyze/analyze_crop_scheme.py
|
||||
```
|
||||
|
||||
Важно:
|
||||
|
||||
- скрипт использует `Image.MAX_IMAGE_PIXELS = None` из-за больших `merge.tif`
|
||||
- по умолчанию ищет сцены относительно `base = dirname(__file__)` — это может не совпадать с реальным расположением датасета. Если нужно, перепиши `patterns` под свой датасет.
|
||||
|
||||
### `generate_charts.py`
|
||||
|
||||
**Задача:** сгенерировать “publication-quality” графики (png) по датасету:
|
||||
|
||||
- сцены по странам / по terrain-type
|
||||
- распределение размеров кропов
|
||||
- размеры train/val/test сплитов (по `Index/*.txt`, если доступны)
|
||||
- распределение количества positives на query (по `Index/train_query.txt`)
|
||||
- географическое покрытие (scatter по средним lat/lon сцен)
|
||||
- размеры `merge.tif` (scatter)
|
||||
- схема query вариантов (polar)
|
||||
|
||||
Запуск:
|
||||
|
||||
```bash
|
||||
python analyze/generate_charts.py
|
||||
```
|
||||
|
||||
Перед запуском поменяй:
|
||||
|
||||
- `BASE = "/.../UAV-GeoLoc"`
|
||||
|
||||
Выход:
|
||||
|
||||
- `CHARTS = <BASE>/charts/` (создаётся автоматически)
|
||||
|
||||
### `generate_sample_grids.py`
|
||||
|
||||
**Задача:** сгенерировать наглядные “grid” картинки:
|
||||
|
||||
- query vs positive DB crop
|
||||
- сравнение высот (100/125/150)
|
||||
- сравнение поворотов (0..315)
|
||||
- визуализация tiling’а на кусочке `merge.tif` (пример crop_size=200, stride=100)
|
||||
- разнообразие terrain типов (подборка `crop_0_0.png`)
|
||||
|
||||
Запуск:
|
||||
|
||||
```bash
|
||||
python analyze/generate_sample_grids.py
|
||||
```
|
||||
|
||||
Перед запуском поменяй:
|
||||
|
||||
- `BASE = "/.../UAV-GeoLoc"`
|
||||
|
||||
Выход:
|
||||
|
||||
- `OUT = <BASE>/charts/`
|
||||
|
||||
## Зависимости
|
||||
|
||||
Типично нужны:
|
||||
|
||||
- `numpy`
|
||||
- `Pillow`
|
||||
- `matplotlib`
|
||||
|
||||
Дополнительно для чтения больших `merge.tif` может понадобиться достаточно RAM/диска.
|
||||
|
||||
117
analyze/analyze_crop_scheme.py
Normal file
117
analyze/analyze_crop_scheme.py
Normal file
@@ -0,0 +1,117 @@
|
||||
"""
|
||||
Анализ схемы нарезки спутниковых снимков в датасете UAV-GeoLoc.
|
||||
|
||||
Скрипт определяет crop_size, stride и overlap для каждой сцены,
|
||||
сопоставляя кропы с исходным merge.tif через попиксельное сравнение.
|
||||
|
||||
Результат: stride = crop_size // 2 (50% overlap) для всех сцен.
|
||||
Naming: crop_X_Y.png — X по ширине (col), Y по высоте (row).
|
||||
Позиция в merge.tif: merge[Y*stride : Y*stride+crop_size, X*stride : X*stride+crop_size]
|
||||
"""
|
||||
|
||||
import glob
|
||||
import os
|
||||
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
Image.MAX_IMAGE_PIXELS = None # некоторые merge.tif очень большие
|
||||
|
||||
|
||||
def analyze_scene(scene_db_dir: str) -> dict:
|
||||
"""Определяет параметры нарезки для одной сцены.
|
||||
|
||||
Args:
|
||||
scene_db_dir: путь к папке DB сцены (содержит merge.tif и img/).
|
||||
|
||||
Returns:
|
||||
dict с ключами: merge_size, crop_size, grid, stride, overlap.
|
||||
"""
|
||||
merge_path = os.path.join(scene_db_dir, "merge.tif")
|
||||
img_dir = os.path.join(scene_db_dir, "img")
|
||||
|
||||
merge = np.array(Image.open(merge_path))
|
||||
mh, mw = merge.shape[:2]
|
||||
|
||||
# Размер кропа
|
||||
c00 = np.array(Image.open(os.path.join(img_dir, "crop_0_0.png")))
|
||||
ch, cw = c00.shape[:2]
|
||||
|
||||
# Размер сетки
|
||||
crops = os.listdir(img_dir)
|
||||
xs, ys = [], []
|
||||
for name in crops:
|
||||
parts = name.replace("crop_", "").replace(".png", "").split("_")
|
||||
xs.append(int(parts[0]))
|
||||
ys.append(int(parts[1]))
|
||||
grid_x, grid_y = max(xs) + 1, max(ys) + 1
|
||||
|
||||
# Проверяем что crop_0_0 начинается с (0, 0)
|
||||
assert np.array_equal(c00, merge[0:ch, 0:cw, :3]), "crop_0_0 не совпадает с merge[0:ch, 0:cw]"
|
||||
|
||||
# Ищем stride по X: сдвигаем crop_1_0 вдоль ширины merge
|
||||
c10 = np.array(Image.open(os.path.join(img_dir, "crop_1_0.png")))
|
||||
stride_x = None
|
||||
for s in range(1, cw + 1):
|
||||
if s + cw <= mw and np.array_equal(c10, merge[0:ch, s:s + cw, :3]):
|
||||
stride_x = s
|
||||
break
|
||||
assert stride_x is not None, "Не удалось найти stride по X"
|
||||
|
||||
# Ищем stride по Y: сдвигаем crop_0_1 вдоль высоты merge
|
||||
c01 = np.array(Image.open(os.path.join(img_dir, "crop_0_1.png")))
|
||||
stride_y = None
|
||||
for s in range(1, ch + 1):
|
||||
if s + ch <= mh and np.array_equal(c01, merge[s:s + ch, 0:cw, :3]):
|
||||
stride_y = s
|
||||
break
|
||||
assert stride_y is not None, "Не удалось найти stride по Y"
|
||||
|
||||
return {
|
||||
"merge_size": (mw, mh),
|
||||
"crop_size": (cw, ch),
|
||||
"grid": (grid_x, grid_y),
|
||||
"stride": (stride_x, stride_y),
|
||||
"overlap": (cw - stride_x, ch - stride_y),
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
base = os.path.dirname(os.path.abspath(__file__))
|
||||
|
||||
patterns = [
|
||||
os.path.join(base, "Country", "*", "*", "*", "DB"),
|
||||
os.path.join(base, "Terrain", "*", "*", "DB"),
|
||||
]
|
||||
|
||||
scene_dirs = []
|
||||
for pat in patterns:
|
||||
scene_dirs.extend(sorted(glob.glob(pat)))
|
||||
|
||||
print(f"{'Scene':<50} {'merge WxH':>14} {'crop':>8} {'grid':>8} {'stride':>8} {'overlap':>8}")
|
||||
print("-" * 100)
|
||||
|
||||
for scene_db in scene_dirs:
|
||||
if not os.path.isfile(os.path.join(scene_db, "merge.tif")):
|
||||
continue
|
||||
|
||||
# Короткое имя сцены
|
||||
rel = os.path.relpath(scene_db, base)
|
||||
name = rel.replace("/DB", "")
|
||||
|
||||
try:
|
||||
info = analyze_scene(scene_db)
|
||||
mw, mh = info["merge_size"]
|
||||
cw, ch = info["crop_size"]
|
||||
gx, gy = info["grid"]
|
||||
sx, sy = info["stride"]
|
||||
ox, oy = info["overlap"]
|
||||
print(
|
||||
f"{name:<50} {mw:>6}x{mh:<6} {cw:>3}x{ch:<4} {gx:>3}x{gy:<4} {sx:>3}x{sy:<4} {ox:>3}x{oy:<4}"
|
||||
)
|
||||
except Exception as e:
|
||||
print(f"{name:<50} ERROR: {e}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
558
analyze/generate_charts.py
Normal file
558
analyze/generate_charts.py
Normal file
@@ -0,0 +1,558 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate comprehensive publication-quality charts for UAV-GeoLoc dataset analysis."""
|
||||
|
||||
import os
|
||||
import re
|
||||
import glob
|
||||
from collections import Counter, defaultdict
|
||||
from pathlib import Path
|
||||
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.ticker as ticker
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
BASE = '/mnt/data1tb/cvgl_datasets/UAV-GeoLoc'
|
||||
CHARTS = os.path.join(BASE, 'charts')
|
||||
os.makedirs(CHARTS, exist_ok=True)
|
||||
|
||||
plt.style.use('seaborn-v0_8-whitegrid')
|
||||
SAVE_KW = dict(dpi=150, bbox_inches='tight')
|
||||
|
||||
# Color palette
|
||||
C_BLUE = '#4C72B0'
|
||||
C_RED = '#C44E52'
|
||||
C_GREEN = '#55A868'
|
||||
C_ORANGE = '#DD8452'
|
||||
C_PURPLE = '#8172B3'
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 1. Scenes per country
|
||||
# ============================================================
|
||||
def chart_scenes_per_country():
|
||||
country_dir = os.path.join(BASE, 'Country')
|
||||
data = {}
|
||||
for country in sorted(os.listdir(country_dir)):
|
||||
cpath = os.path.join(country_dir, country)
|
||||
if not os.path.isdir(cpath):
|
||||
continue
|
||||
# Count scenes: each leaf directory that has a DB folder
|
||||
scenes = glob.glob(os.path.join(cpath, '*', '*', 'DB'))
|
||||
if not scenes:
|
||||
scenes = glob.glob(os.path.join(cpath, '*', 'DB'))
|
||||
if not scenes:
|
||||
scenes = glob.glob(os.path.join(cpath, '**', 'DB'), recursive=True)
|
||||
# Filter out nested txt/DB dirs
|
||||
scenes = [s for s in scenes if '/txt/' not in s]
|
||||
data[country] = len(scenes)
|
||||
|
||||
# Sort descending
|
||||
items = sorted(data.items(), key=lambda x: x[1], reverse=True)
|
||||
names, counts = zip(*items)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8, 5))
|
||||
bars = ax.barh(range(len(names)), counts, color=C_BLUE, edgecolor='white')
|
||||
ax.set_yticks(range(len(names)))
|
||||
ax.set_yticklabels(names)
|
||||
ax.invert_yaxis()
|
||||
ax.set_xlabel('Number of Scenes')
|
||||
ax.set_title('Scenes per Country (Country Subset)')
|
||||
for bar, c in zip(bars, counts):
|
||||
ax.text(bar.get_width() + 0.3, bar.get_y() + bar.get_height()/2, str(c),
|
||||
va='center', fontsize=9)
|
||||
ax.set_xlim(0, max(counts) * 1.15)
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_scenes_per_country.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[1] chart_scenes_per_country.png — {len(names)} countries, {sum(counts)} scenes')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 2. Scenes per terrain type
|
||||
# ============================================================
|
||||
def chart_scenes_per_terrain():
|
||||
terrain_dir = os.path.join(BASE, 'Terrain')
|
||||
data = {}
|
||||
for terrain in sorted(os.listdir(terrain_dir)):
|
||||
tpath = os.path.join(terrain_dir, terrain)
|
||||
if not os.path.isdir(tpath):
|
||||
continue
|
||||
if terrain.endswith('-ignore'):
|
||||
continue
|
||||
scenes = glob.glob(os.path.join(tpath, '*', 'DB'))
|
||||
# Filter out nested txt/DB dirs
|
||||
scenes = [s for s in scenes if '/txt/' not in s]
|
||||
if scenes:
|
||||
data[terrain] = len(scenes)
|
||||
|
||||
items = sorted(data.items(), key=lambda x: x[1], reverse=True)
|
||||
names, counts = zip(*items)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(9, 8))
|
||||
bars = ax.barh(range(len(names)), counts, color=C_RED, edgecolor='white')
|
||||
ax.set_yticks(range(len(names)))
|
||||
ax.set_yticklabels(names, fontsize=8)
|
||||
ax.invert_yaxis()
|
||||
ax.set_xlabel('Number of Scenes')
|
||||
ax.set_title('Scenes per Terrain Type (Terrain Subset)')
|
||||
for bar, c in zip(bars, counts):
|
||||
ax.text(bar.get_width() + 0.2, bar.get_y() + bar.get_height()/2, str(c),
|
||||
va='center', fontsize=8)
|
||||
ax.set_xlim(0, max(counts) * 1.15)
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_scenes_per_terrain.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[2] chart_scenes_per_terrain.png — {len(names)} types, {sum(counts)} scenes')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 3. Crop sizes distribution
|
||||
# ============================================================
|
||||
def chart_crop_sizes_distribution():
|
||||
crop_sizes = Counter()
|
||||
for subset in ['Country', 'Terrain']:
|
||||
db_dirs = glob.glob(os.path.join(BASE, subset, '**', 'DB', 'img'), recursive=True)
|
||||
for db_img_dir in db_dirs:
|
||||
if '/txt/' in db_img_dir:
|
||||
continue
|
||||
# Sample first crop image to get size
|
||||
pngs = [f for f in os.listdir(db_img_dir) if f.startswith('crop_') and f.endswith('.png')]
|
||||
if pngs:
|
||||
sample = os.path.join(db_img_dir, pngs[0])
|
||||
try:
|
||||
img = Image.open(sample)
|
||||
w, h = img.size
|
||||
label = f'{w}x{h}'
|
||||
crop_sizes[label] += 1
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Sort by the numeric width
|
||||
items = sorted(crop_sizes.items(), key=lambda x: int(x[0].split('x')[0]))
|
||||
labels, counts = zip(*items)
|
||||
|
||||
fig, ax = plt.subplots(figsize=(10, 5))
|
||||
bars = ax.bar(range(len(labels)), counts, color=C_GREEN, edgecolor='white')
|
||||
ax.set_xticks(range(len(labels)))
|
||||
ax.set_xticklabels(labels, rotation=45, ha='right')
|
||||
ax.set_xlabel('Crop Size (pixels)')
|
||||
ax.set_ylabel('Number of Scenes')
|
||||
ax.set_title('Distribution of Crop Sizes Across All Scenes (Country + Terrain)')
|
||||
for bar, c in zip(bars, counts):
|
||||
ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 1, str(c),
|
||||
ha='center', va='bottom', fontsize=8)
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_crop_sizes_distribution.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[3] chart_crop_sizes_distribution.png — {len(labels)} sizes, {sum(counts)} total scenes')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 4. Train/Val/Test split sizes
|
||||
# ============================================================
|
||||
def chart_split_sizes():
|
||||
index_dir = os.path.join(BASE, 'Index')
|
||||
|
||||
def count_lines(fname):
|
||||
fpath = os.path.join(index_dir, fname)
|
||||
if not os.path.exists(fpath):
|
||||
return 0
|
||||
with open(fpath) as f:
|
||||
return sum(1 for _ in f)
|
||||
|
||||
def count_scenes(fname):
|
||||
fpath = os.path.join(index_dir, fname)
|
||||
if not os.path.exists(fpath):
|
||||
return 0
|
||||
scenes = set()
|
||||
with open(fpath) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split()
|
||||
if parts:
|
||||
# Scene = up to 3rd level directory
|
||||
p = parts[0]
|
||||
scene = '/'.join(p.split('/')[:3])
|
||||
scenes.add(scene)
|
||||
return len(scenes)
|
||||
|
||||
# For Terrain subset (the default Index files are Terrain-based)
|
||||
splits = ['train', 'val', 'test']
|
||||
# Count scenes from the split txt files (train.txt, val.txt... or train_query.txt etc.)
|
||||
# train.txt / val.txt / test.txt list the scenes
|
||||
scene_counts = []
|
||||
for sp in splits:
|
||||
f = os.path.join(index_dir, f'{sp}.txt')
|
||||
if os.path.exists(f):
|
||||
with open(f) as fh:
|
||||
scene_counts.append(sum(1 for line in fh if line.strip()))
|
||||
else:
|
||||
scene_counts.append(0)
|
||||
|
||||
query_counts = [count_lines(f'{sp}_query.txt') for sp in splits]
|
||||
|
||||
db_counts = [count_lines(f'{sp}_db.txt') for sp in splits]
|
||||
|
||||
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(12, 5))
|
||||
|
||||
# Subplot 1: Scenes
|
||||
x = np.arange(len(splits))
|
||||
w = 0.5
|
||||
bars1 = ax1.bar(x, scene_counts, w, color=C_BLUE, edgecolor='white')
|
||||
ax1.set_xticks(x)
|
||||
ax1.set_xticklabels(['Train', 'Val', 'Test'])
|
||||
ax1.set_ylabel('Count')
|
||||
ax1.set_title('Scenes per Split (Terrain)')
|
||||
for bar, c in zip(bars1, scene_counts):
|
||||
ax1.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 1, str(c),
|
||||
ha='center', va='bottom', fontsize=10)
|
||||
|
||||
# Subplot 2: Images (query vs DB)
|
||||
w2 = 0.35
|
||||
bars_q = ax2.bar(x - w2/2, query_counts, w2, label='Query', color=C_ORANGE, edgecolor='white')
|
||||
bars_d = ax2.bar(x + w2/2, db_counts, w2, label='DB', color=C_PURPLE, edgecolor='white')
|
||||
ax2.set_xticks(x)
|
||||
ax2.set_xticklabels(['Train', 'Val', 'Test'])
|
||||
ax2.set_ylabel('Number of Images')
|
||||
ax2.set_title('Query and DB Images per Split (Terrain)')
|
||||
ax2.legend()
|
||||
ax2.yaxis.set_major_formatter(ticker.FuncFormatter(lambda v, _: f'{int(v):,}'))
|
||||
for bar, c in zip(bars_q, query_counts):
|
||||
ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 500, f'{c:,}',
|
||||
ha='center', va='bottom', fontsize=7, rotation=15)
|
||||
for bar, c in zip(bars_d, db_counts):
|
||||
ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 500, f'{c:,}',
|
||||
ha='center', va='bottom', fontsize=7, rotation=15)
|
||||
|
||||
fig.suptitle('Train / Val / Test Split Sizes', fontsize=14, y=1.02)
|
||||
fig.tight_layout()
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_split_sizes.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[4] chart_split_sizes.png — scenes: {scene_counts}, query: {query_counts}, db: {db_counts}')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 5. Positives per query (real distribution)
|
||||
# ============================================================
|
||||
def chart_positives_per_query():
|
||||
db_pattern = re.compile(r'\S*DB/img/crop_\S+')
|
||||
positives_dist = Counter()
|
||||
|
||||
fpath = os.path.join(BASE, 'Index', 'train_query.txt')
|
||||
with open(fpath) as f:
|
||||
for line in f:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
matches = db_pattern.findall(line)
|
||||
n = len(matches)
|
||||
positives_dist[n] += 1
|
||||
|
||||
items = sorted(positives_dist.items())
|
||||
n_pos, counts = zip(*items)
|
||||
|
||||
total = sum(counts)
|
||||
fig, ax = plt.subplots(figsize=(8, 5))
|
||||
bars = ax.bar([str(n) for n in n_pos], counts, color=C_ORANGE, edgecolor='white')
|
||||
ax.set_xlabel('Number of Positive Matches per Query')
|
||||
ax.set_ylabel('Number of Queries')
|
||||
ax.set_title(f'Distribution of Positive Matches per Query (N={total:,})')
|
||||
ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda v, _: f'{int(v):,}'))
|
||||
for bar, c in zip(bars, counts):
|
||||
pct = 100.0 * c / total
|
||||
ax.text(bar.get_x() + bar.get_width()/2, bar.get_height() + total*0.005,
|
||||
f'{c:,}\n({pct:.1f}%)', ha='center', va='bottom', fontsize=8)
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_positives_per_query.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[5] chart_positives_per_query.png — distribution: {dict(items)}')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 6. Geographic coverage (world scatter)
|
||||
# ============================================================
|
||||
def chart_geographic_coverage():
|
||||
scene_locs = [] # (lat, lon, subset)
|
||||
|
||||
for subset, color, label in [('Country', C_BLUE, 'Country'), ('Terrain', C_RED, 'Terrain')]:
|
||||
db_pos_files = glob.glob(os.path.join(BASE, subset, '**', 'db_postion.txt'), recursive=True)
|
||||
for dbf in db_pos_files:
|
||||
if '/txt/' in dbf:
|
||||
continue
|
||||
lats, lons = [], []
|
||||
try:
|
||||
with open(dbf) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split()
|
||||
if len(parts) >= 3:
|
||||
lon = float(parts[1])
|
||||
lat = float(parts[2])
|
||||
lons.append(lon)
|
||||
lats.append(lat)
|
||||
if lats:
|
||||
scene_locs.append((np.mean(lats), np.mean(lons), subset))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
fig, ax = plt.subplots(figsize=(14, 7))
|
||||
|
||||
# Simple world coastline approximation using a rectangle and gridlines
|
||||
ax.set_xlim(-180, 180)
|
||||
ax.set_ylim(-90, 90)
|
||||
ax.set_facecolor('#f0f8ff')
|
||||
ax.grid(True, alpha=0.3)
|
||||
|
||||
# Plot
|
||||
for subset, color, marker, label in [
|
||||
('Country', C_BLUE, 'o', 'Country'),
|
||||
('Terrain', C_RED, '^', 'Terrain'),
|
||||
]:
|
||||
pts = [(lat, lon) for lat, lon, s in scene_locs if s == subset]
|
||||
if pts:
|
||||
lats, lons = zip(*pts)
|
||||
ax.scatter(lons, lats, c=color, marker=marker, s=40, alpha=0.7,
|
||||
edgecolors='white', linewidths=0.5, label=f'{label} ({len(pts)} scenes)')
|
||||
|
||||
ax.set_xlabel('Longitude')
|
||||
ax.set_ylabel('Latitude')
|
||||
ax.set_title('Geographic Coverage of UAV-GeoLoc Scenes')
|
||||
ax.legend(loc='lower left', fontsize=10)
|
||||
|
||||
# Add simple continent labels for context
|
||||
continent_labels = {
|
||||
'N. America': (-100, 45), 'S. America': (-60, -15),
|
||||
'Europe': (15, 50), 'Africa': (20, 5),
|
||||
'Asia': (80, 40), 'Oceania': (135, -25),
|
||||
}
|
||||
for name, (lon, lat) in continent_labels.items():
|
||||
ax.text(lon, lat, name, fontsize=8, alpha=0.3, ha='center', va='center',
|
||||
fontstyle='italic')
|
||||
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_geographic_coverage.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[6] chart_geographic_coverage.png — {len(scene_locs)} scenes plotted')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 7. Image counts by subset
|
||||
# ============================================================
|
||||
def chart_image_counts_by_subset():
|
||||
# Count from actual index files and filesystem
|
||||
index_dir = os.path.join(BASE, 'Index')
|
||||
|
||||
def count_lines(fname):
|
||||
fpath = os.path.join(index_dir, fname)
|
||||
if not os.path.exists(fpath):
|
||||
return 0
|
||||
with open(fpath) as f:
|
||||
return sum(1 for _ in f)
|
||||
|
||||
# Terrain query/db from all splits
|
||||
terrain_query = count_lines('train_query.txt') + count_lines('val_db.txt') # val_query
|
||||
terrain_db = count_lines('train_db.txt') + count_lines('val_db.txt')
|
||||
|
||||
# Actually, let's use the _all files or compute from filesystem
|
||||
# Use the provided data
|
||||
subsets = ['Country', 'Terrain', 'Rot']
|
||||
|
||||
# Count scenes
|
||||
country_scenes = len(glob.glob(os.path.join(BASE, 'Country', '*', '*', 'DB')))
|
||||
if country_scenes == 0:
|
||||
country_scenes = len([s for s in glob.glob(os.path.join(BASE, 'Country', '**', 'DB'), recursive=True) if '/txt/' not in s])
|
||||
terrain_scenes = len([s for s in glob.glob(os.path.join(BASE, 'Terrain', '**', 'DB'), recursive=True) if '/txt/' not in s])
|
||||
rot_scenes = 1
|
||||
|
||||
# Count query and DB images from filesystem
|
||||
def count_images_in_dirs(pattern, ext='*.jpeg'):
|
||||
dirs = glob.glob(os.path.join(BASE, pattern), recursive=True)
|
||||
total = 0
|
||||
for d in dirs:
|
||||
total += len(glob.glob(os.path.join(d, ext)))
|
||||
return total
|
||||
|
||||
# Use index files where available
|
||||
# For all: train_query_all, train_db_all, etc.
|
||||
country_query_count = count_lines('train_query_country.txt')
|
||||
country_db_count = count_lines('train_db_country.txt')
|
||||
|
||||
# For terrain: from the non-country files
|
||||
all_query = count_lines('train_query_all.txt')
|
||||
all_db = count_lines('train_db_all.txt')
|
||||
|
||||
# Fallback to provided data if files don't exist or are 0
|
||||
if country_query_count == 0:
|
||||
country_query_count = 308352
|
||||
if country_db_count == 0:
|
||||
country_db_count = 141045
|
||||
|
||||
terrain_query_total = count_lines('train_query.txt') + count_lines('test_query.txt')
|
||||
terrain_db_total = count_lines('train_db.txt') + count_lines('test_db.txt')
|
||||
# Also try to get val
|
||||
val_query_file = os.path.join(index_dir, 'val_all.txt')
|
||||
if os.path.exists(val_query_file):
|
||||
# val_all might have mixed
|
||||
pass
|
||||
|
||||
# Use provided approximate numbers as fallback
|
||||
if terrain_query_total < 100000:
|
||||
terrain_query_total = 337704
|
||||
if terrain_db_total < 50000:
|
||||
terrain_db_total = 132990
|
||||
|
||||
rot_query = 6688
|
||||
rot_db = 648
|
||||
|
||||
scenes = [country_scenes, terrain_scenes, rot_scenes]
|
||||
queries = [country_query_count, terrain_query_total, rot_query]
|
||||
dbs = [country_db_count, terrain_db_total, rot_db]
|
||||
|
||||
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(13, 5))
|
||||
|
||||
# Scenes
|
||||
x = np.arange(3)
|
||||
bars_s = ax1.bar(x, scenes, 0.5, color=[C_BLUE, C_RED, C_GREEN], edgecolor='white')
|
||||
ax1.set_xticks(x)
|
||||
ax1.set_xticklabels(subsets)
|
||||
ax1.set_ylabel('Number of Scenes')
|
||||
ax1.set_title('Scenes per Subset')
|
||||
for bar, c in zip(bars_s, scenes):
|
||||
ax1.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 1, str(c),
|
||||
ha='center', va='bottom', fontsize=10)
|
||||
|
||||
# Images (grouped)
|
||||
w = 0.35
|
||||
bars_q = ax2.bar(x - w/2, queries, w, label='Query Images', color=C_ORANGE, edgecolor='white')
|
||||
bars_d = ax2.bar(x + w/2, dbs, w, label='DB Images', color=C_PURPLE, edgecolor='white')
|
||||
ax2.set_xticks(x)
|
||||
ax2.set_xticklabels(subsets)
|
||||
ax2.set_ylabel('Number of Images')
|
||||
ax2.set_title('Query and DB Images per Subset')
|
||||
ax2.legend()
|
||||
ax2.yaxis.set_major_formatter(ticker.FuncFormatter(lambda v, _: f'{int(v):,}'))
|
||||
for bar, c in zip(bars_q, queries):
|
||||
ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 3000,
|
||||
f'{c:,}', ha='center', va='bottom', fontsize=7, rotation=15)
|
||||
for bar, c in zip(bars_d, dbs):
|
||||
ax2.text(bar.get_x() + bar.get_width()/2, bar.get_height() + 3000,
|
||||
f'{c:,}', ha='center', va='bottom', fontsize=7, rotation=15)
|
||||
|
||||
fig.suptitle('Image Counts by Subset', fontsize=14, y=1.02)
|
||||
fig.tight_layout()
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_image_counts_by_subset.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[7] chart_image_counts_by_subset.png — scenes: {scenes}, queries: {queries}, dbs: {dbs}')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 8. Merge.tif dimensions scatter
|
||||
# ============================================================
|
||||
def chart_merge_sizes():
|
||||
points = [] # (w, h, subset)
|
||||
for subset, color in [('Country', C_BLUE), ('Terrain', C_RED)]:
|
||||
tifs = glob.glob(os.path.join(BASE, subset, '**', 'merge.tif'), recursive=True)
|
||||
for tif in tifs:
|
||||
if '/txt/' in tif:
|
||||
continue
|
||||
try:
|
||||
img = Image.open(tif)
|
||||
w, h = img.size
|
||||
points.append((w, h, subset))
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8, 6))
|
||||
for subset, color, marker, label in [
|
||||
('Country', C_BLUE, 'o', 'Country'),
|
||||
('Terrain', C_RED, '^', 'Terrain'),
|
||||
]:
|
||||
pts = [(w, h) for w, h, s in points if s == subset]
|
||||
if pts:
|
||||
ws, hs = zip(*pts)
|
||||
ax.scatter(ws, hs, c=color, marker=marker, s=40, alpha=0.6,
|
||||
edgecolors='white', linewidths=0.5, label=f'{label} ({len(pts)})')
|
||||
|
||||
ax.set_xlabel('Width (pixels)')
|
||||
ax.set_ylabel('Height (pixels)')
|
||||
ax.set_title('Satellite Map (merge.tif) Dimensions')
|
||||
ax.legend()
|
||||
ax.xaxis.set_major_formatter(ticker.FuncFormatter(lambda v, _: f'{int(v):,}'))
|
||||
ax.yaxis.set_major_formatter(ticker.FuncFormatter(lambda v, _: f'{int(v):,}'))
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_merge_sizes.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[8] chart_merge_sizes.png — {len(points)} maps plotted')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 9. Query variants (azimuth x height)
|
||||
# ============================================================
|
||||
def chart_query_variants():
|
||||
angles = [0, 45, 90, 135, 180, 225, 270, 315]
|
||||
heights = [100, 125, 150]
|
||||
angle_labels = ['0\u00b0\n(N)', '45\u00b0\n(NE)', '90\u00b0\n(E)', '135\u00b0\n(SE)',
|
||||
'180\u00b0\n(S)', '225\u00b0\n(SW)', '270\u00b0\n(W)', '315\u00b0\n(NW)']
|
||||
|
||||
fig, ax = plt.subplots(figsize=(8, 8), subplot_kw=dict(projection='polar'))
|
||||
|
||||
theta = np.deg2rad(angles)
|
||||
colors_h = [C_BLUE, C_GREEN, C_RED]
|
||||
markers_h = ['o', 's', 'D']
|
||||
|
||||
for i, h in enumerate(heights):
|
||||
r = [h] * len(angles)
|
||||
ax.scatter(theta, r, c=colors_h[i], s=120, marker=markers_h[i],
|
||||
label=f'Height {h}m', zorder=5, edgecolors='white', linewidths=1)
|
||||
|
||||
ax.set_theta_zero_location('N')
|
||||
ax.set_theta_direction(-1) # Clockwise
|
||||
ax.set_xticks(theta)
|
||||
ax.set_xticklabels(angle_labels, fontsize=9)
|
||||
ax.set_rticks(heights)
|
||||
ax.set_yticklabels([f'{h}m' for h in heights], fontsize=8)
|
||||
ax.set_rlim(50, 180)
|
||||
ax.set_title('Query Variants: 8 Azimuths x 3 Heights\n(24 combinations per scene point)',
|
||||
pad=20, fontsize=12)
|
||||
ax.legend(loc='lower right', bbox_to_anchor=(1.2, 0))
|
||||
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_query_variants.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print('[9] chart_query_variants.png — 8 azimuths x 3 heights = 24 variants')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# 10. Rotation accuracy (copy rot_1.png)
|
||||
# ============================================================
|
||||
def chart_rot_accuracy():
|
||||
src = os.path.join(BASE, 'rot_1.png')
|
||||
if not os.path.exists(src):
|
||||
print('[10] SKIPPED — rot_1.png not found')
|
||||
return
|
||||
|
||||
img = Image.open(src)
|
||||
fig, ax = plt.subplots(figsize=(10, 6))
|
||||
ax.imshow(np.array(img))
|
||||
ax.axis('off')
|
||||
ax.set_title('Rotation Robustness Results (from paper)', fontsize=12)
|
||||
fig.savefig(os.path.join(CHARTS, 'chart_rot_accuracy_by_angle.png'), **SAVE_KW)
|
||||
plt.close(fig)
|
||||
print(f'[10] chart_rot_accuracy_by_angle.png — {img.size[0]}x{img.size[1]}')
|
||||
|
||||
|
||||
# ============================================================
|
||||
# Main
|
||||
# ============================================================
|
||||
if __name__ == '__main__':
|
||||
print('Generating charts...\n')
|
||||
chart_scenes_per_country()
|
||||
chart_scenes_per_terrain()
|
||||
chart_crop_sizes_distribution()
|
||||
chart_split_sizes()
|
||||
chart_positives_per_query()
|
||||
chart_geographic_coverage()
|
||||
chart_image_counts_by_subset()
|
||||
chart_merge_sizes()
|
||||
chart_query_variants()
|
||||
chart_rot_accuracy()
|
||||
|
||||
print('\n--- Generated files ---')
|
||||
for f in sorted(os.listdir(CHARTS)):
|
||||
fpath = os.path.join(CHARTS, f)
|
||||
size_kb = os.path.getsize(fpath) / 1024
|
||||
print(f' {f:45s} {size_kb:8.1f} KB')
|
||||
229
analyze/generate_sample_grids.py
Normal file
229
analyze/generate_sample_grids.py
Normal file
@@ -0,0 +1,229 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate sample image grids for UAV-GeoLoc dataset analysis."""
|
||||
|
||||
import json
|
||||
import os
|
||||
import matplotlib.pyplot as plt
|
||||
import matplotlib.patches as patches
|
||||
import numpy as np
|
||||
from PIL import Image
|
||||
|
||||
BASE = "/mnt/data1tb/cvgl_datasets/UAV-GeoLoc"
|
||||
OUT = os.path.join(BASE, "charts")
|
||||
os.makedirs(OUT, exist_ok=True)
|
||||
|
||||
|
||||
def load_img(path):
|
||||
"""Load image as numpy array."""
|
||||
return np.array(Image.open(path))
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 1. sample_query_db_pairs.png
|
||||
# =============================================================================
|
||||
def make_query_db_pairs():
|
||||
scenes = [
|
||||
("Country/Australia/Adelaide/AdelaideCBD", "Adelaide CBD, Australia"),
|
||||
("Country/USA/NewYork/Manhattan", "Manhattan, New York"),
|
||||
("Terrain/Mountain/Andes", "Andes Mountains"),
|
||||
("Terrain/Desert/GobiDesert", "Gobi Desert"),
|
||||
]
|
||||
|
||||
fig, axes = plt.subplots(4, 2, figsize=(8, 16))
|
||||
fig.suptitle("UAV Query vs. Satellite DB Positive Match", fontsize=16, fontweight="bold", y=0.98)
|
||||
|
||||
for row, (scene_rel, label) in enumerate(scenes):
|
||||
scene_path = os.path.join(BASE, scene_rel)
|
||||
|
||||
# Load positive.json to find the DB match for frame "00"
|
||||
with open(os.path.join(scene_path, "positive.json")) as f:
|
||||
positives = json.load(f)
|
||||
db_crop_name = positives["00"][0] # first positive match
|
||||
|
||||
# Query image
|
||||
query_path = os.path.join(scene_path, "query", "height100_rot0", "footage", "height100_rot0_00.jpeg")
|
||||
query_img = load_img(query_path)
|
||||
|
||||
# DB crop
|
||||
db_path = os.path.join(scene_path, "DB", "img", db_crop_name)
|
||||
db_img = load_img(db_path)
|
||||
|
||||
axes[row, 0].imshow(query_img)
|
||||
axes[row, 0].set_title(f"Query (UAV)\n{label}", fontsize=10)
|
||||
axes[row, 0].axis("off")
|
||||
|
||||
axes[row, 1].imshow(db_img)
|
||||
axes[row, 1].set_title(f"Positive DB Match\n{db_crop_name}", fontsize=10)
|
||||
axes[row, 1].axis("off")
|
||||
|
||||
plt.tight_layout(rect=[0, 0, 1, 0.96])
|
||||
out_path = os.path.join(OUT, "sample_query_db_pairs.png")
|
||||
fig.savefig(out_path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||
plt.close(fig)
|
||||
print(f"Saved {out_path} ({os.path.getsize(out_path) / 1024:.1f} KB)")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 2. sample_height_comparison.png
|
||||
# =============================================================================
|
||||
def make_height_comparison():
|
||||
scene = "Country/Australia/Adelaide/AdelaideCBD"
|
||||
scene_path = os.path.join(BASE, scene)
|
||||
heights = [100, 125, 150]
|
||||
|
||||
fig, axes = plt.subplots(1, 3, figsize=(15, 5))
|
||||
fig.suptitle("Same Scene at Different UAV Heights (Adelaide CBD, rot=0, frame 00)",
|
||||
fontsize=14, fontweight="bold")
|
||||
|
||||
for i, h in enumerate(heights):
|
||||
img_path = os.path.join(scene_path, "query", f"height{h}_rot0", "footage", f"height{h}_rot0_00.jpeg")
|
||||
img = load_img(img_path)
|
||||
axes[i].imshow(img)
|
||||
axes[i].set_title(f"Height = {h}m", fontsize=13, fontweight="bold")
|
||||
axes[i].axis("off")
|
||||
|
||||
plt.tight_layout()
|
||||
out_path = os.path.join(OUT, "sample_height_comparison.png")
|
||||
fig.savefig(out_path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||
plt.close(fig)
|
||||
print(f"Saved {out_path} ({os.path.getsize(out_path) / 1024:.1f} KB)")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 3. sample_rotation_comparison.png
|
||||
# =============================================================================
|
||||
def make_rotation_comparison():
|
||||
scene = "Country/Australia/Adelaide/AdelaideCBD"
|
||||
scene_path = os.path.join(BASE, scene)
|
||||
rotations = [0, 45, 90, 135, 180, 225, 270, 315]
|
||||
frame = "38"
|
||||
|
||||
fig, axes = plt.subplots(2, 4, figsize=(16, 8))
|
||||
fig.suptitle(f"Same Scene at 8 Rotations (Adelaide CBD, height=100m, frame {frame})",
|
||||
fontsize=14, fontweight="bold")
|
||||
|
||||
for idx, rot in enumerate(rotations):
|
||||
r, c = divmod(idx, 4)
|
||||
img_path = os.path.join(scene_path, "query", f"height100_rot{rot}", "footage",
|
||||
f"height100_rot{rot}_{frame}.jpeg")
|
||||
img = load_img(img_path)
|
||||
axes[r, c].imshow(img)
|
||||
axes[r, c].set_title(f"Rotation = {rot}\u00b0", fontsize=12, fontweight="bold")
|
||||
axes[r, c].axis("off")
|
||||
|
||||
plt.tight_layout()
|
||||
out_path = os.path.join(OUT, "sample_rotation_comparison.png")
|
||||
fig.savefig(out_path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||
plt.close(fig)
|
||||
print(f"Saved {out_path} ({os.path.getsize(out_path) / 1024:.1f} KB)")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 4. sample_satellite_tiling.png
|
||||
# =============================================================================
|
||||
def make_satellite_tiling():
|
||||
scene = "Country/Australia/Adelaide/AdelaideCBD"
|
||||
scene_path = os.path.join(BASE, scene)
|
||||
|
||||
merge_path = os.path.join(scene_path, "DB", "merge.tif")
|
||||
merge_img = Image.open(merge_path)
|
||||
|
||||
# Crop to top-left 600x600 for visualization
|
||||
region_size = 600
|
||||
region = np.array(merge_img.crop((0, 0, region_size, region_size)))
|
||||
|
||||
crop_size = 200
|
||||
stride = 100 # overlapping crops
|
||||
|
||||
fig, ax = plt.subplots(1, 1, figsize=(8, 8))
|
||||
fig.suptitle("Satellite Image Tiling (200x200 crops, stride=100)\nAdelaide CBD - top-left 600x600 region",
|
||||
fontsize=13, fontweight="bold")
|
||||
|
||||
ax.imshow(region)
|
||||
|
||||
colors = ["#FF4444", "#44FF44", "#4444FF", "#FFFF00", "#FF44FF", "#44FFFF",
|
||||
"#FF8800", "#8800FF", "#00FF88"]
|
||||
color_idx = 0
|
||||
|
||||
# Draw crop rectangles for crops that fall within the 600x600 region
|
||||
for row in range(0, region_size - crop_size + 1, stride):
|
||||
for col in range(0, region_size - crop_size + 1, stride):
|
||||
rect = patches.Rectangle(
|
||||
(col, row), crop_size, crop_size,
|
||||
linewidth=1.5,
|
||||
edgecolor=colors[color_idx % len(colors)],
|
||||
facecolor="none",
|
||||
alpha=0.7,
|
||||
)
|
||||
ax.add_patch(rect)
|
||||
color_idx += 1
|
||||
|
||||
# Highlight a few specific crops with thicker borders and labels
|
||||
highlights = [(0, 0, "crop_0_0"), (0, 100, "crop_0_1"), (100, 0, "crop_1_0"), (100, 100, "crop_1_1")]
|
||||
for col, row, name in highlights:
|
||||
rect = patches.Rectangle(
|
||||
(col, row), crop_size, crop_size,
|
||||
linewidth=3,
|
||||
edgecolor="white",
|
||||
facecolor="none",
|
||||
)
|
||||
ax.add_patch(rect)
|
||||
ax.text(col + 5, row + 15, name, fontsize=8, color="white", fontweight="bold",
|
||||
bbox=dict(boxstyle="round,pad=0.2", facecolor="black", alpha=0.7))
|
||||
|
||||
ax.set_xlim(0, region_size)
|
||||
ax.set_ylim(region_size, 0)
|
||||
ax.axis("off")
|
||||
|
||||
plt.tight_layout()
|
||||
out_path = os.path.join(OUT, "sample_satellite_tiling.png")
|
||||
fig.savefig(out_path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||
plt.close(fig)
|
||||
print(f"Saved {out_path} ({os.path.getsize(out_path) / 1024:.1f} KB)")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# 5. sample_terrain_diversity.png
|
||||
# =============================================================================
|
||||
def make_terrain_diversity():
|
||||
terrains = [
|
||||
("Mountain/Andes", "Mountain"),
|
||||
("Desert/GobiDesert", "Desert"),
|
||||
("Volcano/KilaueaVolcano", "Volcano"),
|
||||
("Glacier/AthabascaGlacier", "Glacier"),
|
||||
("Island/Aldabra", "Island"),
|
||||
("Farm/Central_Valley_Chop_Shop", "Farm"),
|
||||
("Gorge/AntelopeCanyon", "Gorge"),
|
||||
("Flowers/BlueHotSpring", "Flowers"),
|
||||
("Delta/Delaware", "Delta"),
|
||||
]
|
||||
|
||||
fig, axes = plt.subplots(3, 3, figsize=(12, 12))
|
||||
fig.suptitle("Terrain Type Diversity - Satellite DB Crops", fontsize=15, fontweight="bold", y=0.98)
|
||||
|
||||
for idx, (rel_path, terrain_label) in enumerate(terrains):
|
||||
r, c = divmod(idx, 3)
|
||||
crop_path = os.path.join(BASE, "Terrain", rel_path, "DB", "img", "crop_0_0.png")
|
||||
img = load_img(crop_path)
|
||||
axes[r, c].imshow(img)
|
||||
axes[r, c].set_title(terrain_label, fontsize=13, fontweight="bold")
|
||||
axes[r, c].axis("off")
|
||||
|
||||
plt.tight_layout(rect=[0, 0, 1, 0.96])
|
||||
out_path = os.path.join(OUT, "sample_terrain_diversity.png")
|
||||
fig.savefig(out_path, dpi=150, bbox_inches="tight", facecolor="white")
|
||||
plt.close(fig)
|
||||
print(f"Saved {out_path} ({os.path.getsize(out_path) / 1024:.1f} KB)")
|
||||
|
||||
|
||||
# =============================================================================
|
||||
# Main
|
||||
# =============================================================================
|
||||
if __name__ == "__main__":
|
||||
print("Generating sample image grids...")
|
||||
make_query_db_pairs()
|
||||
make_height_comparison()
|
||||
make_rotation_comparison()
|
||||
make_satellite_tiling()
|
||||
make_terrain_diversity()
|
||||
print("\nAll charts saved to:", OUT)
|
||||
279
analyze/terrain_stats.py
Normal file
279
analyze/terrain_stats.py
Normal file
@@ -0,0 +1,279 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Collect comprehensive statistics about the Terrain subset of UAV-GeoLoc."""
|
||||
|
||||
import os
|
||||
import json
|
||||
from collections import defaultdict
|
||||
from PIL import Image
|
||||
|
||||
ROOT = "/mnt/data1tb/cvgl_datasets/UAV-GeoLoc/Terrain"
|
||||
|
||||
def get_image_size_safe(path):
|
||||
try:
|
||||
with Image.open(path) as im:
|
||||
return im.size
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
def parse_db_position(path):
|
||||
"""Parse db_postion.txt -> list of (lon, lat)."""
|
||||
coords = []
|
||||
if not os.path.isfile(path):
|
||||
return coords
|
||||
with open(path) as f:
|
||||
for line in f:
|
||||
parts = line.strip().split()
|
||||
if len(parts) >= 3:
|
||||
try:
|
||||
lon, lat = float(parts[1]), float(parts[2])
|
||||
coords.append((lon, lat))
|
||||
except ValueError:
|
||||
pass
|
||||
return coords
|
||||
|
||||
def count_files_in_dir(d, exts=None):
|
||||
if not os.path.isdir(d):
|
||||
return 0
|
||||
if exts is None:
|
||||
return len(os.listdir(d))
|
||||
return sum(1 for f in os.listdir(d) if os.path.splitext(f)[1].lower() in exts)
|
||||
|
||||
def analyze_scene(scene_path):
|
||||
info = {}
|
||||
# DB crops
|
||||
db_img_dir = os.path.join(scene_path, "DB", "img")
|
||||
info["db_crops"] = count_files_in_dir(db_img_dir, {".png", ".jpg", ".jpeg", ".tif"})
|
||||
|
||||
# DB crop size (sample first image)
|
||||
info["crop_size"] = None
|
||||
if os.path.isdir(db_img_dir):
|
||||
for f in sorted(os.listdir(db_img_dir)):
|
||||
sz = get_image_size_safe(os.path.join(db_img_dir, f))
|
||||
if sz:
|
||||
info["crop_size"] = sz
|
||||
break
|
||||
|
||||
# merge.tif size
|
||||
merge_path = os.path.join(scene_path, "DB", "merge.tif")
|
||||
info["merge_size"] = get_image_size_safe(merge_path) if os.path.isfile(merge_path) else None
|
||||
|
||||
# Query variants
|
||||
query_dir = os.path.join(scene_path, "query")
|
||||
variants = []
|
||||
frames_per_variant = {}
|
||||
if os.path.isdir(query_dir):
|
||||
for v in sorted(os.listdir(query_dir)):
|
||||
vpath = os.path.join(query_dir, v)
|
||||
if os.path.isdir(vpath):
|
||||
footage_dir = os.path.join(vpath, "footage")
|
||||
n = count_files_in_dir(footage_dir, {".png", ".jpg", ".jpeg"})
|
||||
variants.append(v)
|
||||
frames_per_variant[v] = n
|
||||
info["variants"] = variants
|
||||
info["num_variants"] = len(variants)
|
||||
info["frames_per_variant"] = frames_per_variant
|
||||
# Use first variant's frame count as representative
|
||||
info["frames_per_variant_sample"] = list(frames_per_variant.values())[0] if frames_per_variant else 0
|
||||
info["total_query_frames"] = sum(frames_per_variant.values())
|
||||
|
||||
# db_postion.txt
|
||||
db_pos_path = os.path.join(scene_path, "DB", "db_postion.txt")
|
||||
coords = parse_db_position(db_pos_path)
|
||||
if coords:
|
||||
lons = [c[0] for c in coords]
|
||||
lats = [c[1] for c in coords]
|
||||
info["gps"] = {
|
||||
"lon_min": min(lons), "lon_max": max(lons),
|
||||
"lat_min": min(lats), "lat_max": max(lats),
|
||||
"num_entries": len(coords)
|
||||
}
|
||||
else:
|
||||
info["gps"] = None
|
||||
|
||||
# positive.json
|
||||
pos_path = os.path.join(scene_path, "positive.json")
|
||||
if os.path.isfile(pos_path):
|
||||
with open(pos_path) as f:
|
||||
pos = json.load(f)
|
||||
counts = [len(v) if isinstance(v, list) else 1 for v in pos.values()]
|
||||
info["positive"] = {
|
||||
"num_frames": len(pos),
|
||||
"total_positives": sum(counts),
|
||||
"avg_per_frame": sum(counts) / len(counts) if counts else 0,
|
||||
"min_per_frame": min(counts) if counts else 0,
|
||||
"max_per_frame": max(counts) if counts else 0,
|
||||
}
|
||||
else:
|
||||
info["positive"] = None
|
||||
|
||||
# semi_positive.json
|
||||
sp_path = os.path.join(scene_path, "semi_positive.json")
|
||||
if os.path.isfile(sp_path):
|
||||
with open(sp_path) as f:
|
||||
sp = json.load(f)
|
||||
counts = [len(v) if isinstance(v, list) else 1 for v in sp.values()]
|
||||
info["semi_positive"] = {
|
||||
"num_frames": len(sp),
|
||||
"total_semi_positives": sum(counts),
|
||||
"avg_per_frame": sum(counts) / len(counts) if counts else 0,
|
||||
"min_per_frame": min(counts) if counts else 0,
|
||||
"max_per_frame": max(counts) if counts else 0,
|
||||
}
|
||||
else:
|
||||
info["semi_positive"] = None
|
||||
|
||||
return info
|
||||
|
||||
|
||||
def main():
|
||||
terrain_types = sorted([d for d in os.listdir(ROOT) if os.path.isdir(os.path.join(ROOT, d))])
|
||||
|
||||
all_data = {} # terrain_type -> {scene_name -> info}
|
||||
grand_total_db = 0
|
||||
grand_total_query = 0
|
||||
grand_total_scenes = 0
|
||||
|
||||
for tt in terrain_types:
|
||||
tt_path = os.path.join(ROOT, tt)
|
||||
scenes = sorted([d for d in os.listdir(tt_path)
|
||||
if os.path.isdir(os.path.join(tt_path, d))])
|
||||
all_data[tt] = {}
|
||||
for sc in scenes:
|
||||
sc_path = os.path.join(tt_path, sc)
|
||||
# Check it's actually a scene (has DB dir)
|
||||
if not os.path.isdir(os.path.join(sc_path, "DB")):
|
||||
continue
|
||||
info = analyze_scene(sc_path)
|
||||
all_data[tt][sc] = info
|
||||
grand_total_db += info["db_crops"]
|
||||
grand_total_query += info["total_query_frames"]
|
||||
grand_total_scenes += 1
|
||||
|
||||
# ===================== PRINT RESULTS =====================
|
||||
|
||||
print("=" * 120)
|
||||
print("UAV-GeoLoc TERRAIN SUBSET - COMPREHENSIVE STATISTICS")
|
||||
print("=" * 120)
|
||||
|
||||
# 1. Hierarchy
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 1: COMPLETE HIERARCHY (TerrainType -> Scenes)")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<25} {'#Scenes':>7} Scenes")
|
||||
print("-" * 120)
|
||||
for tt in terrain_types:
|
||||
scenes = list(all_data.get(tt, {}).keys())
|
||||
if not scenes:
|
||||
print(f"{tt:<25} {'0':>7} (no valid scenes)")
|
||||
continue
|
||||
print(f"{tt:<25} {len(scenes):>7} {', '.join(scenes)}")
|
||||
print(f"\n{'TOTAL TERRAIN TYPES:':<25} {len(terrain_types)}")
|
||||
print(f"{'TOTAL SCENES:':<25} {grand_total_scenes}")
|
||||
|
||||
# 2. Per-scene counts
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 2: PER-SCENE IMAGE COUNTS")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<20} {'Scene':<40} {'DB Crops':>9} {'#Variants':>10} {'Frames/Var':>11} {'Total QFrames':>14}")
|
||||
print("-" * 120)
|
||||
for tt in terrain_types:
|
||||
for sc, info in sorted(all_data.get(tt, {}).items()):
|
||||
print(f"{tt:<20} {sc:<40} {info['db_crops']:>9} {info['num_variants']:>10} {info['frames_per_variant_sample']:>11} {info['total_query_frames']:>14}")
|
||||
|
||||
print(f"\n{'GRAND TOTAL DB CROPS:':<50} {grand_total_db:>14}")
|
||||
print(f"{'GRAND TOTAL QUERY FRAMES:':<50} {grand_total_query:>14}")
|
||||
print(f"{'GRAND TOTAL ALL IMAGES:':<50} {grand_total_db + grand_total_query:>14}")
|
||||
|
||||
# 3. Crop & merge sizes
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 3: CROP AND MERGE.TIF SIZES (pixels)")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<20} {'Scene':<40} {'Crop WxH':>12} {'Merge WxH':>14}")
|
||||
print("-" * 120)
|
||||
for tt in terrain_types:
|
||||
for sc, info in sorted(all_data.get(tt, {}).items()):
|
||||
cs = f"{info['crop_size'][0]}x{info['crop_size'][1]}" if info['crop_size'] else "N/A"
|
||||
ms = f"{info['merge_size'][0]}x{info['merge_size'][1]}" if info['merge_size'] else "N/A"
|
||||
print(f"{tt:<20} {sc:<40} {cs:>12} {ms:>14}")
|
||||
|
||||
# Summary of unique sizes
|
||||
crop_sizes = defaultdict(int)
|
||||
merge_sizes = defaultdict(int)
|
||||
for tt in terrain_types:
|
||||
for sc, info in all_data.get(tt, {}).items():
|
||||
if info['crop_size']:
|
||||
crop_sizes[info['crop_size']] += 1
|
||||
if info['merge_size']:
|
||||
merge_sizes[info['merge_size']] += 1
|
||||
print(f"\nUnique crop sizes: {dict(crop_sizes)}")
|
||||
print(f"Unique merge.tif sizes: {dict(merge_sizes)}")
|
||||
|
||||
# 4. GPS coordinate ranges
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 4: GPS COORDINATE RANGES (from db_postion.txt)")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<20} {'Scene':<35} {'#Entries':>8} {'Lon Min':>12} {'Lon Max':>12} {'Lat Min':>12} {'Lat Max':>12}")
|
||||
print("-" * 120)
|
||||
for tt in terrain_types:
|
||||
for sc, info in sorted(all_data.get(tt, {}).items()):
|
||||
g = info["gps"]
|
||||
if g:
|
||||
print(f"{tt:<20} {sc:<35} {g['num_entries']:>8} {g['lon_min']:>12.6f} {g['lon_max']:>12.6f} {g['lat_min']:>12.6f} {g['lat_max']:>12.6f}")
|
||||
else:
|
||||
print(f"{tt:<20} {sc:<35} {'N/A':>8} {'N/A':>12} {'N/A':>12} {'N/A':>12} {'N/A':>12}")
|
||||
|
||||
# 5. positive.json stats
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 5: positive.json STATS")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<20} {'Scene':<35} {'#Frames':>8} {'TotalPos':>9} {'AvgPos':>8} {'MinPos':>7} {'MaxPos':>7}")
|
||||
print("-" * 120)
|
||||
all_pos_avg = []
|
||||
for tt in terrain_types:
|
||||
for sc, info in sorted(all_data.get(tt, {}).items()):
|
||||
p = info["positive"]
|
||||
if p:
|
||||
print(f"{tt:<20} {sc:<35} {p['num_frames']:>8} {p['total_positives']:>9} {p['avg_per_frame']:>8.2f} {p['min_per_frame']:>7} {p['max_per_frame']:>7}")
|
||||
all_pos_avg.append(p['avg_per_frame'])
|
||||
else:
|
||||
print(f"{tt:<20} {sc:<35} {'N/A':>8} {'N/A':>9} {'N/A':>8} {'N/A':>7} {'N/A':>7}")
|
||||
if all_pos_avg:
|
||||
print(f"\nOverall avg positives per frame across all scenes: {sum(all_pos_avg)/len(all_pos_avg):.2f}")
|
||||
|
||||
# 6. semi_positive.json stats
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 6: semi_positive.json STATS")
|
||||
print("=" * 120)
|
||||
print(f"{'TerrainType':<20} {'Scene':<35} {'#Frames':>8} {'TotalSP':>9} {'AvgSP':>8} {'MinSP':>7} {'MaxSP':>7}")
|
||||
print("-" * 120)
|
||||
all_sp_avg = []
|
||||
for tt in terrain_types:
|
||||
for sc, info in sorted(all_data.get(tt, {}).items()):
|
||||
sp = info["semi_positive"]
|
||||
if sp:
|
||||
print(f"{tt:<20} {sc:<35} {sp['num_frames']:>8} {sp['total_semi_positives']:>9} {sp['avg_per_frame']:>8.2f} {sp['min_per_frame']:>7} {sp['max_per_frame']:>7}")
|
||||
all_sp_avg.append(sp['avg_per_frame'])
|
||||
else:
|
||||
print(f"{tt:<20} {sc:<35} {'N/A':>8} {'N/A':>9} {'N/A':>8} {'N/A':>7} {'N/A':>7}")
|
||||
if all_sp_avg:
|
||||
print(f"\nOverall avg semi-positives per frame across all scenes: {sum(all_sp_avg)/len(all_sp_avg):.2f}")
|
||||
|
||||
# 7. Variant breakdown (unique variant names across dataset)
|
||||
print("\n" + "=" * 120)
|
||||
print("TABLE 7: QUERY VARIANT NAMES (height/rot combinations)")
|
||||
print("=" * 120)
|
||||
all_variants = set()
|
||||
for tt in terrain_types:
|
||||
for sc, info in all_data.get(tt, {}).items():
|
||||
all_variants.update(info["variants"])
|
||||
for v in sorted(all_variants):
|
||||
print(f" {v}")
|
||||
print(f"\nTotal unique variant names: {len(all_variants)}")
|
||||
|
||||
print("\n" + "=" * 120)
|
||||
print("END OF REPORT")
|
||||
print("=" * 120)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user