TALOS detects aircraft and ships in satellite video, keeps a stable ID on each one, and estimates the probability that each will enter a protected zone within the next 10 seconds. An operator gets a warning before the zone is reached.
Demo (precomputed clips): huggingface.co/spaces/dustin-demo/talos-satellite-tracking. It shows 8 clips from VISO, a satellite dataset the model never trained on, with the misses included.
API on AWS Lambda: https://a6yezc4gltpzekp4a4tasydyxm0rqdgx.lambda-url.us-east-1.on.aws/. POST a short clip and get tracks and zone-entry alerts back; see Call the API.
VISO clip 039, CC BY-NC-SA 4.0 (Yin et al., IEEE TGRS 2021); this annotated GIF is shared under the same license. The plane heading down the runway is flagged 7.3 s before it enters the zone.
| Stage | Method |
|---|---|
| Detect | YOLO26s fine-tuned on 1024 px tiles of real Jilin-1 video. The NMS-free end-to-end head runs, and an edge-aware merge rejoins aircraft cut by tile borders. |
| Track | TALOS-Track: a Kalman filter per object (state cx, cy, vx, vy, w, h). It adds a tentative/confirmed/lost lifecycle, the real time step, Mahalanobis χ² gating, two-stage (high/low confidence) association, OC-SORT-style re-update and duplicate-track suppression. Measurement noise is calibrated from detector errors. |
| Threat | P(enter zone within 10 s): Monte Carlo rollouts from each track's state and covariance. Motion noise is fitted on train by Brier score, and the alert threshold is the 5%-false-alarm point on val. |
Per-stage timings are in Speed: detection is nearly all the cost.
Every number in this README comes from a JSON file in results/ written by a Slurm job, and the tables are regenerated by scripts/readme_tables.py. The test split was used once per final configuration, and each use is logged in results/TEST_LOG.md.
| Dataset | Use | Notes |
|---|---|---|
| AIR-MOT-100 (He et al., IEEE TGRS 2022) | train / val / test | 69 annotated Jilin-1 videos at 1920×1080, aircraft and ships, split 47/11/11 by video. The split is frozen in configs/splits/airmot.json. No license is stated, so no frames are redistributed. See docs/data/airmot.md. |
| VISO (Yin et al., IEEE TGRS 2021) | cross-dataset check only | CC BY-NC-SA 4.0. It labels only moving objects. See docs/data/viso.md. |
AIR-MOT does not state a frame rate; 10 fps is assumed throughout. Distances are in pixels.
Every tracker gets the same detections and the same tuning budget: 30 random configs on val, scored with TrackEval HOTA. Best value per column in bold.
Aircraft (AIR-MOT test, 5 videos)
| tracker | HOTA | IDF1 | MOTA | AssA | ID switches |
|---|---|---|---|---|---|
| TALOS v1 | 64.1 | 76.8 | 46.5 | 81.8 | 5 |
| SORT | 69.3 | 85.2 | 77.0 | 76.7 | 47 |
| ByteTrack | 73.2 | 88.7 | 86.2 | 78.9 | 42 |
| OC-SORT | 71.9 | 87.4 | 88.3 | 74.9 | 36 |
| BoT-SORT | 75.0 | 92.4 | 86.4 | 82.7 | 3 |
| TALOS-Track | 77.4 | 96.4 | 92.6 | 83.8 | 0 |
Ships (AIR-MOT test, 7 videos)
| tracker | HOTA | IDF1 | MOTA | AssA | ID switches |
|---|---|---|---|---|---|
| TALOS v1 | 48.1 | 59.6 | -7.3 | 71.3 | 5 |
| SORT | 42.6 | 50.6 | 8.5 | 54.2 | 118 |
| ByteTrack | 51.7 | 66.7 | 29.1 | 69.8 | 17 |
| OC-SORT | 47.5 | 63.6 | 34.1 | 62.0 | 31 |
| BoT-SORT | 52.3 | 68.4 | 38.6 | 69.4 | 7 |
| TALOS-Track | 52.9 | 68.8 | 37.6 | 69.7 | 12 |
The whole v1 system (its own detector and tracker) scores 0.0 HOTA on the same aircraft test videos: its detector, trained on synthetic images, finds almost nothing in real satellite video.
On aircraft, TALOS-Track is best on every column, with no ID switches. On ships it edges BoT-SORT on HOTA and IDF1 but trails it on MOTA and has more ID switches (12 vs 7); v1 has the highest ship AssA because it keeps few, long tracks. Ship scores are low for every tracker because the detector's ship precision is low (see below).
Fair baselines matter. How much depends on the detector. With the final detector, tuning moves HOTA only a few points. With the earlier, noisier YOLO26m detections, library defaults understated some baselines by up to 24 HOTA, so comparing against untuned baselines would have flattered TALOS-Track:
| val HOTA, defaults → tuned | final detector (YOLO26s, 1024 tiles) | gain | earlier detector (YOLO26m, 640 tiles) | gain |
|---|---|---|---|---|
| SORT (planes) | 61.0 → 62.9 | +1.9 | 58.8 → 60.5 | +1.8 |
| ByteTrack (planes) | 66.1 → 66.1 | +0.0 | 58.0 → 62.3 | +4.4 |
| OC-SORT (planes) | 67.2 → 67.2 | +0.0 | 50.3 → 58.6 | +8.4 |
| BoT-SORT (planes) | 67.4 → 68.4 | +1.0 | 56.5 → 61.4 | +5.0 |
| TALOS-Track (planes) | 66.7 → 67.7 | +1.0 | 61.5 → 62.4 | +0.9 |
| SORT (ships) | 29.0 → 36.3 | +7.3 | 38.0 → 41.2 | +3.2 |
| ByteTrack (ships) | 42.6 → 43.4 | +0.7 | 25.5 → 44.4 | +18.9 |
| OC-SORT (ships) | 39.6 → 41.8 | +2.2 | 16.4 → 40.5 | +24.0 |
| BoT-SORT (ships) | 42.7 → 44.8 | +2.0 | 26.0 → 42.4 | +16.5 |
| TALOS-Track (ships) | 43.3 → 45.4 | +2.1 | 35.4 → 43.4 | +8.0 |
Duplicate-track suppression. The ID-switch report traced most aircraft ID switches to one cause: a second track born on an aircraft that already had one. Suppressing the younger twin took aircraft ID switches from 49 to 0 on val. On ships it raised ID switches (7 to 20) while HOTA and IDF1 still improved slightly, and so the tuned ship setting suppresses only near-identical twins (a higher overlap threshold than for aircraft).
| val | ID switches before | after | HOTA | IDF1 |
|---|---|---|---|---|
| planes | 49 | 0 | 66.2 → 67.7 | 78.3 → 80.1 |
| ships | 7 | 20 | 44.9 → 45.4 | 53.8 → 55.4 |
Component ablation (val, early design check)
Day-3 ablation on AIR-MOT val aircraft. It uses ground-truth boxes degraded with 20% misses, jitter and false positives, with untuned settings and before duplicate suppression existed. Steps are cumulative. The Mahalanobis gate and two-stage association give the largest gains. Lifecycle management and the real time step on their own lower HOTA on these degraded boxes; they pay off only once the gate and the second association stage are in place.
| cumulative step | HOTA | IDF1 | MOTA | ID switches |
|---|---|---|---|---|
| constant-velocity Kalman + Hungarian (v1-like) | 43.4 | 46.2 | -42.9 | 569 |
| + track lifecycle (tentative / confirmed / lost) | 37.3 | 51.3 | 42.2 | 818 |
| + real time step | 31.8 | 40.4 | 41.6 | 835 |
| + Mahalanobis gate | 41.4 | 60.2 | 44.1 | 475 |
| + two-stage (ByteTrack-style) association | 56.9 | 72.9 | 62.7 | 635 |
| + OC-SORT re-update | 56.5 | 72.2 | 61.5 | 772 |
| + coasting output | 61.1 | 76.9 | 64.4 | 183 |
| ByteTrack, same detections | 55.4 | 68.5 | 67.9 | 69 |
YOLO26s trained on 1024 px tiles was chosen over 640 px tiles, YOLO26m, and 512 px tiles with 2× upsampling, on full-frame val AP50. Tile-level validation mAP ranked these models in the wrong order. Recall is reported by object size because that is where tiling choices win or lose.
| AIR-MOT | AP50 | precision | recall | recall <16 px | 16–32 px | >32 px |
|---|---|---|---|---|---|---|
| planes, val | 0.778 | 0.79 | 0.77 | 0.06 | 0.70 | 1.00 |
| ships, val | 0.441 | 0.32 | 0.71 | 0.76 | 0.81 | 0.45 |
| planes, test | 0.984 | 0.83 | 1.00 | – | 1.00 | 1.00 |
| ships, test | 0.586 | 0.50 | 0.76 | 0.88 | 0.87 | 0.41 |
| v1 detector, planes, val | 0.007 |
The test split has no aircraft under 16 px, so test AP is higher than val AP. On val, aircraft under 16 px are mostly missed.
Each zone is placed on a track's future path ("on-path") or offset beside it ("near-miss"). A good score ranks the on-path zones above the near-miss zones before entry. v1 is the original hand-weighted score; v2 is the entry probability.
| data | zones | AUROC v2 | AUROC v1 | AP v2 | AP v1 | warned v2 | warned v1 | median lead v2 |
|---|---|---|---|---|---|---|---|---|
| AIR-MOT val | 50 | 0.855 | 0.736 | 0.583 | 0.422 | 19/24 | 19/24 | 8.4 s |
| VISO (never trained on) | 68 | 0.978 | 0.922 | 0.880 | 0.704 | 34/34 | 32/34 | 9.4 s |
On the eight VISO clips in the demo, run with real detections:
| VISO clip | class | labelled objects found | zone warning |
|---|---|---|---|
| viso-039 | plane | 90% | warned 7.3 s before it entered the zone |
| viso-042 | plane | 95% | warned 5.0 s before it entered the zone |
| viso-043 | plane | 99% | warned 4.6 s before it entered the zone |
| viso-040 | plane | 42% | late: first alert 10.1 s after it entered the zone |
| viso-041 | plane | 40% | missed: tracked, but no alert before it entered the zone |
| viso-044 | plane | 0% | missed: the detector never found the labelled object |
| viso-045 | ship | 0% | missed: the detector never found the labelled object |
| viso-047 | ship | 0% | missed: the detector never found the labelled object |
The model was exported to ONNX (the NMS-free head is kept, preprocessing is fused into the graph) and TensorRT. Every runtime is checked against PyTorch on the full val split: AP50 must be within 0.005 and the box count within 1%.
| runtime (one A100) | ms / frame | speed-up | plane AP50 | ship AP50 | matches PyTorch |
|---|---|---|---|---|---|
| ONNX Runtime + TensorRT fp16 | 12.7 | 2.11× | 0.7776 | 0.4399 | yes |
| TensorRT engine (Ultralytics) fp16 | 16.0 | 1.68× | 0.7778 | 0.4399 | yes |
| PyTorch fp16 | 22.2 | 1.21× | 0.7776 | 0.4400 | yes |
| PyTorch fp32 (reference) | 26.9 | 1.00× | 0.7775 | 0.4406 | yes |
| ONNX Runtime, CUDA fp32 | 82.0 | 0.33× | 0.7774 | 0.4405 | yes |
End to end (decode, detect, track, threat), AIR-MOT clip 043:
| hardware, runtime | frames | fps | decode ms | detect ms | track ms | threat ms |
|---|---|---|---|---|---|---|
| A100, PyTorch fp32 | 327 | 27.43 | 5.3 | 28.7 | 1.6 | 0.2 |
| 32 CPU cores, PyTorch | 40 | 1.07 | 47.4 | 879.6 | 1.6 | 4.8 |
| 32 CPU cores, ONNX Runtime fp32 | 40 | 2.70 | 32.6 | 336.3 | 1.5 | 0.6 |
| 32 CPU cores, ONNX Runtime dynamic INT8 | 40 | 0.34 | 26.2 | 2931.9 | 1.4 | 0.4 |
| 2 CPU threads, ONNX Runtime fp32 | 40 | 0.55 | 23.7 | 1807.0 | 1.5 | 0.6 |
Dynamic INT8 quantization made the CPU slower, so it is not used. Tracking and threat scoring stay in the low milliseconds on every machine; detection is the cost.
git clone https://github.com/Gakshith/Tactical-Aerospace-Localization-Objective-Scoring-TALOS-.git
cd Tactical-Aerospace-Localization-Objective-Scoring-TALOS-
pip install torch==2.11.0 torchvision==0.26.0 --index-url https://download.pytorch.org/whl/cpu
pip install -r env/requirements-cpu.txt
bash env/patch_trackeval.sh "$(python -c 'import sys; print(sys.prefix)')" # NumPy 2 fix for TrackEvalOr use the Docker image (CPU), which every push to main rebuilds and tests:
docker run --rm ghcr.io/gakshith/talos:latest # README check + tests
docker build -t talos . # or build it yourselfThe same pipeline runs on AWS Lambda (CPU, ONNX Runtime, 3008 MB), behind a public function URL. Send up to 40 frames of a clip (mp4, at most about 4 MB) and an optional zone in the clip's pixels:
python - <<'PY'
import base64, json, urllib.request
body = {"video_b64": base64.b64encode(open("clip.mp4", "rb").read()).decode(),
"zone_circle": [776, 376, 25], "fps": 10}
req = urllib.request.Request("https://a6yezc4gltpzekp4a4tasydyxm0rqdgx.lambda-url.us-east-1.on.aws/", json.dumps(body).encode(), {"content-type": "application/json"})
out = json.load(urllib.request.urlopen(req, timeout=150))
print(out["tracks"], out["alerts"][:3], out["timing"])
PYThe reply lists each track (class, id, first and last frame, highest entry probability), every alert over the val-tuned threshold, and per-stage timing. GET on the URL returns the limits. Frames longer than 1280 px are downscaled first. A 40-frame 1454×750 VISO clip takes about 60 s (1.4 s per frame).
The code is in deploy/lambda/. Every push to main that touches the pipeline rebuilds the image, pushes it to ECR and updates the function (workflow); GitHub reaches AWS through an OIDC role only main can assume, so no AWS keys are stored in GitHub. setup_aws.sh creates the AWS side once.
GPU pins (CUDA 12.8, as used on Polaris) are in env/requirements-v2.txt.
Download the detector (ONNX, 38 MB, YOLO26s fine-tuned on AIR-MOT tiles):
curl -LO https://huggingface.co/spaces/dustin-demo/talos-satellite-tracking/resolve/main/models/talos_y26s_t1024.onnxWhole pipeline on a video or a folder of frames:
python -m src.TALOS.pipeline.run VIDEO_OR_FRAME_DIR --out out/ --zone-circle CX,CY,R --device cpu \
--backend onnx --weights talos_y26s_t1024.onnxIt writes annotated.mp4, tracks.csv, alerts.jsonl (one line per alerting track and frame, with p_entry and time to entry) and timing.json. Settings come from configs/talos_v2.json: the tuned tracker per class and the alert threshold.
The benchmark, tuning and export runs are Slurm jobs in jobs/ (SFSU Polaris, A100 nodes):
| Job | What it does |
|---|---|
train_det.sbatch |
Fine-tunes YOLO26 on AIR-MOT tiles |
detect.sbatch |
Fills the per-sequence detection cache |
tune.sbatch |
Equal-budget tracker tuning on val |
export_parity.sbatch |
ONNX/TensorRT export and the parity + speed table |
run_e2e.sbatch, e2e_cpu_ort.sbatch |
End-to-end timing |
space_examples.sbatch |
Renders the demo clips |
Tests: python -m pytest tests. CI runs them, and python scripts/readme_tables.py --check, on every push and pull request.
src/TALOS/
data/ AIR-MOT and VISO converters to MOTChallenge format, frozen split
detect/ tiled detector, ONNX Runtime backend, export, detection cache
track/ TALOS-Track, baselines (via roboflow trackers), v1 baseline, noise calibration
threat/ zone-entry probability and its evaluation
eval/ TrackEval runner, benchmark and tuning harness, detector metrics
pipeline/ run.py: video in, tracks + alerts out
configs/ tuned settings, split, ablation configs
data_meta/ dataset conversion reports and the per-video track list used for the split
docs/data/ dataset notes: format, fixes, caveats
deploy/ AWS Lambda handler, image and one-time AWS setup
env/ pinned environments (GPU on Polaris, CPU for CI and Docker)
jobs/ Slurm scripts
results/ every metrics file the README reads
scripts/ README tables, static Space builder, sync, reports
space/ Gradio app (tested; Hugging Face hosts Gradio only on PRO)
tests/
v1 (a fine-tuned YOLO26s on 500 synthetic images, a fixed-gate tracker, a hand-weighted score, run on one synthetic clip) is kept verbatim in src/TALOS/track/v1_legacy.py and appears in the tables as "TALOS v1".
- Aircraft under 16 px are mostly missed (val recall 5%), and 2× upsampling did not fix it.
- Ships have many false positives (val precision 0.32), which caps every tracker's ship scores.
- The detector misses VISO's ships (a few pixels across) and some VISO aircraft. The demo lists these misses.
- Pixels, not metres; 10 fps assumed for AIR-MOT.
- The Lambda API runs on CPU at about 1.4 s per frame, so it suits short clips, not a live feed.
Code: MIT. The demo GIF and the Space's clips are VISO derivatives, CC BY-NC-SA 4.0.
