A self-updating, open threat intelligence feed.
xfeeds publishes curated lists of known-bad IP addresses that anyone can pull into a firewall, WAF, reverse proxy, or SIEM. The repository itself is the distribution mechanism: feeds are plain files under feeds/, regenerated on a schedule by GitHub Actions and served over GitHub Pages.
What makes it different from the dozens of public blocklists it draws on: xfeeds only publishes what independent sources agree on. Most public feeds quietly mirror each other, so naive aggregation manufactures false confidence. xfeeds models that dependency explicitly and throws away the ~77% of candidate IPs that only one independent sensor network has ever seen.
xfeeds started as a one-time publication. During a ThreatX SOC "do good" day, ~5 million WAF match events were analyzed to profile the sources of nefarious HTTP traffic. The resulting IPs were validated against GreyNoise to strip benign scanner noise — GreyNoise had previously seen 71 of them and had already tagged 33 as malicious, leaving 2,901 previously-unpublished bad actors. That snapshot lives on, frozen, as legacy/IP_List (2,973 entries, April 2022).
That dataset is stale and there is no longer a private WAF telemetry source behind it. Phase 2 replaces one-time human curation with an automated aggregation pipeline built on public, free-to-use intelligence.
- Automate. No manual steps. A scheduled workflow ingests, normalizes, scores, filters, and commits.
- Corroborate, don't concatenate. Value comes from independent agreement, not from republishing someone else's list.
- Be conservative. A published block feed must be safe to drop traffic on. False positives are the primary risk.
- Be transparent. Every published IP traces back to the sources, licences, and scores that put it there.
- Cost nothing to run. Free tiers and GitHub-hosted runners only.
- Stay current, stay boring. Newest tool that has universal support and can be swapped out later.
Full rationale, alternatives considered, and measurements in docs/DECISIONS.md.
| Layer | Choice | Rationale |
|---|---|---|
| Runtime | Python 3.13 (floor 3.11, CI also tests 3.14) | One release behind newest — complete wheel coverage, supported to Oct 2029 |
| Packaging | uv + pyproject.toml + uv.lock, exact version pinned in CI |
Fast, but chosen because its metadata is standard [project] — pip and Poetry read the same file |
| HTTP | httpx (sync) | HTTP/2, real timeouts, first-class test mocking |
| Validation | pydantic v2 | Publishes a JSON Schema for consumers for free |
| Retries | tenacity | Policy for 429/5xx that transport-level retry doesn't cover |
| CLI / logs / config | typer, structlog (JSON), pyyaml | Boring, stable, SIEM-ingestible logs |
| Interop | STIX 2.1 | Deterministic, hand-built emitter; no stix2 dependency |
| Quality | ruff, mypy --strict, pytest + pytest-httpx | One linter binary; no unit test touches the network |
| State | feeds/all.json — no database |
Self-contained repo, every state change reviewable in a diff |
Deliberately not used: netaddr (stdlib ipaddress suffices), orjson/pandas/polars (wrong tool at 50k records), a database, free-threaded Python, and a TAXII server.
upstream sources ──► collectors ──► normalizer ──► scorer ──► filters ──► emitters ──► feeds/
(HTTP fetch) (per-source (dedupe, (independence (allowlist, (txt, csv, json,
parsers) CIDR expand, -weighted, aging, stix, misp,
validate) saturating) sanity caps) nftables)
| Stage | Responsibility |
|---|---|
| Collect | Fetch over HTTPS with retry/backoff, ETag caching, per-source timeout and minimum interval. A failing source degrades the run; it never fails it. |
| Normalize | Parse into a common record: ip, cidr, source, independence_class, first_seen, last_seen, categories[]. Validate as public IPv4/IPv6. |
| Score | Independence-weighted confidence with recency decay and exponential saturation. |
| Filter | Drop allowlisted, reserved, and known-good infrastructure. Enforce CIDR-width and feed-size caps. Enforce per-source redistribute flags. |
| Emit | Write feed artifacts, feeds/manifest.json, and a run report. |
| Publish | Commit changed files to main; open an issue on anomalies. |
Configured entirely in sources.yaml. Adding a source means adding a YAML entry and, if the format is new, a parser — never a pipeline change.
For what each upstream documents about its own sensors and scoring — and why we derive our own score from independent presence rather than composing theirs — see docs/source-methodology-2026-08.md.
Eleven independent voting classes, each a distinct sensor network, reporter community, or research team. AbuseIPDB and ThreatFox are keyed sources: both are enabled in sources.yaml, and both skip cleanly when their key is absent from the environment, so a fresh clone with no secrets still produces a correct feed and simply gains accuracy once the keys are set. ABUSEIPDB_API_KEY was added to repo secrets on 2026-08-14; AbuseIPDB remains redistribute: false per ADR-012, so it raises confidence without contributing rows to any published file.
| Source | Class | Weight | Volume | Notes |
|---|---|---|---|---|
| Spamhaus DROP v4/v6 | spamhaus |
1.0 | 1,687 + 92 CIDRs | Auto-promotes to high confidence |
| Feodo Tracker | abusech |
1.0 | 5 IPs | Expired — frozen since March 2026, contributes nothing |
| SSLBL | abusech |
1.0 | 0 currently | Wired up, harmless |
| ThreatFox | abusech |
1.0 | key required | Auth-Key mandatory |
| Blocklist.de | blocklist_de |
0.8 | 28,605 | Largest independent sensor net |
| CINS Army | cins |
0.8 | 15,000 | No formal licence — see ADR-012 |
| AbuseIPDB | abuseipdb |
0.9 | 10,000 | Scoring only, not redistributed |
| GreenSnow | greensnow |
0.6 | 3,595 | |
| Binary Defense | binary_defense |
0.6 | 3,300 | Requires browser-like UA |
| bruteforceblocker | bruteforceblocker |
0.6 | 549 | Upstream of ET compromised-ips |
| DataPlane.org | dataplane |
0.7 | 9,771 | Own SSH sensors, 7-day window. Scoring only in both tiers |
| DShield | dshield |
0.5 | 20 × /24 | CC BY-NC-SA — non-commercial tier only |
Not voting, and why — the important half of the design:
| Source | Status | Reason |
|---|---|---|
| IPsum L3–L8 | Prior only, weight 0 | Aggregates 30+ lists including most of ours. 35% of Blocklist.de and 64% of Binary Defense reappear inside it. |
| FireHOL level1 | Disabled | Its header declares it composed of sources we already ingest — and it spans 611M addresses including 224.0.0.0/3. |
| ET compromised-ips | Disabled | Jaccard 0.953 against bruteforceblocker. A mirror, not a second opinion. |
| ET compromised-ips | Disabled | Also 96% contained in the duggytuxy list, and class-pinned to bruteforceblocker, so it cannot add a vote. |
| SSLBL (IP list) | Disabled | Not empty — retired. The file says "deprecated on 2025-01-03". |
| sefinek | Disabled | MIT and the most independent list we have measured, but measured across 140 upstream commits it removes nothing — zero retractions in 32 days. See ADR-057. |
| duggytuxy Data-Shield | Disabled | Claims own probes; contains 90.7% of everything we publish. A re-aggregator by measurement. |
| Tor exits | Tag only | Blocking Tor is the consumer's policy choice, not a threat assertion. |
raw = Σ over distinct independence classes:
max(weight × recency_factor × severity for sources in that class)
recency_factor = max(0.2, 1 − days_since_last_seen / ttl_days)
score = 100 × (1 − exp(−raw))
One vote per class, and exponential saturation, so no single source can alone reach the top band. Spamhaus DROP membership and active abuse.ch C2 listings promote directly — justified by precision rather than agreement.
| Directory | Contents | Licence |
|---|---|---|
feeds/ |
Everything we may republish | Per-source; several publishers grant nothing explicitly |
feeds/noncommercial/ |
Adds CC BY-NC-SA sources | CC BY-NC-SA 4.0 |
feeds/clean/ |
Only sources with a written, named licence permitting commercial redistribution | Per-source, all permissive — see clean/LICENSE.txt |
Use feeds/clean/ if you have to satisfy a legal review. It is much smaller — 23 entries against 4,270 on a 2026-08-14 run — and that is the trade: the primary feed also contains data from publishers who distribute freely but never actually granted a reuse licence, and absence of a prohibition is not a grant. clean/LICENSE.txt names the licence, the licence URL and the required credit for every contributing source. See ADR-051.
Only classes we are licensed to republish count toward the threshold that admits a record. Sources we may consume but not republish (AbuseIPDB, GreenSnow, ThreatFox, DataPlane) can raise a record from medium to high, but can never bring one into the feed on their own, and their names are withheld from published records so the feed cannot disclose their membership. See ADR-035.
That split is what makes a restricted source safe to use, and ADR-048 measured the payoff: adding DataPlane and DShield left the published count unchanged at 3,518 while upgrading 375 records from medium to high. Pure corroboration, zero new exposure.
A successful fetch means the endpoint answered, not that anything behind it is current. Evidence age is taken from the source's own declared publication time where it has one, falling back to the HTTP Last-Modified header and then to a content hash — in that order, because abuse.ch serves a Last-Modified that moves independently of its payload.
Three states follow from that one number:
| Evidence age | State | What it contributes |
|---|---|---|
within min(30 days, ttl_days) |
Active | full vote, can admit, can promote |
| past that, up to 90 days | Stale | damped ×0.2, cannot admit, cannot promote |
past 90 days, or dormant: true |
Expired | nothing — records dropped before scoring |
Expired sources are still fetched. The fetch stops being a scoring input and becomes a review trigger: evidence_age_days falling in the manifest is what tells a maintainer the upstream is alive again.
They do not come back on their own. Re-admission needs a reviewed_on date in sources.yaml on or after the expiry recorded in feeds/source-freshness.json, which is the point at which somebody re-checks the licence, the sensor method, and the content shape. Fresh data is the prompt to review, not the review. Full policy in docs/source-lifecycle.md.
Those are two different rights, and the difference is larger than it looks. As of 2026-09-01 there are 13 voting classes and 6 admitting ones. The other seven — abusech, abuseipdb, dataplane, dshield, greensnow, stopforumspam, turris — can strengthen a record that two live classes already admitted, and can never be one of those two, because their licence forbids redistribution or the upstream behind them has stopped publishing.
manifest.json reports both, plus voting_only_classes and a per-category category_coverage block. Four categories currently have no admitting class at all:
| Category | Voting | Admitting | Why |
|---|---|---|---|
botnet-c2 |
1 | 0 | Feodo Tracker expired, ThreatFox non-redistributable |
abuse |
2 | 0 | StopForumSpam and DataPlane both non-redistributable |
spam-source |
2 | 0 | the same two |
telnet-attack |
1 | 0 | DataPlane only |
Those are reported as status: "corroboration-only" rather than as a bare zero, because the two situations a zero could mean are very different: we watch all four categories and vote on them, we just may not publish an address on that evidence alone. It is a licensing outcome, not a blind spot. See ADR-058.
For botnet-c2 specifically the free ecosystem has no fix, and the 2026-09-01 review measured why: ET's emerging-botcc.rules is generated from abuse.ch and carries the identical five dead addresses as Feodo; Viriback states no licence and accumulates back to 2019; TweetFeed is CC0 but its sensor method is "somebody tweeted it". The one live, high-quality option is ThreatFox, and licence is the single thing blocking it.
Measured against live data on 2026-08-11, of 54,241 unique candidate IPs:
| Distinct classes | IPs | Share | Band |
|---|---|---|---|
| 1 | 42,034 | 77.5% | withheld |
| 2 | 10,294 | 19.0% | medium |
| 3+ | 1,913 | 3.5% | high |
Expected published high-confidence feed: ~2,000–4,000 entries — the same order of magnitude as the original 2022 list. Discarding 77% of the input is the product working, not a bug.
Live at https://neilweitzel.github.io/xfeeds/ — a dashboard with current counts, history and per-source status, plus every artifact below.
Pages is the canonical URL rather than raw.githubusercontent.com, which caps
near 5,000 requests/hour/IP and blocks the IP for 30 minutes when exceeded.
| File | Family | Use |
|---|---|---|
high-confidence.txt |
both | Safe-to-block. One IP or CIDR per line, # comments. |
high-confidence-v4.txt |
IPv4 | Same, IPv4 only. Use this if your tooling is single-stack. |
high-confidence-v6.txt |
IPv6 | Same, IPv6 only. |
medium-confidence.txt |
both | Challenge or rate-limit rather than a hard block. |
medium-confidence-v4.txt / -v6.txt |
one each | Family-split equivalents. |
all.csv |
both | MISP CSV feeds, OpenCTI CSV mappers, Splunk lookups. Carries address_family and source_reference. |
all.json + schema.json |
both | Full provenance, with a published JSON Schema. |
stix-bundle.json |
both | STIX 2.1 — OpenCTI, Elastic, most TIPs. |
misp-manifest.json |
both | Native MISP feed. |
nftables.conf |
both | blocklist4 and blocklist6 sets in one file. |
iptables.ipset |
IPv4 | ipset restore format, set xfeeds. |
iptables6.ipset |
IPv6 | ipset restore format, set xfeeds6. An ipset holds one family. |
manifest.json |
— | Per-source status, licences, counts, deltas, per-family breakdown. Carries both active_voting_classes and active_admitting_classes — see below, they are different numbers. Also evidence_time, evidence_age_days, and evidence_basis per source — how old the upstream's own data is, and which of the three mechanisms in docs/source-lifecycle.md established that. Distinct from last_modified, which is the raw HTTP header and is not always honest about the payload behind it. |
source-freshness.json |
— | When each source's body last actually changed. Backs the content-hash arm of evidence ageing for sources that declare no timestamp. |
history.json |
— | Rolling per-run history behind the charts. |
The combined files carry both address families and are unchanged — existing firewall URL tables keep working. If your tooling is single-stack, point it at the suffixed file: a v4-only parser will choke on an IPv6 line.
ipset / iptables
An ipset holds exactly one address family, so IPv6 needs its own set and rule. Skip the second pair if you are single-stack IPv4.
curl -sS https://neilweitzel.github.io/xfeeds/iptables.ipset | sudo ipset restore -!
sudo iptables -I INPUT -m set --match-set xfeeds src -j DROP
curl -sS https://neilweitzel.github.io/xfeeds/iptables6.ipset | sudo ipset restore -!
sudo ip6tables -I INPUT -m set --match-set xfeeds6 src -j DROPnftables
curl -sSO https://neilweitzel.github.io/xfeeds/nftables.conf
sudo nft -f nftables.confAnything else — pfSense and OPNsense URL tables, MikroTik address lists,
Cloudflare lists, nginx deny maps: point them at high-confidence.txt, or at
high-confidence-v4.txt if the target only understands IPv4.
xfeeds publishes IPv6 as a separate track. It is small and structurally different from the IPv4 feed: 91 prefixes from a single source family, Spamhaus DROPv6.
What the limitation actually is. Those records are promoted on that source's precision alone. So are 1,681 IPv4 records — 44% of the IPv4 high-confidence feed rests on exactly the same basis. The problem with IPv6 is not that the data is weaker; it is that there is no second source, so nothing corroborates it and nothing covers for it if that source degrades. Treat it as a concentration risk, not a quality one. The feed header says so, and says it only while it remains true — the notice is computed from the independence-class count.
The prefixes are wide on purpose. /29 to /48, with no individual addresses. General IPv6 practice is to work at /64 and to treat a /32 as an entire ISP that should almost never be blocked wholesale. DROP is the documented exception: it lists netblocks leased or stolen outright by criminal operations, published for firewall and backbone use, where the whole allocation is the finding.
Applying the IPv6 feed covers 11,488,576 /48 sites, and 18 entries account for 82% of that. Entry count is a poor guide here — review the widest entries before deploying them. The dashboard breaks this down by prefix width.
Other sources do report IPv6, and it is deliberately not published.
Blocklist.de contributes about 431 IPv6 hosts per run. Every one is withheld,
because nothing corroborates them: zero of those hosts fall inside a Spamhaus
DROPv6 prefix. The two sources observe different things — compromised individual
hosts versus criminal-controlled allocations — so they cannot corroborate each
other even in principle. Publishing IPv6 host indicators needs a second
redistributable source that also reports hosts. insights.json reports the
per-source IPv6 observation counts so this stays measurable.
Which file. Single-stack IPv4: high-confidence-v4.txt and iptables.ipset.
Dual-stack: both, or nftables.conf, which has always carried both families in
separate sets.
Pull every 6 hours or less often. The feed is rebuilt on a 6-hour cadence, so polling faster only wastes both our bandwidth.
uv run xfeeds explain 45.33.32.156
uv run xfeeds explain 2001:678:254::1Prints the sources that reported it, their independence classes and weights, the score, and — if it was excluded — which specific rule excluded it. This is the tool for triaging a false-positive report.
Where the upstream publishes a stable listing identifier, it is carried through as
source_reference in all.json and all.csv and shown in the dashboard lookup.
Spamhaus DROP records carry their SBL ticket, whose details are viewable at
check.spamhaus.org by IP, range, or ticket number —
so "why is this listed" can be answered with the upstream's own reference rather
than "a source we trust said so". It is deliberately absent from the plain-text
feeds, which firewalls parse.
- Allowlist last. Applied after every other stage, from live upstreams: Cloudflare, Google Cloud, Googlebot, Bingbot, GitHub, plus static RFC1918/bogons and public resolvers. A failed allowlist fetch is a hard failure, not a warning.
- CIDR width cap. Reject prefixes wider than /22 (IPv4) or /48 (IPv6) unless from Spamhaus DROP. The IPv6 exemption is load-bearing, not incidental: DROPv6 legitimately publishes /29s and /32s because it lists whole criminal-controlled allocations, so every IPv6 entry wider than /48 in the feed is there on that exemption.
- Churn guard. A run that would add or remove more than 25% of the feed fails, opens an issue, and leaves the previous feed in place.
- Redistribution flags enforced in code. Sources marked
redistribute: falseinform scoring and never reach an emitter. - Provenance always. No IP ships without a named source in
all.json. - Staleness detection. A source whose own evidence exceeds 30 days is damped and stops counting toward admission, so a dead upstream is never mistaken for a quiet internet. Past 90 days it is dropped entirely and does not return without a recorded review — see below.
The IPv4 space as one strip. 512 slices of 8.4 million addresses, log-scaled, lowest address on the left. It answers something the feed files cannot: how much of the internet we see activity in at all. Currently 402 of 512 slices. Multicast and reserved space is shaded so an empty tail is not mistaken for a broken chart.
IPv6 gets structure, not a copy of the IPv4 charts. The v6 corpus has zero variance in score, band, source and category, so a corroboration histogram or score distribution would render one bar each — the appearance of analysis with none of the substance. Those are suppressed, and the page lists which ones and why, so a reader can tell "no signal" from "we did not look". What is shown instead is structural and clears the same MIN_CELL threshold used for named ASN cells: entries by prefix width, the /64 subnets each width actually reaches, which /12 of 2000::/3 the listings fall in, and adjacent prefixes under apparent common control. Adjacency is reported, never merged — merging would diverge from what Spamhaus published and destroy the per-entry listing reference.
Reach, not entry count. For IPv6 the bar beside each prefix width encodes address space reached rather than number of entries, which is why the most numerous row (/48, 32 entries) has the shortest bar and the least numerous (/29, 18 entries) carries 82% of the total. A feed that reports "91 entries" and stops has told an operator nothing about what applying it does.
ASN analysis is IPv4-only, because the iptoasn.com enrichment table is. That is now reported as its own figure rather than folded into an unexplained "unenriched" residual.
There is no map, deliberately. The country in an IP-to-ASN table is where the AS number is registered — for a hosting company that is where its paperwork lives, not where traffic came from. A chart headed "listed addresses by country" would put 19,000 addresses on Romania because M247 is registered there, and be the most confidently wrong thing on the page. Address space is the coordinate system this data actually has.
Networks ranked by persistence, not volume. Sorted by how many distinct days a network appeared, because individual addresses churn out within about a week: a big one-day number is an incident, a network seen on nine separate days is a pattern. Every row divides by the address space the ASN announces, which is what separates a disproportionately hostile small network from a merely enormous one — DigitalOcean at 716 address-days per million announced against ChinaNet at 18.
30- and 60-day windows are real, not padded. bruteforceblocker publishes about a month of dated history and ipthreat about ten days; both are preserved in source_last_reported and drive feeds/asn-history.json. Those dates deliberately do not touch last_seen, so upstream history cannot silently restate confidence scores. Windows shorter than the history available say so on the page.
Statistics are computed over every source, including those whose licences forbid republishing their addresses — a count is a derived fact, not an extract, so this is where GreenSnow and ThreatFox appear by name against a number. Two rules keep it that way, both enforced by tests: no individual address is ever emitted, and cells below 5 addresses fold into an unnamed bucket.
ASN data from iptoasn.com (Public Domain, PDDL v1.0). No geography is published anywhere, including as a table column: the country in an IP-to-ASN table is a registration detail, not a location.
Most public IP feeds that forbid something forbid commercial use, not redistribution. Those are two different restrictions, and treating them the same was costing real coverage.
Primary — feeds/ |
Non-commercial — feeds/noncommercial/ |
|
|---|---|---|
| Licence | Source terms only; no commercial restriction | CC BY-NC-SA 4.0 |
| Use at work, or in a paid product | Yes | No |
| Published addresses (last run) | 4,204 | 5,281 |
| Extra sources published in full | — | Turris Sentinel, DShield |
If you are running a home lab, a personal server, a school or a charity, take the non-commercial tier: it sees about a quarter more. If you are at a company or building anything anyone pays for, take the primary feed. The distinction is real, not a formality — feeds/noncommercial/LICENSE.txt states it plainly and every file in that directory leads with a banner.
The tier is built by a second scoring pass rather than by filtering the first, because which sources are publishable changes what counts as corroboration, which changes the confidence bands.
One rule in that second pass is not obvious: CC BY-SA and CC BY-NC-SA cannot be combined in one file. ShareAlike forbids applying additional terms to an adaptation, and NonCommercial is an additional term. So ipthreat.net data, which carries ShareAlike but not NonCommercial, is excluded from the non-commercial tier by noncommercial_sources(), with a test that asserts it never leaks in. See ADR-041.
Every published file names its contributing sources and carries their terms in the header. Licence conclusions in sources.yaml quote the upstream text verbatim rather than paraphrasing it — a paraphrase is how we came to spend a week not republishing a feed that was always fine to republish, and to keep republishing two that were not (ADR-039, ADR-040).
Threat data provided by IPThreat at https://ipthreat.net. Spamhaus DROP data is © The Spamhaus Project SLU and its attribution travels with the data as their terms require. The Turris Sentinel greylist is CC BY-NC-SA 4.0 and appears only in the non-commercial tier.
Only while a source still reports it. An address leaves the feed on the next run after its last source delists it — there is no independent retention window, and nothing is held for 30, 60 or 90 days.
This is deliberate. Around 86% of blocklisted addresses are short-lived offenders averaging about a week of activity, and dynamically allocated addresses are typically delisted upstream within three days (AsiaCCS 2019, IMC 2020). Holding entries longer mostly accumulates addresses that have since been reassigned to somebody innocent: reused addresses have been measured sitting in public blocklists for up to 44 days, affecting as many as 78 legitimate users. If you are going to drop traffic on this feed without reviewing it, that is the failure mode that matters.
What we do keep is history. first_seen survives delisting and re-listing, so a repeat offender is visible as one, and ttl_days lets a source's vote decay gracefully rather than vanish if that source misses a fetch (ADR-037).
xfeeds/
├── README.md
├── sources.yaml # source registry — the only place feeds are configured
├── allowlist.txt
├── pyproject.toml / uv.lock
├── docs/DECISIONS.md # architecture decision record
├── src/xfeeds/
│ ├── cli.py # xfeeds run | validate | dashboard | explain <ip>
│ ├── collectors/ # plaintext, netset, json, spamhaus, abuseipdb, threatfox
│ ├── normalize.py
│ ├── score.py # independence-class weighting
│ ├── filters.py # allowlist, CIDR caps, redistribute flags
│ ├── emit/ # txt, csv, json, stix, misp, nftables, ipset, cloudflare
│ └── report.py
├── tests/ # recorded fixtures per source; no network in unit tests
├── feeds/ # generated output — committed, served via Pages
├── legacy/IP_List # original 2022 ThreatX SOC snapshot, frozen
└── .github/workflows/
├── update-feeds.yml # cron */6h + workflow_dispatch
├── pages.yml # publish feeds/ to GitHub Pages
└── ci.yml # ruff, mypy, pytest on PR
update-feeds.yml:
- Triggers: hourly at
:17andworkflow_dispatch, entirely inside GitHub. Both use the same guard: at most four attempts per UTC calendar day and at least 5h30 between attempts, a 30-minute tolerance around the six-hour target. - Steps: checkout latest main, restore dependencies/cache, evaluate guard, commit
a quota reservation, then run the pipeline, commit feed output, and deploy Pages.
A failed attempt still consumes its reserved slot; a rejected reservation push
aborts before any feed is fetched.
forcebypasses churn only, not the guard. - Secrets, all optional — absent keys degrade to unauthenticated sources rather than failing:
ABUSEIPDB_API_KEY,THREATFOX_AUTH_KEY,GREYNOISE_API_KEY. All three are configured; GreyNoise is enrichment only and a failed lookup caps nothing (ADR-049). - One concurrency group serializes refreshes without cancelling an active run.
- On failure or churn trip: open/update an issue with the run report.
The 6-hour cadence is set by the tightest external constraint: AbuseIPDB allows 5 blacklist calls/day on the free tier, and Spamhaus requires automated fetches at least an hour apart.
AbuseIPDB resets its APIv2 quota at 00:00 UTC,
not on a rolling 24-hour window. feeds/refresh-attempts.json persists the recent
attempt reservations so failed refreshes and lost caches cannot reset the guard.
The cap is on pipeline attempts, not individual HTTP retries inside a collector.
More trigger opportunities do not guarantee punctual refreshes: GitHub can still
delay or drop scheduled jobs. No external scheduler or paid monitor is required.
A push made with GITHUB_TOKEN does not start another workflow — GitHub blocks that to prevent recursion. So the refresh commit can never trigger pages.yml by push, and there are three deliberate paths instead:
| Path | Trigger | Role |
|---|---|---|
update-feeds.yml → deploy-pages |
inside the refresh run | Primary. Publishes every scheduled refresh. |
pages.yml |
workflow_run on the producers completing |
Fallback. Publishes if the primary failed or was cancelled. |
refresh-dashboard.yml |
push to src/xfeeds/dashboard.py or insights.py |
Re-renders the page when the generator changes, then pages.yml publishes it. |
Both publishers share the pages concurrency group, so they can never deploy at once and land an older artifact last. pages.yml compares the committed feeds/index.html against the live one by hash and skips when consumers are already current, so the fallback costs nothing on a healthy run.
Why the fallback exists: on 2026-08-17 a refresh committed new feeds and its deploy job then died fetching an action from codeload (HTTP 429). main was fresh, Pages was a build behind, and nothing could retry until the next six-hourly run.
uv run xfeeds dashboardRebuilds feeds/index.html and feeds/lookup.json from the feed files already on disk. No network access, and it spends none of the rationed AbuseIPDB budget that xfeeds run needs — so this is the way to pick up a presentation change.
Two workflows exist to make an unattended outage visible, because update-feeds.yml
cannot report on itself failing to run.
heartbeat.yml has a best-effort schedule every 6 hours at :47; GitHub can
delay execution. It checks committed feeds/manifest.json and the live Pages
manifest independently, classifying each as fresh, stale, invalid, or unreachable.
The freshness threshold remains 24 hours. Unknown age is never reported as a
stale age or as proof that deployment stopped.
Transient HTTP failures (408, 429, 500, 502, 503, 504) and transient network errors get up to three attempts with 5s/10s backoff. Each request has a 10s connection timeout, 30s total timeout, and 35s outer process deadline, bounding request/backoff time to 120s. Permanent HTTP/TLS errors and successfully fetched but invalid or stale manifests alert without retrying. Persistent failures still fail the workflow; retries do not weaken the freshness gate.
The incident includes separate states and ages, every request attempt's HTTP status/transport outcome, and recent refresh runs. A fresh repository with a stale public manifest points to deployment or CDN propagation; an unreachable public manifest means availability failed and its freshness is unknown. One serialized incident is updated rather than duplicated and closes automatically only when both manifests are valid and fresh. Existing legacy alerts are also recognized and closed on recovery. See ADR-072.
keepalive.yml — weekly, and normally does nothing at all.
GitHub disables scheduled workflows in a public repository after 60 days without
repository activity. Healthy operation commits about four times a day, so that clock
never gets near expiring. The failure it guards against is the compounding one:
historically, update-feeds.yml committed only when a run succeeded, so a persistently
failing pipeline could stop committing and eventually have its schedule disabled, turning
"broken and retrying" into "broken and not even trying", which needs a manual
re-enable to recover from. v1.0.1 also commits quota reservations before attempts;
the heartbeat still measures the actual manifest, not those reservation commits.
If no commit has landed in 14 days, keepalive commits a
timestamp to .github/keepalive.txt and opens an issue making clear that it
protected the schedule and fixed nothing.
What this does not cover. Both are self-monitoring. If every scheduled workflow
in the repository were disabled, neither would run to tell you. keepalive.yml
makes that situation very unlikely rather than impossible. Genuine out-of-band
monitoring means something outside GitHub polling
https://neilweitzel.github.io/xfeeds/manifest.json and checking generated_at.
Tagged releases and the change history live in CHANGELOG.md.
Versions describe the pipeline and its published contracts — feed paths, record schema, and manifest fields — not the feed contents, which are regenerated every six hours. Feed URLs are stable and served from GitHub Pages, so pinning a tag does not change what a consumer fetches; it only tells you which pipeline produced the contracts you are integrating against.
v1.0.0 is the first stable release, dated 2026-10-05. It promotes the rc.7
pipeline with the reporting-only correction in PR #55, observed unchanged since
2026-09-02. No source admission, scoring, expiry, or licensing behavior changes
as part of promotion. The release evidence and separately versioned operational
follow-up are recorded in docs/RELEASE_2026-10-05.md.
v1.0.1 is the current operational patch: GitHub-only scheduling hardening and
quarterly source reviews. It does not change source admission or scoring, and
does not claim the new scheduler completed the September burn-in.
v1.1.0-rc.1 (2026-10-08) opens the v1.1.0 window. It admits six sources from the
October review (ReportedIP, jacobrakai, and Carpathian as admitting classes; Sblam
and SANS ISC sources/attacks as voting-only; ISC research-scanner labels as a
benign cap) and reports what every class contributes. v1.1.0-rc.2 (2026-10-09)
applies the scanner caps to every published tier (ADR-071). The current candidate,
v1.1.0-rc.3 (2026-10-30), hardens heartbeat retries and diagnoses (ADR-072).
Per the lifecycle policy, this workflow change restarts the candidate window at
the first scheduled refresh on rc.3, not at a manual heartbeat verification.
v1.0.1 remains the stable release until v1.1.0 is promoted.
The promotion steps are enumerated in
docs/RELEASE_CHECKLIST.md.
band is authoritative for policy decisions: use high for your high-confidence
policy and medium for monitoring or your chosen lower-confidence policy.
score ranks evidence strength within a band; it is not a threshold across bands.
Do not replace the band with a rule such as score >= 80.
For example, an indicator admitted by two fresh redistributable classes can be
upgraded to high by a third, restricted class. Its numerical score can still be
lower than a medium indicator supported by two higher-weight classes. That
inversion is intentional: corroboration structure determines the band, while
weights and recency affect the score. Restricted evidence cannot admit a record
or expose its source identity in published records. See ADR-040 and ADR-053 in
docs/DECISIONS.md.
Phase 2a — aggregation MVP ✅ shipped
- Source registry, collectors and parsers for every voting class
- Live allowlist fetching, CIDR caps,
redistributeenforcement - Independence-weighted scoring, recency decay, ageing with persisted state
- All output formats, manifest, rolling history
- Churn guard, run report,
xfeeds explain - 6-hour scheduled refresh, self-committing, Pages dashboard
- 52 tests, no network access in the suite
Phase 2b — enrichment
- AbuseIPDB and ThreatFox collectors (quota-aware; keys configured)
- ASN and country annotation
- GreyNoise benign-scanner capping over the whole published feed. RIOT itself is a paid add-on and is not obtainable — see ADR-049
Phase 2c — enforcement integrations
- Reference
fail2banaction and Cloudflare IP Access Rules sync - False-positive PR workflow against
allowlist.txt - Source health monitor with automatic issue filing
Phase 3 — original telemetry
- Honeypot collectors feeding xfeeds a first-party independence class, restoring the original-research character of the 2022 dataset without a commercial WAF.
- Not a replacement for a commercial threat intel platform.
- IPs only in Phase 2 — no domains, URLs, or hashes.
- No paid data sources.
- No redistribution of any feed whose licence forbids it.
- No TAXII server.
Open an issue to propose a source — include URL, format, cadence, licence, and what makes it independent of the existing classes. A source that mirrors one already ingested adds no value and will be class-pinned rather than merged.
False-positive reports are triaged first. Include the IP, why it is legitimate, and observable evidence.
If xfeeds informs published work — a paper, a report, a talk, or another tool — please cite it. Machine-readable metadata lives in CITATION.cff, so GitHub renders a Cite this repository button with pre-formatted APA and BibTeX in the sidebar.
Cite the concept DOI in prose; it always resolves to the latest published version:
Weitzel, N. (2026). xfeeds: an independence-aware public threat intelligence feed. Zenodo. https://doi.org/10.5281/zenodo.22045733
Cite the version DOI where exact reproducibility matters: the stable
v1.0.0 software archive is
10.5281/zenodo.23163156, published
2026-10-05 under the same concept DOI. It contains the exact tagged software,
excluding generated feeds, legacy data, and upstream-response fixture payloads.
Use the GitHub tag for the complete fixture-based test suite. The operational
patch v1.0.1 is available as a GitHub release, not mislabeled as that archive.
If you are citing feed contents rather than the pipeline, cite the generated_at timestamp from the relevant feeds/manifest.json as well — feeds are regenerated every six hours and are not themselves versioned by tag.
See docs/CITABILITY.md for the archival plan and docs/RELEASE_CHECKLIST.md for the v1.0.0 promotion procedure.
Code: MIT — see LICENSE.
Feed data is licensed separately and per tier, because upstream source terms are not uniform and cannot be relicensed by aggregation:
| Tier | Path | Terms |
|---|---|---|
| Primary | feeds/ |
Compiled from every usable public source, including publishers that grant no explicit reuse licence. Provided as-is; absence of a prohibition is not a grant. |
| Clean provenance | feeds/clean/ |
Only sources with a written, named licence affirmatively permitting redistribution including commercial use. Smaller by design. Per-source terms in feeds/clean/LICENSE.txt. |
| Non-commercial | feeds/noncommercial/ |
CC BY-NC-SA 4.0 share-alike material, republished under the same licence in a separately marked tier. See feeds/noncommercial/LICENSE.txt. |
Upstream licences are recorded per-source in sources.yaml and in every feeds/manifest.json. Feeds derived from Spamhaus DROP carry Spamhaus attribution as required. If your use needs to survive a legal review, use feeds/clean/.
Several sources publish freely and say nothing about reuse; a few say two things that do not quite agree. The policy is to read each licence as written, record that reading, and publish on it — not to hold data back pending correspondence, and not to assume a grant that was never made.
Three safeguards sit underneath that, and none of them depend on interpretation:
redistribute: falseis enforced in code with a test, not in documentation.- Publishing an address requires two independent classes we are permitted to republish, so a restricted source can never put one into a feed.
feeds/clean/contains only sources with a written, named grant — for anyone who cannot rely on a reading.
If you maintain a source and disagree with how we have read your licence, open an issue or a pull request and it will be actioned. Removal or restriction on request, no argument. That line appears in the header of every published feed file, because the publisher is the person most likely to disagree and least likely to be reading this page. See ADR-060.