Open News

A lightweight Python toolkit for pulling, filtering, and understanding the news.

Fetch → Extract → Filter → Group → Rank → Summarize → Export

Quick start · Features · Architecture · Docs · Colab examples

---

Why Open News?

Open News is built for real-world news workflows: pulling headlines and article bodies, checking sources, filtering noise, grouping similar stories, and exporting results in a format your app or shell can use.

It is not a hosted reader or a single website scraper. You can use it as a Python library in your own project, or run the CLI directly from a terminal.

At a glance

Area	What it does
News discovery	Fetches category and location-based feeds, or searches by keyword
Source resilience	Tries multiple acquisition paths so one broken source does not kill the whole request
Article extraction	Pulls text, metadata, authors, dates, images, and source details
Website discovery	Finds articles through RSS discovery and fallback crawling
Processing	Filters by language, topic, and domains; merges exact normalized URL copies
Story grouping	Groups similar headlines while keeping distinct articles and their source names
Async workflows	Supports async fetch/search/article retrieval and batch summarization
Summarization	Produces lightweight summaries with optional NLP helpers
Export	Writes Markdown and clean JSON output
CLI	Works well in shell scripts and automation

✨ Features

News acquisition and search

• fetch() for category and location-based news
• search() for keyword search, exclusions, locale, and date ranges
• Streaming helpers for incremental results
• Multi-source fallback logic when one provider is down or noisy

Article extraction and website discovery

• get_article() and get_article_async() for a single URL
• discover_and_get() for RSS-first discovery with same-domain crawling when needed
• Optional Playwright rendering for JavaScript-heavy pages
• Metadata extraction using structured data, Open Graph, and HTML heuristics

Processing and analysis

• Topic and token filtering
• Domain allowlists and denylists
• Exact URL merging and similar-story grouping
• Relevance ranking with BM25, TF-IDF, or a simple fallback
• Similar-story clustering
• Batch and keyword-driven summarization

Async API

The library exposes async entry points such as fetch_async(), search_async(), get_article_async(), batch_summarize_async(), and search_and_summarize_async() for non-blocking workflows.

📦 Installation

Requirements: Python 3.10 or newer.

python -m pip install open-news-api

Optional extras:

# Browser rendering for JavaScript-heavy pages
python -m pip install "open-news-api[js]"
playwright install chromium

# Optional NLP components, including BM25 and summarization helpers
python -m pip install "open-news-api[nlp]"

Install and run with uv

Install the CLI as a standalone tool:

uv tool install open-news-api
open-news --help

Or run it without installing:

uvx --from open-news-api open-news search "artificial intelligence" --country in --limit 10

For use as a dependency in an application:

uv add open-news-api
# Choose optional features as needed:
uv add "open-news-api[js]"
# Or install both extras together:
uv add "open-news-api[js,nlp]"

After adding the js extra, install its browser runtime:

uv run playwright install chromium

Install from a local checkout:

git clone https://github.com/alphap365/open-news.git
cd open-news
uv sync --group dev

uv sync creates and manages the project environment. Run project commands with uv run; there is no need to activate the environment manually.

🧪 Colab examples

For notebook-friendly testing and quick experiments, see the Colab examples. For runnable API demos covering story grouping, filters and ranking, async search, website discovery, and streaming, see the examples index.

The Colab folder includes:

• a ready-to-open Google Colab notebook
• a few small project-specific scripts for fetch/search/export flows
• a simple getting-started path for developers who want to test the library in a notebook environment

The project-level examples can be run with uv run python examples/<script>.py after uv sync --group dev.

The folder contains:

• examples/colab/open_news_colab_demo.ipynb
• examples/colab/fetch_demo.py
• examples/colab/search_demo.py
• examples/colab/export_demo.py

After running uv sync --group dev from the repository root, run an example with:

uv run python examples/colab/fetch_demo.py

🚀 Quick start

Fetch current technology news

from open_news import fetch

articles = fetch(category="tech", max_results=5)
for article in articles:
    print(article["title"], article["url"], article.get("sources", []))
    for related in article.get("similar_articles", []):
        print("  -", related["title"], related["url"])

Search with a region and date range

from open_news import search

articles = search(
    "renewable energy",
    country="in",
    start_date="2026-01-01",
    max_results=10,
)

Use the async API

import asyncio
from open_news import fetch_async

async def main():
    articles = await fetch_async(category="science", max_results=10)
    for article in articles:
        print(article["title"])

asyncio.run(main())

Extract an article

from open_news import get_article

article = get_article("https://example.com/news/story")
print(article["title"])
print(article["text"])

Rank and cluster stories

from open_news import fetch, rank_articles, cluster_articles

# fetch() groups similar stories by default. Each result includes a
# representative article, sources, and similar_articles.
articles = fetch(category="tech", max_results=40)

ranked_stories = rank_articles(
    articles,
    query="artificial intelligence",
    method="auto",
)

for story in ranked_stories:
    print(story["title"], story.get("sources", []))
    for related in story.get("similar_articles", []):
        print("  -", related["title"], related["url"])

# Use cluster_articles() when you need explicit cluster metadata and members.
clusters = cluster_articles(ranked_stories, threshold=0.72, drop_singletons=True)

Export results

from open_news import export

markdown = export.to_markdown(clusters, path="out/stories.md")
json_text = export.to_json(clusters, path="out/stories.json")

Both exporters return the serialized string; if you pass a path, they also write the file to disk.

🧭 Architecture

Open News separates acquisition, extraction, processing, and presentation. The public API coordinates the workflow while specialized modules handle each stage.

_{Open full-size architecture diagram (SVG)}

The main processing path looks like this:

_{Open full-size pipeline diagram (SVG)}

See Architecture for module responsibilities and data flow.

⌨️ Command-line interface

Once installed, the command is open-news:

open-news --help
open-news --version

open-news fetch --category tech --limit 5
open-news search "artificial intelligence" --country in --limit 10
open-news extract https://example.com/news/story
open-news discover https://example.com --limit 10
open-news search-site "climate policy" reuters.com
open-news cluster --query "AI regulation" --limit 30
open-news export input.json --md output.md --json clean.json

Run the same CLI from a project managed by uv with uv run:

uv run open-news --help
uv run open-news fetch --category tech --limit 5 --format json
uv run open-news cluster --query "AI regulation" --limit 30

For scripts, request JSON and pipe it into your preferred tools:

open-news search "AI" --format json | jq '.[].title'
# Or, without separately installing the CLI:
uvx --from open-news-api open-news search "AI" --format json | jq '.[].title'

For release history and version notes, see the changelog.

📚 Documentation

Guide	Contents
Installation	Python setup, editable installs, and optional extras
Python API	Sync and async entry points with examples
CLI reference	Commands, options, and output formats
Parameters	Locale, dates, filters, streaming, and ranking
Architecture	Modules, acquisition flow, and processing pipeline
Release notes	What changed across releases

🧪 Development and tests

The project uses uv and declares development tools in the dev dependency group:

uv sync --group dev
uv run pytest -q
uv run mypy src/open_news
uv build

Use uv run pytest tests/test_cli.py -q to run a focused test file. The repository includes unit, contract, mocked integration, feed-engine, extraction, processing, export, and CLI tests. Network-dependent behavior should be tested separately from deterministic offline tests.

fetch(), search(), search_site(), and discover_and_get() group similar stories by default. Each result is a representative article with a sources list and similar_articles records. Exact normalized URL copies are merged; articles with similar titles but distinct URLs stay in the story group rather than being discarded.

⚠️ Operational notes

• News acquisition depends on external providers and may return different results over time.
• location is a hint, not a guarantee that every result comes from that region. Use a domain allowlist when strict publisher geography matters.
• JavaScript rendering is optional and requires the js extra plus a Playwright browser installation.
• BM25 support is optional. With method="auto", ranking can fall back to TF-IDF when the optional backend is unavailable.
• Extracted article metadata varies by publisher; dates, descriptions, and source attribution are best-effort fields.

License

Open News is distributed under the MIT License. See LICENSE.