Topic pages

A topic page is a way of monitoring news content on a particular topic without having to express the topic as a strict boolean query. Instead, you describe the topic with a set of weighted concepts, keywords, categories, sources and locations. Each article or event then receives a score based on how many of the specified conditions it matches and how important they are, and only the results above a chosen threshold are returned - ordered by how well they match the whole topic.

Topic pages can be created and managed interactively on the Event Registry website, or programmatically using the TopicPage class described here.

Creating a topic page

from eventregistry import *
er = EventRegistry(apiKey = YOUR_API_KEY)

topic = TopicPage(er)

# describe the topic with weighted concepts and keywords.
# weights are typically in range 1 - 50; higher weight = more important condition
topic.addConcept(er.getConceptUri("renewable energy"), 50)
topic.addConcept(er.getConceptUri("solar power"), 30)
topic.addKeyword("photovoltaic", 20)
# a required condition - all results HAVE TO be annotated with this category
topic.addCategory(er.getCategoryUri("energy"), 20, required = True)
# an excluded condition - results that match it are removed
topic.addKeyword("horoscope", 0, excluded = True)

# limit the results to English and German articles from the last 3 days
topic.setLanguages(["eng", "deu"])
topic.setMaxDaysBack(3)

# the article has to reach this total weight to be included among the results
topic.setArticleThreshold(50)
topic.setEventThreshold(50)

The methods that can be used to define the topic are:

• addConcept(conceptUri, weight, label = None, conceptType = None, required = False, excluded = False)
• addKeyword(keyword, weight, required = False, excluded = False)
• addCategory(categoryUri, weight, required = False, excluded = False)
• addSource(sourceUri, weight, excluded = False)
• addSourceLocation(sourceLocationUri, weight, excluded = False)
• addSourceGroup(sourceGroupUri, weight, excluded = False)
• addLocation(locationUri, weight)

Setting required = True means that every result has to match this condition; setting excluded = True means that results matching the condition are removed. The corresponding clearConcepts(), clearKeywords(), clearCategories(), clearSources(), clearSourceLocations(), clearSourceGroups() and clearLocations() methods remove the previously added conditions.

Additional filters can be set with:

• setArticleThreshold(value) / setEventThreshold(value): the minimum total weight that an article/event has to reach to be included among the results
• setLanguages(languages): limit the results to the given language(s)
• setMaxDaysBack(maxDaysBack): the maximum allowed age of the results in days (default: 7)
• setDataTypes(dataTypes): which data types to search - news (default), pr, blog or a list of these
• setSourceRankPercentile(startPercentile, endPercentile): limit the results to sources of a certain ranking
• setSentiment(minSentiment, maxSentiment): limit the results by sentiment
• setArticleIsDuplicateFilter(value), setArticleHasDuplicateFilter(value), setArticleHasEventFilter(value): control the handling of duplicates and of articles without a known event
• restrictToSetConceptsAndKeywords(restrict), restrictToSetCategories(restrict), restrictToSetSources(restrict), restrictToSetLocations(restrict): if set to True, the results have to match at least one of the specified conditions of that type

Getting the results

Once the topic page is defined, use getArticles() and getEvents() to retrieve the matching content:

# get the first page of matching articles, sorted by how well they match the topic
res = topic.getArticles(page = 1, count = 100, sortBy = "rel")
for art in res.get("articles", {}).get("results", []):
    print(art)

# get the matching events
res = topic.getEvents(page = 1, count = 50, sortBy = "rel")

The methods accept the following arguments:

getArticles(page = 1,
    count = 100,
    sortBy = "rel",
    sortByAsc = False,
    returnInfo = None,
    uri = None,
    onlyAfterTm = None,
    maxDaysBack = None)

getEvents(page = 1,
    count = 50,
    sortBy = "rel",
    sortByAsc = False,
    returnInfo = None,
    uri = None,
    onlyAfterTm = None,
    maxDaysBack = None)

• page: which page of the results to return (starting from 1).
• count: the number of articles (at most 100) or events (at most 50) to return.
• sortBy, sortByAsc: the order of the results. The options are the same as on the searching for articles and searching for events pages; the default rel returns first the results that match the topic the best.
• returnInfo: what details should be included in the returned information. See details.
• uri: the URI of a topic page saved in your Event Registry account. If given, the content that matches the saved topic page is returned, instead of the content that matches the definition in this object (the definition is then not sent to the server). There is no need to call loadTopicPageFromER() first.
• onlyAfterTm: return only the articles that were added to Event Registry (or the events that were added or updated) after the given time. The value can be a datetime or a string in the 'YYYY-MM-DDTHH:MM:SS' format, in UTC (a value with a timezone is converted to UTC). Useful for polling for new content - see the example below.
• maxDaysBack: the returned content should not be older than this number of days. If None, the value set in the topic page is used.

The uri, onlyAfterTm and maxDaysBack arguments are available since version 11.0. Other keyword arguments are sent to the server as they are.

Polling for new content

To repeatedly get only the new articles that match a topic page, remember the time of the previous call and pass it as onlyAfterTm in the next one. The example uses a topic page saved in your account, identified by its URI:

import datetime, time
from eventregistry import *
er = EventRegistry(apiKey = YOUR_API_KEY)

topic = TopicPage(er)
topicUri = "your-topic-page-uri"
# start with the articles from the last hour
lastTm = datetime.datetime.now(datetime.timezone.utc) - datetime.timedelta(hours = 1)
while True:
    callTm = datetime.datetime.now(datetime.timezone.utc)
    res = topic.getArticles(uri = topicUri, onlyAfterTm = lastTm, sortBy = "date", count = 100)
    for art in res.get("articles", {}).get("results", []):
        print(art["dateTime"], art["title"])
    lastTm = callTm
    # wait 10 minutes before asking again
    time.sleep(600)

If more than 100 new articles can appear between two calls, also request the next pages (page = 2, ...) with the same onlyAfterTm.

Saving and loading topic pages

The topic page definition can be saved and restored in several ways:

# save the definition to a python dict / file
definition = topic.saveTopicPageDefinition()
topic.saveTopicPageDefinitionToFile("myTopic.json")

# restore it later
topic2 = TopicPage(er)
topic2.loadTopicPageFromDefinition(definition)
# or
topic2.loadTopicPageFromFile("myTopic.json")

If you have created a topic page in your account on the Event Registry website, you can load its definition by its URI, or (since version 11.0) get its content directly, without loading the definition:

topic = TopicPage(er)
# load the definition of the saved topic page into the object (it can then be changed)
topic.loadTopicPageFromER("your-topic-page-uri")
res = topic.getArticles(page = 1)

# or just get the content of the saved topic page
res = TopicPage(er).getArticles(uri = "your-topic-page-uri", page = 1)

The list of topic pages owned by your account can be retrieved using the TopicPages class:

topicPages = TopicPages(er)
for topicInfo in topicPages.getMyTopicPages():
    print(topicInfo.get("uri"), topicInfo.get("label"))