Add packaging.index

[brettcannon]

This focuses on two things: helping with typing the JSON version of the Simple Repository API and code to help translate the HTML version of the API to the JSON one.

Closes #1088

[brettcannon]

One thing to note about the docs is the TypedDicts only have the docstring as autosphinx can't handle the subclassing necessary to support the required and optional keys as-is (until Python 3.11 is the oldest supported version for Required and NotRequired). If we want to document every key then we will have to do it by hand. Then again, the spec says what's supported which somewhat acts as the documentation anyway.

[brettcannon]

In the future we can add a high-level API like with packaging.metadata, but since this is already useful, I didn't want to hold it up.

[notatallshaw]

A few initial observations applying this to pip and nab and seeing if there any issues:

1. An empty yanking reason loses the yanked status:

   html = '<a href="demo-1.tar.gz" data-yanked="">demo-1.tar.gz</a>'
   f = parse_details("text/html", html, name="demo")["files"][0]
   print(repr(f["yanked"]), bool(f["yanked"]))  # '' False

   HTML uses the attribute's presence to indicate yanking, but JSON uses a truthy boolean or a nonempty reason. Both applications therefore treat this converted entry as unyanked. The empty HTML value needs to become True.

2. Relative links ignore <base href>.

   html = '<base href="https://files.example/"><a href="demo-1.tar.gz">demo-1.tar.gz</a>'
   f = parse_details("text/html", html, request_url="https://index.example/demo/")["files"][0]
   print(f["url"])  # https://index.example/demo/demo-1.tar.gz

   The expected URL is https://files.example/demo-1.tar.gz. Both applications currently honor the base element; conversion changes the download target.

3. An unknown pypi:* meta tag aborts parsing.

   parse_details("text/html", '<meta name="pypi:custom" content="x">', name="demo")

   This raises AttributeError for _handle_custom_content, instead of ignoring the unknown tag and returning an empty file list. parse_list has the same problem. The parser should select a recognized name before validating its content.

4. URL normalization adds work even for absolute URLs.

   from unittest.mock import patch
   from urllib.parse import urljoin
   
   html = '<a href="https://files.example/demo-1.tar.gz">demo-1.tar.gz</a>'
   with patch("urllib.parse.urljoin", wraps=urljoin) as join:
       parse_details("text/html", html, request_url="https://index.example/demo/")
       print(join.call_count)  # 1

   A Python package dependency resolve can have to go through hundreds of thousands of URLs, so this can be very performance sensitive.

[KRRT7]

Following up on @notatallshaw's list: I found a few more HTML parsing issues that don't overlap with his (I'm leaving out the yanked, <base href>, unknown-meta and urljoin points, since he covered those). All of these reproduce at 6dfb116 on every interpreter in the CI matrix (CPython 3.10–3.15, 3.13t–3.15t, PyPy 3.10/3.11).

1. Anchor attributes can call arbitrary _handle_* methods.

   _handle_a_tag turns every attribute name into _handle_<name> and calls it whenever hasattr finds a match. That covers more than the href/data-* handlers:

   parse_details("text/html", '<a href="x.whl" repository-version-content="9.9">x.whl</a>', name="p")["meta"]
   # {'api-version': '9.9'}
   parse_details("text/html", '<a href="x.whl" project-status-content="archived">x.whl</a>', name="p")["project-status"]
   # {'status': 'archived'}
   parse_details("text/html", '<a href="x.whl" a-tag="1">x.whl</a>', name="p")
   # AttributeError: 'str' object has no attribute 'items'

   An explicit mapping from attribute name to handler would fix this, and would also let the # pragma: no cover on that branch go.

2. parse_list crashes on the project status meta tags.

   This isn't just about unknown names. pypi:project-status and pypi:project-status-reason only have handlers on the details parser, so a server that uses the same <head> on both pages breaks list parsing:

   parse_list("text/html", '<meta name="pypi:project-status" content="active"><a href="/p/">p</a>')
   # AttributeError: '_RawProjectListHTMLParser' object has no attribute '_handle_project_status_content'

3. Every </a> becomes an entry, whether or not it's a file or project link.

   parse_details("text/html", '<a name="top">Top</a><a href="x.whl">x.whl</a>', name="p")["files"][0]
   # {'hashes': {}, 'filename': 'Top'}  <- no 'url', which is a required key
   parse_list("text/html", 'Hello</a><a href="/p/">p</a>')["projects"]
   # [{'name': 'hello'}, {'name': 'p'}]

   Only recording anchors that were actually opened, and that have an href in the details parser, would handle both cases.

4. name="" gets past the argument check.

   The guard checks name is None but the HTML branch checks if name:. With name="" and no request_url, it falls through to urlsplit(None), and what happens then depends on the Python version:

   parse_details("text/html", '<a href="x.whl">x.whl</a>', name="")
   # 3.10–3.14: TypeError: a bytes-like object is required, not 'str'
   # 3.15:      returns {"name": "", ...}

   The comment above the typing.cast ("request_url is guaranteed to be a string if 'name' is not set") doesn't hold, so None ends up in urlsplit(). None was never a supported argument there, and CPython 3.15 reworked urllib.parse._coerce_args so that all-empty or None arguments are now treated as str rather than bytes. That's why the same call raises on one version and quietly returns invalid data on another. Making the guard if not name and not request_url: would fix both.

   If request_url is also passed, the name from the URL silently wins over the explicit name="".

Smaller things:

• HTMLParser already unescapes attribute values, so the html.unescape() in _handle_data_requires_python is redundant and unescapes a second time (>=3.8 becomes >=3.8). That's harmless for valid specifiers, but the existing >=3 test passes without it.
• href="...#sha256=a=b" is rejected, but data-core-metadata="sha256=a=b" is accepted as {"sha256": "a=b"}.
• Lowercasing hash names silently drops a key when two differ only in case ({"SHA256": "a", "sha256": "b"} becomes {"sha256": "a"}).
• When one of core-metadata/dist-info-metadata is filled in from the other, both keys end up pointing at the same dict object.
• _RawProjectListHTMLParser.parsing_anchor is declared but never used.
• The parse_details docstring says the 'filename' key is used to make URLs absolute; it's the 'url' key.

---

_{These were found by wtfyd-agent, then reproduced and checked by hand.}