Alert-hub.org data extraction with Python

Alert Hub exposes its exportable alert archive as an authenticated, ordered record stream. Starting at cursor zero builds a local copy. Retaining the last committed cursor and polling again keeps that copy current.

This is the normative Python recipe. It preserves the raw CAP XML and provides an explicit post-processing hook without prescribing what a downstream system should derive from it.

What the endpoint returns

The request is:

GET /api/v1/entryLog?sinceId=0&pageSize=500
Authorization: Bearer <short-lived machine JWT>
Accept: application/json

The current service base URL is https://ah-node-v2.api.alerting-apps.net. Use the base URL supplied with an approved connection if it differs.

Each response contains ascending records whose numeric id is strictly greater than sinceId. Each record includes an ownerFeed.code, an optional source link, and entry: the stored raw CAP XML.

The response sinceId repeats the requested cursor. It is not the next cursor. After durably storing a page, use the final record’s id. An empty page means only that the consumer is caught up at that moment.

1. Install the Python reference client

The maintained example is in the hub-extract repository, separated from the existing Java implementation:

git clone https://gitlab.com/alert-hub-org/alertingservicesplatform/hub-extract.git
cd hub-extract/python
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -e .

Python 3.11 or newer is required.

2. Sign in and create the connection

1. Sign in to the Alert Hub workbench.
2. Open API → Connections. This page contains only connections owned by your account context.
3. Select New archive connection.
4. Enter a durable name, such as Production archive reader.
5. Describe the intended use and create the connection.

The connection is owned by the signed-in organisation when the identity token contains an organisation id; otherwise it is owned by the signed-in user. Machine credentials cannot use these management endpoints.

3. Generate the identity locally

Run:

alert-hub-extract identity create \
  --output-dir ~/.config/alert-hub/archive-reader \
  --credential-key archive-reader-prod \
  --display-name "Production archive reader"

This creates two files:

• identity.private.json: the RSA private identity used to sign requests;
• registration.public.json: the public registration bundle accepted by the workbench.

The command creates the private file with mode 0600 and refuses to overwrite either file. Do not upload, email, paste, or commit identity.private.json. Back it up as a production secret. Alert Hub neither needs nor accepts it.

Check the public file before uploading it:

alert-hub-extract identity validate-public \
  ~/.config/alert-hub/archive-reader/registration.public.json

The command rejects RSA private parameters and prints the public-key fingerprint.

4. Register the public key and prove possession

In the selected connection:

1. choose registration.public.json—not the private file;
2. select Register public key;
3. compare the displayed fingerprint with the one printed locally;
4. copy the one-time proof command displayed by the workbench;
5. run it on the machine holding the private identity;
6. paste the resulting proof JWT into the workbench;
7. select Submit for approval.

The local command has this form:

alert-hub-extract identity prove \
  --identity ~/.config/alert-hub/archive-reader/identity.private.json \
  --connection-id CONNECTION_ID \
  --challenge ONE_TIME_CHALLENGE

The proof is short-lived and bound to the connection, credential key and one-time challenge. It shows possession of the private key without disclosing it.

Registration does not itself grant access. An Alert Hub administrator reviews the intended use, proof and fingerprint, then approves or rejects the archive-read capability and assigns a rate profile. The connection page shows that decision. No certificate or private key should be sent by email.

The administrator uses a separate Admin → Connection requests page. Consumer registration and administrator approval are not mixed in the same workspace.

5. Test approved access

After the connection shows ACTIVE and ARCHIVE_READ: APPROVED, run:

alert-hub-extract test \
  --base-url https://ah-node-v2.api.alerting-apps.net \
  --identity ~/.config/alert-hub/archive-reader/identity.private.json \
  --page-size 1

A successful response begins Authenticated successfully. A 403 normally means approval, capability, suspension or expiry state—not that another role should be placed in the locally signed token. Roles and limits are server assigned.

6. Backfill and keep the copy current

The complete command is:

alert-hub-extract sync \
  --base-url https://ah-node-v2.api.alerting-apps.net \
  --identity ~/.config/alert-hub/archive-reader/identity.private.json \
  --database ./alert-hub-archive.sqlite3

On a new database the stored cursor is zero, so the first request is exactly:

/api/v1/entryLog?sinceId=0&pageSize=500

The client stores every page and advances its cursor in one SQLite transaction. Restarting the same command resumes after the last committed record. Once caught up it keeps polling, because this is an ongoing stream rather than a one-time archive file. Stop it with Ctrl-C; committed work remains safe. Add --once only when a bounded catch-up run is intentional.

The client retries transport failures, HTTP 408, 429, and server errors with bounded backoff. It obeys Retry-After. Do not work around rate limits by creating parallel connections; request a reviewed BACKFILL profile when initial catch-up needs more capacity.

The equivalent Python hook

The CLI uses the same public classes as application code. This example receives each source record before the page cursor is committed:

from pathlib import Path
from xml.etree import ElementTree

from alert_hub_extract import EntryLogClient, SQLiteArchive, synchronize


def post_process(entry):
    """Return any JSON-serializable derivative needed by the local system."""
    root = ElementTree.fromstring(entry.raw_xml)
    return {
        "identifier": root.findtext("{*}identifier"),
        "sender": root.findtext("{*}sender"),
        "source_feed": entry.owner_feed_code,
    }


client = EntryLogClient(
    "https://ah-node-v2.api.alerting-apps.net",
    str(Path("~/.config/alert-hub/archive-reader/identity.private.json").expanduser()),
    page_size=500,
)
archive = SQLiteArchive("alert-hub-archive.sqlite3")

synchronize(
    client,
    archive,
    process_entry=post_process,
    continuous=True,
    poll_interval_seconds=60,
)

If post_process raises an exception, that page is not committed and the cursor does not advance. The source envelope and exact XML remain separate from the returned derivative. This gives downstream code a clean interception point while retaining replay and audit options.

For asynchronous pipelines, queues or another database, preserve the same boundary: make output durable and idempotent for every record in the page before committing the final entry-log id. Delivery should be treated as at least once, with entry-log id as the idempotency key.

Accessing raw XML and other output forms

Raw XML is stored unchanged in archive_entry.raw_xml:

sqlite3 alert-hub-archive.sqlite3 \
  'select raw_xml from archive_entry where entry_log_id = 123456789;'

The original non-XML fields are retained in envelope_json; post-processing results are stored separately in derived_json.

For a portable record stream, export newline-delimited JSON:

alert-hub-extract export ndjson \
  --database ./alert-hub-archive.sqlite3 \
  --output ./alert-hub-archive.ndjson

Each physical line is one JSON record. Embedded XML newlines are escaped. Applications may instead iterate SQLiteArchive.records() and write to their own sink. The reference client deliberately does not choose that sink or downstream schema.

Operational rules

• Keep the private identity in a secret store and the host clock synchronized.
• Never advance a cursor after partial or uncertain output.
• Reject non-increasing ids or changed replayed records; the reference client does both.
• Treat CAP Update and Cancel messages as records in the stream.
• Monitor the workbench connection statistics and throttled-request count.
• Rotate keys through the same connection rather than creating a new identity owner.

The endpoint supplies the complete, unfiltered entry-log stream. Its record shape requires a Snowflake id, source code and stored raw XML; malformed storage rows missing those required values cannot form an entry-log record.

The implementation is intentionally simple: an ascending cursor, exact source XML, local transactional storage and a user-owned approved connection. That same registration model is also intended for feed publishers; only the approved capability differs.