aveloxis-cve

A sidecar for Aveloxis that analyses commits on every branch of a repository, and complexity over time.

Some projects fix CVEs on separate maintenance branches (release, stable and LTS backports). Aveloxis walks only the default branch, so those fixes never reach its tables. aveloxis-cve records, for a chosen subset of the repositories Aveloxis already collects:

• which commits are on which branch (branch_repos, branch_commits);

• rolling complexity: sliding 60-day windows advanced one month at a time, with a full-repository scc snapshot and a changed-files-only analysis per window (repo_labor_rolling).

Zero coupling

• Aveloxis does not know aveloxis-cve exists.

• aveloxis-cve reads aveloxis_data, aveloxis_ops and aveloxis_scan, and writes only its own schema, aveloxis_cve.

• Nothing in aveloxis_cve references an Aveloxis object (no foreign keys, views or functions). So no object of the sidecar's can stand in the way of Aveloxis's DDL, and DROP SCHEMA aveloxis_cve CASCADE removes the sidecar completely.

• init-db and migrate wait for their migration lock by polling, never with a blocking lock request that would stall CREATE INDEX CONCURRENTLY elsewhere in the database.

• aveloxis-cve keeps its own, fully independent clones, made from the forge URL Aveloxis records. It never reads, seeds from or writes Aveloxis's clone directory.

• A repository must already be in Aveloxis, with a completed collection, before aveloxis-cve will analyse it.

Requirements

• A complete Aveloxis install that has finished at least one collection cycle for every repository you track. aveloxis-cve doctor checks this.

• Go 1.26+, git, and scc (for the complexity analysis).

Install

go install github.com/aveloxis/aveloxis-cve/cmd/aveloxis-cve@latest

A DBA creates the least-privilege role once, as a superuser, in the Aveloxis database (see the header of sql/roles.sql). The role can read aveloxis_data, aveloxis_scan and aveloxis_augur_data, except chaoss_user, which holds password hashes. Default privileges cannot exclude a table by name, so if Aveloxis creates chaoss_user after the script ran, re-run the script; doctor fails until you do. In aveloxis_ops it can read only schema_meta and collection_queue; the rest of that schema holds forge tokens, password hashes and API keys, so the role gets no wider grant there. It owns aveloxis_cve.

psql -d aveloxis -v ON_ERROR_STOP=1 \
     -v cve_role=aveloxis_cve \
     -v cve_password="$AVELOXIS_CVE_DB_PASSWORD" \
     -v aveloxis_owner=aveloxis \
     -f sql/roles.sql

The script refuses to create the role with an empty cve_password (for example, with AVELOXIS_CVE_DB_PASSWORD unset). To use another authentication method (peer, cert), create the login role first; the script then leaves its authentication alone. Re-running it needs no password.

Then:

cp aveloxis-cve.example.json aveloxis-cve.json   # edit; see docs/configuration.md
chmod 600 aveloxis-cve.json
aveloxis-cve init-db                             # creates aveloxis_cve tables (first installation)
aveloxis-cve doctor                              # read-only health check
aveloxis-cve repos add --repo-group-id 3         # or --repo-id / --url / --url-file / --query
aveloxis-cve repos add --url-file urls.txt       # one URL per line; - reads stdin
aveloxis-cve repos list

After installing a new release of aveloxis-cve, apply its schema changes:

aveloxis-cve migrate

Commands

Command	What it does
version	Print the version.
init-db	First installation: create the aveloxis_cve schema and apply every migration. It never touches Aveloxis's schemas.
migrate	After installing a new release: apply its schema changes to the installed aveloxis_cve schema, as aveloxis migrate does for Aveloxis. It never installs: on a database without the installed schema (its migration ledger) it fails, says to run init-db first, and creates nothing. Like init-db, it refuses a schema a newer aveloxis-cve migrated. doctor and the commands that find the schema older than the binary say to run it.
doctor	Read-only checks: tools, directories, the connection, the Aveloxis columns aveloxis-cve reads, write privileges it should not have, the schema version, and the readiness of each tracked repo.
repos add	Track repositories, selected by --repo-id, --url, --url-file (a file of URLs, one per line, or - for stdin; repeatable; blank lines, spaces and tabs around a URL, and Windows line endings are ignored), --repo-group-id, or --query (SQL returning repo_id, run read-only). An unknown id or URL, or an empty selection, adds nothing; for --url-file the error names the file and line.
repos list / enable / disable	Manage the tracked subset. enable and disable change all the given ids or none.
sync [--repo-id N]	For each tracked repo that is ready in Aveloxis: clone it (first run) or fetch it into clone_dir (later runs). It then records branches, commits, full branch membership with landed_at, contributor identities, patch-ids, and every commit's files counted as Aveloxis counts them (commit_facade_files). Repos Aveloxis has not finished collecting are skipped, with the reason recorded.
labor [--repo-id N]	On each synced repo's default branch, computes every complete sliding window not yet stored, or marked due (labor.window_days, default 60, advanced every labor.step_months, default 1). Each window records a full-repository scc snapshot at its end, with Aveloxis's repo_labor columns, and the files changed by commits that landed in the window, with complexity at start and end. scc runs only on file contents it has not measured before. --recompute requests that the selected repositories' windows of the current window settings be computed again, before any repository runs; each run applies the request to the default branch it reads and recomputes those windows, each replacing its stored one in one transaction. A repository skipped as busy, or a run that fails part-way, is finished by the next labor or run; so is a run already under way in another process, which may have measured before the request (other settings' windows are kept). --retry-uncounted also re-measures files scc recorded as not counted, and recomputes every enabled repository; it refuses to start while another labor or run process computes windows, and labor runs started while it runs skip every repository as busy (the scc cache is shared by every repository).
run [--repo-id N]	sync, then resolve (unresolved authors only), then labor, for each repository whose sync succeeded. Sync resolves the authors of the commits it ingests; the resolve step retries those it could not match, as Aveloxis's contributor data improves. It runs before labor, so the windows labor computes already count the authors it resolves. A repository whose sync failed gets no resolve or labor run; a failed sync, resolve or labor run makes the command fail. One Aveloxis has not finished collecting, or one another aveloxis-cve process is working on, is skipped. An interrupt (Ctrl-C, SIGTERM) during sync stops before labor starts.
cves --repo-id N [--cve ID]	When each CVE fix reached each branch (including archived ones), and how long after the first branch. A fix is found by its message, a cherry-pick trailer, or an identical patch-id (manual backports).
resolve [--all]	Re-resolve commit authors to Aveloxis contributors (read-only lookups): unresolved authors, or with --all every author. A repository another aveloxis-cve process is working on is skipped; run retries its unresolved authors later, but nothing redoes --all, so resolve --all fails when it skips one and says to run it again.
status	Per-repo summary: default branch, branches, archived tips, commits, CVE-tagged commits, last sync.
stop	Graceful shutdown. Records a stop request in the database. The commands it stops are every sync, labor, resolve and run of 0.4.32 or later already running (connected) at that moment, on any machine using this database. Each finishes and records the repositories it is working on and starts no other (N repositories not started: stop requested; run also skips its later steps). It exits 0 unless something failed. resolve --all exits non-zero, naming the repositories to rerun, since nothing redoes it later. Commands started after the request, and older binaries, are not affected. stop does not wait for them to exit. Ctrl-C or SIGTERM still interrupts at once instead.

What gets recorded (schema aveloxis_cve)

Table / view	Contents
tracked_repos	The repositories you chose, with their last sync and labor outcomes.
branch_repos	Every live branch and archived tip (refs/cve-archive/<deleted or forced>/...), with the default-branch flag and ahead/behind counts.
commits, commit_files	Every commit on any branch: CVE/GHSA ids, cherry-pick trailers, patch-id, and the Aveloxis contributor it resolves to.
commit_facade_files	Every commit's files counted exactly as Aveloxis counts them in aveloxis_data.commits (cmt_filename, cmt_added, cmt_removed, cmt_whitespace), for commits on every branch. Renames are named old => new, counts are whitespace-adjusted by Aveloxis's walker rules, binary files are 0/0, and a merge or empty commit has one (no files) row. Join branch_commits for Aveloxis-identical counts on any branch. commit_files keeps per-path counts (a rename as a delete plus an add), which the labor windows use.
branch_commits	Full membership: one row per (branch, commit), with landed_at (when the change reached that branch) and on_first_parent.
branch_commit_files (view)	Commits by branch and file.
repo_labor_windows, repo_labor_rolling, repo_labor_rolling_changed	Rolling complexity windows: per-window totals, the full scc snapshot, and changed files (A added / D deleted / M modified / N touched but unchanged / X created and removed). A window with needs_recompute true is stale (a --recompute, a late push or force-push, or an older count scheme) and stays until its replacement is written; filter on NOT needs_recompute for current windows only. A --recompute request still pending (labor_recompute_requests.requested > applied, for a repository busy at the time) marks its windows when the next run gets past its checks (clone, scc, history).
scc_blob_cache	scc results keyed by file content, file name and scc version.
stop_requests	Each aveloxis-cve stop, with its time.
cve_fix_commits, cve_fix_propagation, commit_equivalence (views)	CVE fixes, how they were matched, and when they landed on each branch.

Windows are (start, end]: a commit that lands exactly at start belongs to the previous window. When a sync changes history inside computed windows (a late push of an old-dated commit, a force-pushed default branch), the next labor run recomputes them; the stored windows stay, marked, until each is replaced. Landing times never run backwards along a branch, even when committer clocks did. The first window is [first, end], so the first commit is counted. A snapshot counts the files Aveloxis's repo_labor would: those scc's own walk reaches in a checkout. A tracked file that a .gitignore, .ignore or .sccignore names is not counted (for example, redis's deps/jemalloc/configure), and neither are .gitmodules paths or .git/.hg/.svn directories. A .gitignore, .ignore or .sccignore that is a symlink is read through the link, as in a checkout. A symlinked .gitmodules is not, because git cannot check such a tree out. aveloxis-cve applies that walk to each snapshot's tree without checking it out, using the matcher scc 3.7.0 is built with. A file the walk skips still appears among a window's changed files, but without metrics. Each file the walk reaches is measured alone with scc (--no-gitignore --no-ignore --no-scc-ignore), by its name and content only, so cached measurements carry over between snapshots.

Documentation

docs/ holds the full documentation:

• overview;

• operations;

• configuration;

• the schema diagram;

• data provenance;

• development;

• the analysis guide, with tested SQL for every research question: CVE fixes and their propagation and backports, complexity over the windows, and the two together.

The same pages build into an HTML site with Sphinx (docs/conf.py); see development for the command.

Data

docs/data.md describes every table, view and column of the aveloxis_cve schema: what a row is, where each value comes from, which command writes it and when, the Aveloxis data read, the personal data stored, and how to remove it.

Git version

New clones use git's reftable ref format when git is 2.45 or newer. That lets them hold branch names a filesystem-backed ref store cannot: non-UTF-8 names, and names differing only in case on macOS. Such a clone cannot be read by an older git; if you downgrade git, delete clone_dir and let sync re-clone.

Development

go test ./...                                                # unit tier
AVELOXIS_CVE_TEST_DB="postgres:///postgres?host=/tmp" go test ./...   # + DB tier
gofmt -l . && go vet ./... && staticcheck ./... && golangci-lint run ./...

AVELOXIS_CVE_TEST_DB is a DSN for a superuser on a disposable test cluster, never a production one. CREATEDB is not enough: the sql/roles.sql tests run that script, which requires a superuser. Each DB-tier test gets its own scratch database (cvet_*), which is dropped afterwards. Some tests also create cluster-wide login roles (cvet_role_*) and drop them when they finish; a run killed midway can leave some behind. Those tests log in as the scratch roles, with random passwords, over the DSN's host and port, so the cluster's pg_hba.conf must allow that (password or trust authentication, not peer only). Scratch databases are created with ICU en-US collation, so golden results agree on every machine: the cluster must be PostgreSQL 15 or later, built with ICU, with a UTF-8 or C default locale (LC_COLLATE and LC_CTYPE).

The tests run git with no global or system git configuration, so your own ~/.gitconfig does not change their results. Tests that need reftable refs skip when git is older than 2.45. Under CI ($CI set), a missing tool or a git that is too old fails the run instead.