Changelog

0.4.54 (unreleased)

Review rounds 9 (of 0.4.52 and 0.4.53: 4 Low, 4 nits), 10 (1 Low, 1 nit) and 11 (1 Low, 4 nits), all fixed. Errata for 0.4.53, whose "pins" and "GitHub-style heading anchors" both claimed more than the build did.

• Low. docs/requirements.txt pinned only the four direct dependencies, so every install took the newest of the rest. On 2026-11-11 that meant markdown-it-py, mdit-py-plugins and soupsieve uploaded the day before, which broke the two-week rule. It also meant CI's docs job could fail with no change in the repository. The file now pins all 34 packages, resolved from uploads before 2026-10-28 (pip install --uploaded-prior-to), and its header says how to refresh it. Installed and built (-W) with Python 3.12 and 3.15.

• Low. The site's HTML ids were docutils's, not GitHub's. MyST's GitHub-style slugs only resolved links at build time. So data.md#commit_files was #commit-files in the site, and CHANGELOG versions were #unreleased, #id1, #id2, which shift with each new version. A transform in docs/conf.py (GitHubAnchors) now makes each heading's GitHub anchor its first id. Rounds 10 and 11, below, complete it.

• Low. Docutils's smartquotes rewrote text: --all rendered as an en dash followed by all, so copying the flag gave an invalid one, and about 1,200 curly quotes appeared. smartquotes = False.

• Low. On GitHub Pages, Jekyll drops _static/ and _sources/, so the site lost its theme and search. sphinx.ext.githubpages now writes .nojekyll, and docs/development.md says so.

• Nit. The documented .venv-docs/ was not gitignored; it is now.

• Nit. The command in docs/conf.py's docstring installed into the system Python, which Homebrew and Debian refuse (externally-managed-environment). It now uses the virtual environment, as docs/development.md does.

• Nit. A missing ToolVersion called sys.exit, and Sphinx then printed only "called sys.exit()". It now raises ConfigError with the message.

• Nit. Erratum for 0.4.52's erratum: "inside the second bullet" meant the second bullet of 0.4.51's list, which is its first nit bullet.

• Review round 10 (1 Low, 1 nit), both fixed:

  • Low. Round 9's fix was a doctree-read hook, which ran after Sphinx had already recorded the old ids for the table of contents. So the sidebar still linked CHANGELOG versions as #id1, #id2 and so on: 84 entries on 6 pages. Links within the site also kept docutils ids, because MyST resolves them from its own table of slugs. The fix is now a docutils transform, run just before MyST resolves #anchor links in a page (priority 870, before 879). It sets the ids and rekeys MyST's slug table in place, so the sidebar, links within a page and links between pages all use GitHub's anchors.

    • It numbers repeated titles as GitHub does: foo, foo-1, foo-2, where MyST gave foo-1-2 for the third.

    • An anchor already used by another element is now a warning (an error under -W); round 9's hook skipped it silently.

    • Checked against the Markdown with a script (not committed). It computes GitHub's anchor for every heading outside code fences in the 15 pages (211) and finds it as an HTML id. Every site link to a heading uses that anchor (0 off; on 0.4.53's conf.py, 84 headings lacked theirs and 179 links were off).

    • On a copy: three ## Foo headings got foo, foo-1 and foo-2; links to #foo-2 from the same page and from another page resolved; and a collision failed the build.

  • Nit. A draft of this entry counted 210 headings, but the entry's own heading makes 211. The count was corrected before commit.

• Review round 11 (1 Low, 4 nits), all fixed:

  • Low. The collision check failed the build for two different GitHub anchors. Under ## `cache_dir` then ## Cache dir, docutils gives the first heading the id cache-dir, which is the second heading's GitHub anchor. 0.4.53 built that, and the order of the headings decided the outcome.

    • The transform now works out every anchor first. An anchor that is only a docutils id of another heading is taken from that heading; both headings then carry their GitHub anchors, in either order.

    • An anchor that is another element's own id still fails the build. That covers an explicit (label)=, including a label that names the anchor GitHub gives a later heading (GitHub does not render MyST labels), and two headings with one anchor.

  • Nit. Anchors came from the title's text including raw HTML (## Raw <b>bold</b> heading gave raw-bboldb-heading). They now use the text alone, as GitHub does.

  • Nit. Level-5 and level-6 headings had no slug (myst_heading_anchors = 4), so they got no GitHub anchor and were left out of the numbering of repeats. A link to #foo-3 could then land on another heading than on GitHub. All six levels are now slugged and counted.

  • Nit. The transform's docstring said the sidebar was collected "after all transforms". It is collected on doctree-read, emitted at priority 880, so the transform must run before that. The docstring now says so.

  • Nit. Round 10's nit bullet cited a "210" that no committed text contained; it now says the count was in a draft.

  • Each case was checked on a copy of the repository with -W: both heading orders, inline HTML, an h5 among repeated titles (links to #foo-3 and #foo-4 land as on GitHub), and both kinds of explicit-label collision.

0.4.53 (unreleased)

The documentation builds into an HTML site with Sphinx (user request).

• docs/conf.py builds from the repository root, reading only README.md, CHANGELOG.md and docs/**.md, so links between them resolve in the site as on GitHub. It uses MyST for Markdown with GitHub-style heading anchors, renders ```mermaid fences as diagrams, uses the Furo theme, and takes the version from internal/db/version.go.

• The navigation comes from NAV in docs/conf.py and is added to the page sources at build time, so the Markdown carries no Sphinx markup. The HTML build writes an index.html that redirects to the README page, for static hosts.

• The query docs' ### headings directly under the page title stay, because run.py and the tests find queries by them. The resulting myst.header warning is suppressed.

• docs/requirements.txt pins Sphinx 9.1.1, myst-parser 5.1.0, sphinxcontrib-mermaid 2.1.1 and furo 2025.12.19, and needs Python 3.12 or later. Builds were checked with 3.12, 3.15 and a warnings-as-errors run. .gitignore excepts it from *.txt and ignores docs/_build/.

• CI's new Docs job builds with -W --keep-going on Python 3.13 and keeps the HTML as the docs-html artifact. Any warning fails the build, including a broken link or heading anchor, or a page missing from NAV; a stray page was checked to fail.

• docs/development.md ("Documentation site") and README.md describe the build.

0.4.52 (unreleased)

Review rounds 6 to 8 (1 nit each), fixed:

• Nit (test, round 6). Decoding psql's stderr in the locale's encoding passed the test. The failing query ran only under a UTF-8 locale, where that encoding is UTF-8. On macOS under a legacy locale, which run.py accepts, server errors would then reach the report garbled. TestRunPyMainExitStatus now also fails DQ-07 under its ASCII locale where the filesystem encoding stays UTF-8 (macOS), and expects the message's U+00AB intact.

• Nit (round 7). "On Linux that locale is refused, so no locale-dependent decode can run there" overstated. run.py accepts a C or legacy locale on Linux under UTF-8 mode (-X utf8, as the refusal advises, PYTHONUTF8=1, or Python 3.15), and locale.getencoding() still returns the locale's real encoding there. The macOS-only check never ran in CI. The test now also fails DQ-07 in its C-locale run under -X utf8, on every platform, and a locale.getencoding() decode fails it. A getpreferredencoding decode is UTF-8 under UTF-8 mode, so only the macOS run catches that, and on Linux it is harmless in every run run.py accepts.

• Erratum for 0.4.51, noted in passing by round 5 and not counted as a finding: its header counts 3 nits for round 4, but the list shows two nit bullets. The third, a test comment that still said "a non-UTF-8 locale on Linux", is inside the second bullet.

• Nit (round 8). This entry had the same mismatch: its header counted one nit per round, but one bullet held both rounds' nits, followed by an erratum with no round. Each round's finding now has its own bullet, and the erratum says where it came from.

0.4.51 (unreleased)

Review rounds 4 and 5 (round 4 on 0.4.50: 1 Low, 3 nits; round 5: 3 nits), all fixed. Errata for 0.4.50:

• Low. The 0.4.50 example "C with PYTHONUTF8=0" was only partly right. LANG=C or LC_CTYPE=C with PYTHONUTF8=0 is coerced to C.UTF-8, which 0.4.49 accepted. Only LC_ALL=C (which turns coercion off) or PYTHONCOERCECLOCALE=0 was refused. The right example is LC_ALL=C with PYTHONUTF8=0.

• Nit. run.py and docs/analysis/README.md were imprecise about when the refusal happens:

  • PYTHONUTF8=0 under a UTF-8 locale is not refused;

  • the C locale is refused only with coercion off;

  • other POSIX systems besides Linux refuse too.

  They now say: under a non-UTF-8 locale, with Python 3.14 or older or PYTHONUTF8=0, not on macOS, and for C only with LC_ALL=C or PYTHONCOERCECLOCALE=0. A test comment still said "a non-UTF-8 locale on Linux"; it now says "not on macOS".

• Nit (test). 0.4.50 said a mutation pass killed every other mutation of the encoding handling. Decoding psql's stderr strictly survived. psql translates its messages into the locale's encoding, and under Python 3.15 (UTF-8 mode by default) a Latin-1 locale is not refused, so a strict decode would end the run with a traceback. The stand-in's failure message now carries a byte that is not UTF-8, and the test still expects the failure in the report.

• Review round 5 (3 nits), all fixed:

  • Nit. The C-locale clause above left out PYTHONUTF8=0. Python enables UTF-8 mode under C by itself, so LC_ALL=C alone is never refused.

  • Nit. It also said "only when coercion is off", but coercion fails too where no C.UTF-8 locale is installed. Both docs now say: for C, only with PYTHONUTF8=0 and when Python does not coerce the locale to C.UTF-8 (LC_ALL=C, PYTHONCOERCECLOCALE=0, or no C.UTF-8 locale).

  • Nit (test). The stand-in's message pinned no particular decoding: a Latin-1 decode passed. The message now also holds a UTF-8 U+00AB, and DQ-07's section must show it, with the byte that is not UTF-8 as U+FFFD.

0.4.50 (unreleased)

Review round 2 of 0.4.49 (1 Low, 3 nits), all fixed or answered, and a commit erratum.

• Erratum (commit). The 0.4.49 commit (dbe01ab) was taken while the reviewer had a mutation applied in place: its run.py lacks sys.stdout.reconfigure(errors='backslashreplace'), the line under the comment about the progress line, so TestRunPyMainExitStatus fails at that commit. This version has the line.

• Low. 0.4.49's check of the locale's encoding refused runs that worked. On macOS the filesystem encoding is always UTF-8, so psql gets the path as UTF-8 under any locale. Nothing else used the locale's encoding: run.py opens its files as UTF-8 and its progress line uses backslashreplace. So a macOS run under an explicit legacy locale (LANG=en_US.ISO8859-1 with Python 3.14 or older, or C with PYTHONUTF8=0) worked under 0.4.48 and was refused by 0.4.49. run.py again checks only the filesystem encoding, and the docs say when that refuses. This corrects 0.4.49's nit fix, which should have corrected the claim, not widened the check.

  • Test: TestRunPyMainExitStatus asks the interpreter for its filesystem encoding under an ASCII locale (C, without UTF-8 mode or coercion). Where that encoding is ASCII (Linux, CI), it expects the refusal and no report. Where it stays UTF-8 (macOS), it expects the run to work.

  • The configured user is now non-ASCII, and the stand-in checks that it arrives as UTF-8. The macOS case therefore covers the explicit UTF-8 on the configuration and report files, which 0.4.49 said no test could reach. The analysis docs are ASCII today (no test enforces it), so neither their explicit encoding nor the SQL's on psql's stdin is reached.

  • A mutation pass on macOS kills every other mutation of the encoding handling. Round 3 added a test for the call to the check.

• Nit. Erratum for 0.4.49: before 0.4.48, a SQL_ASCII value that is not UTF-8 did not fail on the server. psql piped sets no client encoding, so the server sent the raw bytes, and run.py stopped with a UnicodeDecodeError traceback.

• Nit. docs/analysis/README.md now says that bytes the server refuses in the sample :file_path or :cve_id stop the run at that repository, not just one query.

• Nit. Removing the locale's encoding from 0.4.49's check in main was caught only on macOS: under C on Linux, the filesystem check alone refused. The check now takes one encoding, so the two-encoding test is gone.

• Review round 3 (1 Low, 4 nits), all fixed:

  • Low. The example in the Low above was wrong. It said cron, launchd or ssh with no LANG, but Python enables UTF-8 mode under the C locale, so 0.4.49 ran there. The example now names the runs it really refused.

  • Nit. "Refuses a non-UTF-8 locale on Linux" overstated what is refused. Python coerces the C locale or enables UTF-8 mode under it, and from 3.15 UTF-8 mode is the default. run.py and docs/analysis/README.md now say when it refuses: a legacy locale on Linux with Python 3.14 or older, or PYTHONUTF8=0.

  • Nit. This entry called the docs "ASCII by rule", but only Go sources are ASCII by rule. It also said every mutation was killed but one, when the SQL's encoding on stdin survives too. Both are corrected above.

  • Nit (test). Removing the call to the check from main passed on macOS. TestRunPyConnectionSettings now runs main with the filesystem encoding reported as ASCII and expects the refusal, on any platform.

  • Nit. A test comment said "the report's path" for the sample path written in the report.

0.4.49 (unreleased)

Review round 1 of 0.4.48 (2 Low, 1 nit), all fixed:

• Low. Erratum for 0.4.48: bytes that are not UTF-8 in a SQL_ASCII database do not "round-trip as surrogates". With PGCLIENTENCODING=UTF8 the server checks what it sends, so such a value fails the query (invalid byte sequence for encoding "UTF8"), as it did before 0.4.48. The surrogate handling could never run; run.py now decodes psql's output strictly and says so, and the report no longer takes surrogates.

• Low (tests). Several 0.4.48 changes had no test that failed without them: splitting report lines on newlines only, the progress line's backslashreplace, and the call to the locale check in main (only the function was tested). TestRunPyMainExitStatus now covers each, and a mutation pass kills each of them:

  • a result row holds U+2028, which splitlines broke;

  • the sample path holds a letter outside Latin-1, and a run with PYTHONIOENCODING=latin-1 must finish;

  • run.py under a C locale without Python's UTF-8 mode or locale coercion is refused before it writes a report, and runs under -X utf8.

  The explicit UTF-8 on the configuration, docs and report files stays, but no test can reach it: a locale that would change it is refused first.

• Nit. 0.4.48 said run.py refuses a non-UTF-8 locale, but it checked only the filesystem encoding, which is always UTF-8 on macOS. It now checks the locale's encoding too, so the claim holds; TestRunPyConnectionSettings covers a UTF-8 filesystem encoding with a US-ASCII locale.

0.4.48 (unreleased)

Copilot review 5853583428 of PR #3 (1 Medium), fixed:

• Medium. run.py read psql's output in text mode. Universal newlines turned a carriage return in a stored path (src/a\rb.c) into a newline, so LAB-07 and the other :file_path queries were given another path. An inherited PGCLIENTENCODING=LATIN1 made psql send LATIN1, which does not decode as UTF-8. run.py now sets PGCLIENTENCODING=UTF8 and reads psql's output as bytes, decoded as UTF-8 with no newline translation; bytes that are not UTF-8 (a SQL_ASCII database) round-trip as surrogates. It refuses a non-UTF-8 filesystem encoding, which would send psql the path in another encoding (python3 -X utf8 run.py there). It reads its configuration and the analysis docs, and writes the report, as UTF-8 whatever the locale. Report lines are split on newlines only. Tests: TestRunPyMainExitStatus runs with a stand-in psql that writes in the client encoding and logs each file_path argument's bytes. Its sample path holds a carriage return and a non-ASCII letter; the path must reach the report and every query unchanged, with and without an inherited PGCLIENTENCODING=LATIN1. TestRunPyConnectionSettings checks the pinned encoding and the refusal.

0.4.47 (unreleased)

Review rounds 18 to 20 (4 nits), all fixed. Nothing was High, Medium or Low. Round 18's throwaway clusters confirmed the stated requirement for the scratch-database CREATE DATABASE:

• PostgreSQL 14 fails (42601);

• 15, 16 and 18 succeed with UTF-8, C, POSIX, C.UTF-8 and ICU-initialised defaults;

• a LATIN1 LC_COLLATE or LC_CTYPE fails (22023).

Findings:

• Nit. An erratum for the 0.4.46 entry, which said the failure message names both LC_COLLATE and LC_CTYPE. It did not; it now does.

• Nit (test). The test now checks every part of the requirement in the failure message: "PostgreSQL 15 or later", "built with ICU", "UTF-8 or C default locale", LC_COLLATE and LC_CTYPE. Before, it checked only "PostgreSQL 15" and "ICU", so dropping the locale clause passed. (Round 20: the checked phrases now include "or later" and "default locale", whose loss passed too.)

• Nit (round 19). In this entry, the cluster results and the findings rendered as one list. A "Findings:" line now separates them.

0.4.46 (unreleased)

Review round 16 of 0.4.45 (1 Low, 1 nit), both fixed. It replicated the CI test job on linux/amd64: an ubuntu:24.04 user with Go 1.26.10, psql 16, Python 3.12, git 2.56 and CI=true, against a default postgres:18 (glibc en_US.utf8), with the workflow's DSN. The full suite passed for scc 3.7.0 and 4.1.0. With stock Ubuntu git 2.43, the three reftable tests fail with "fix the runner", as designed; GitHub's image has git 2.56.

• Low. Pushes to main could still lose CI runs. cancel-in-progress: false spares only a running run. Within one concurrency group, a newly queued run cancels a pending one, so with three quick pushes the middle one went untested. Runs other than pull requests now each get a group of their own (github.run_id). The 0.4.45 entry's "pushes to main each keep theirs" held only for a running run.

• Nit. The README's test-cluster paragraph now states what scratch databases need: PostgreSQL 15 or later, built with ICU, with a UTF-8 or C default locale (LC_COLLATE and LC_CTYPE). A failed CREATE DATABASE in testdb.New names that requirement as a possible cause, with a test.

• Review round 17 (1 Low, 1 nit), both fixed:

  • Low. The requirement named only LC_CTYPE, but PostgreSQL checks the template's LC_COLLATE against ENCODING 'UTF8' too. A cluster with a UTF-8 LC_CTYPE and a LATIN1 LC_COLLATE fails every DB-tier test. The README, docs/development.md, the failure message and this entry now name both.

  • Nit. The hint is added to every CREATE DATABASE failure, including a lost connection or a duplicate name. It now reads as a possible cause ("if every DB-tier test fails here"), with the reason written at the site.

0.4.45 (unreleased)

GitHub Actions CI (user request), and review rounds 13 to 15. Round 14 found one Medium, in round 13's new test. Round 15 found one High: CI as first written would have failed, because two golden results depended on the cluster's collation.

• CI (.github/workflows/ci.yml), on pushes to main, pull requests and by hand:

  • Lint: gofmt, go vet, staticcheck v0.8.2 and golangci-lint v2.15.0. actions/setup-go@v7 installs the toolchain go.mod names (its toolchain line), and go run builds the linters with it, as docs/development.md advises;

  • Test: the full suite, unit and database tiers, against a PostgreSQL 18 service container, for scc 3.7.0 (the version Aveloxis runs) and scc 4.1.0. Both scc versions are installed with go install, plus psql if the runner lacks it. GitHub sets CI=true, so a test missing git, scc, psql or python3 fails rather than skips (internal/testdb).

  Concurrency: a newer push to a pull request cancels its older run; pushes to main each keep theirs. Checked locally: actionlint passes, and the full suite passes with CI=true and a TCP superuser with a password (as the CI DSN gives), with scc 3.7.0 and with 4.1.0. Round 15 ran it on Linux against postgres:18.

• Scratch databases collate with ICU en-US (testdb.New, TestNewCollatesWithICUEnUS). They are created TEMPLATE template0 LOCALE_PROVIDER icu ICU_LOCALE 'en-US', whatever the cluster's default. This needs PostgreSQL 15 or later, built with ICU.

• Review round 13 (1 Low, 1 nit):

  • Low (tests). The 0.4.44 entry said each check fails on its mutation of run.py. Three mutations passed:

    • scalar() using strip() again: the test called scalar_value directly, and the stand-in's sample path had no spaces;

    • run_all ignoring failed global queries: no global query failed in the test;

    • main() not calling finish: main() was not tested.

    The sample path now has its own spaces, and the test checks that they reach the per-repo queries. A global query now fails in the test. The new TestRunPyMainExitStatus runs run.py as a script. Each of the three mutations now fails.

  • Nit. The test comment says a type refusal never prints the value. Range refusals still print the number, and a host-list refusal the host; neither is a credential.

• Review round 14 (1 Medium, 1 Low, 2 nits), all fixed:

  • Medium. TestRunPyMainExitStatus was not hermetic. Its stand-in psql had the absolute python path as its shebang interpreter. A python shim (pyenv or asdf on macOS) or a path with a space cannot be one, so the stand-in failed to start. run.py then found the real psql further along PATH, and contacted the developer's local server at host=/tmp. The fix:

    • the stand-in is now psql.py, started by a /bin/sh wrapper with quoted paths;

    • PATH holds only the wrapper's directory;

    • the configured host is a socket directory that does not exist.

    (Corrected in round 15: this passed with a /bin/sh shim only.)

  • Low. The test did not check the "full report" the entry above claimed. It now checks that both reports list every documented query, in run.py's order: the global ones once, then repository 1's. A documented query in a doc run.py does not read also fails the test. It checks that a passing section holds the stand-in's rows, and that the failing query's section holds psql's error. The failing query is now DQ-07, which has another global query after it, so stopping the global loop at a failure is caught. Seven mutations of run.py were tried; each now fails a test:

    • stopping the global loop after a failure;

    • dropping the per-repo queries;

    • reading only one doc;

    • leaving out the error;

    • leaving out the output;

    • calling finish after each query;

    • not calling finish.

  • Nits:

    • run.py runs with the temp directory as its working directory;

    • the failing query's table name (scc_blob_cache, only in DQ-07) is noted in the test, which fails, not passes, if another query uses it.

• Review round 15 (1 High, 1 Medium, 1 Low, 3 nits), all fixed:

  • High. CI would have failed on both legs. CVE-10's golden result and TestMissingEmailIndexes order text. They match macOS libc en_US.UTF-8, but the postgres:18 image initialises glibc en_US.utf8, which ignores punctuation when it orders. The C locale fails too, being case-sensitive. ICU en-US matches the golden results, so testdb.New now creates every scratch database with it. Local runs and CI then agree, whatever either cluster's default. The full suite passes locally with ICU scratch databases.

  • Medium. Round 14's PATH-only fix broke pyenv and asdf shims: they are #!/usr/bin/env bash scripts that need bash and coreutils on PATH. The test now asks the python3 it finds for sys.executable, with the inherited environment, and runs run.py and the stand-in with that real interpreter. It passes with an env bash shim in a directory whose name has a space.

  • Low. actions/setup-go@v5 reads only go.mod's go line, so it installed 1.26.0 and built the linters with it, not with the toolchain 1.26.10 the entry claimed. The workflow now uses actions/setup-go@v7, which reads the toolchain line, and actions/checkout@v7.

  • Nits:

    • the test builds its configuration with json.Marshal;

    • the stand-in answers SHOW default_transaction_read_only with on (stand-in) only when PGOPTIONS asks for read-only sessions, and the test checks that line, so a hard-coded read only: on or a lost read-only option fails it;

    • the report is no longer removed between runs, so a report opened for append fails the test;

    • pushes to main no longer cancel each other's CI.

0.4.44 (unreleased)

Copilot review 5852421019 of PR #3 (3 comments, all on run.py), each fixed red-first, and review round 12 of the 0.4.43 changes.

• Copilot review 5852421019:

  • No values in refusals. A malformed database block printed itself, and a password of the wrong type printed the password, so credentials could reach a terminal or a job log. Refusals now name the JSON type (got a JSON array), never the value, for blocks, typed keys and aveloxis_config_path.

  • Sample values keep their whitespace. strip() on a psql scalar removed a path's own leading and trailing spaces, so LAB-07 could read another file. Only psql's final newline is removed now (scalar_value).

  • Failed queries fail the run. A failed query was marked ERROR in the report, but run.py still exited 0. It now finishes the report and then exits with status 1, naming the failed queries (run_all, finish). The runner's loop is now module-level so the test can drive it with a stand-in psql.

  TestRunPyConnectionSettings checks each case, and each check fails on its mutation of run.py.

• Review round 12 (1 Low, 2 nits), all fixed. Nothing was High or Medium, and no finding was a wrong definite answer.

  • Low. CVE-12's tie_order comment still said, unqualified, that the latest change is always among the latest commits. 0.4.43 corrected the note but not the comment. The comment now limits it to one landing, and says that across landings a later landing's merge naming the CVE counts as a change.

  • Nits:

    • an erratum for the committed 0.4.43 entry, left as committed. Its first Low said four conditions of later_than could each be deleted. Before 0.4.43, two later_than conditions survived deletion: the merged-in rule's same-landing condition, now caught by CVE-2024-0030, and its ancestry check, now caught by CVE-2024-0031. So did dropping below's first-parent start, now caught by CVE-2024-0032, whose first parent is not itself a tie end. So did narrowing the non-merge rule to non-merge commits, now caught by CVE-2024-0029. Deleting the first-parent condition itself was already caught by CVE-2024-0024;

    • CVE-2024-0029's revert is a git revert -m 1 whose message names the CVE. A default git revert -m 1 message quotes only the merge's subject, so unless that names the CVE, the revert is no undo and CVE-12 does not list it. The seed comment says so.

0.4.43 (unreleased)

Review round 11 of the 0.4.41 and 0.4.42 changes (3 Low, 4 nits), all fixed. Nothing was High or Medium, and no finding was a wrong definite answer.

• Low (tests). Four conditions of CVE-12's later_than could each be deleted without changing any seed result. The seed adds one case for each:

  • CVE-2024-0029: a non-merge after a merge naming the CVE is later than it. A git revert -m 1 of a pull-request merge that names the CVE, whose own commit does not, reads "reverted".

  • CVE-2024-0030: the merged-in rule applies only within one landing. A revert on the first-parent chain, then in the same second a merge of the release branch into a topic named for the CVE, reads "fixed again", as across landings. Without the condition, the two order each other.

  • CVE-2024-0031: a merge outranks only what it has in its history. A revert and an unrelated merge naming the CVE in one octopus landing read "order unknown".

  • CVE-2024-0032: a merge's first-parent side is not "merged in". A revert on a merge's first-parent side, behind a commit naming nothing, reads "order unknown".

  Each condition, and the across-landings rule, now fails the test when deleted.

• Low. The 0.4.42 entry said CVE-2024-0028 tests the non-merge rule. It does not; CVE-2024-0029 does.

• Low. The notes said the latest change is always among the latest commits, so an answer naming one kind is right. That holds within one landing. Across landings, a merge naming the CVE counts as a change: a later landing merge naming the fix reads as a reapply, as it would a second later. The notes now say so.

• Nits:

  • the 0.4.42 entry's "(CVE-2024-0023 to -0025, wrongly)" should read "-0023 and -0024": 0.4.41 said "reverted" for -0025, which is right;

  • "pgx ignores PGTZ" was wrong. pgx reads it, but aveloxis-cve's DSN sets timezone=UTC, which pgx puts above the environment. run.py's fix (PGTZ=UTC) stands;

  • the seed comment for CVE-2024-0026 describes the current rule;

  • the redundant same-landing condition on the non-merge rule is gone.

0.4.42 (unreleased)

Review rounds 4 to 10 of the 0.4.41 changes. Rounds 4 to 7 and 9 found nothing High or Medium; rounds 8 and 10 each found one Medium in CVE-12's tie ordering.

• Review round 4 (1 Low, 4 nits), all fixed:

  • Low. With PGSERVICE set, libpq lets a service file override the settings run.py passes in the environment, and PGHOSTADDR redirects the connection. So run.py could read a different database than aveloxis-cve. run.py now refuses to run with PGSERVICE set, and drops PGHOSTADDR, which pgx ignores.

  • Nits:

    • an imported "Host" was dropped, where Go's decoder reads it as the host (round 5 settles how such keys are handled);

    • a database or labor value that is not an object is refused; only null means the defaults;

    • CVE-12's first note covers a tie ordered undo-last;

    • an erratum for the committed 0.4.41 entry, left as committed: its review round 2 had 6 nits, not 4, and its round 3 lists 4 bullets for 5 nits.

• Review round 5 (2 Low, 4 nits), all fixed:

  • Low. CVE-12 ordered a tie inconsistently. It left merges naming the CVE out of tied fixes, so a pull-request merge that reapplied a fix after a first-parent revert, in the same second, read "reverted", while one a second later read "fixed again". Merges now stay in the tie. Across first-parent landings they are ordered like any commit. Within the merge that landed them, they come before the changes they brought, since a merge carries no change of its own. This replaces both 0.4.41 merge rules (tied merge fixes left out; tied merge undos left out unless all were merges), with the same answers for CVE-2024-0021 and -0022. The seed adds CVE-2024-0023 for the first-parent case.

  • Low. run.py read only exact-case top-level keys, and Python keeps the last of a repeated key where Go merges repeated objects. So "Database", "Labor" or a second "database" gave a different database or series than aveloxis-cve. Rather than emulate Go's decoder (case-insensitive with Unicode folding, merging repeats), run.py now refuses:

    • a key given twice, anywhere;

    • any spelling of a known key but the exact one, at the top level and in the database, labor and imported blocks.

  • Nits:

    • run.py's reason for refusing PGSERVICE is corrected: pgx does read service files, but lets them fill only the values the configuration leaves empty;

    • run.py documents that a relative aveloxis_config_path is read from the working directory, as aveloxis-cve reads it, and turns an unreadable configuration into an error message;

    • run.py refuses a host list (a comma in host), a negative pool_max_conns and integers beyond int64;

    • LAB-13's note says "NULL ratios".

• Review round 6 (3 Low, 4 nits), all fixed:

  • Low. Round 5's merge rule ranked every merge in a landing before the other changes in it, not only the merge that landed them. A sub-merge that reapplied a fix after a revert therefore read "reverted"; fast-forwarded, the same commits read "fixed again". Only the landing merge now comes first, and every other commit in the landing, merges included, is ordered by ancestry. The seed adds CVE-2024-0024 for the sub-merge case.

  • Low. run.py read only the exact database key of the imported Aveloxis file. Go's decoder takes a "Database" there too (the last of them). run.py now refuses any other spelling of database or collection in that file.

  • Low (test). The round-4 "Host" fixture also misspelled dbname, so it was refused for that and did not test the "Host" refusal. The fixture now spells dbname exactly.

  • Nits:

    • tests now reach check_top_level through database_block and labor_series, the int64 bound through connect_timeout_seconds, and the imported file's duplicate keys;

    • run.py refuses an aveloxis_config_path that is not a string (0, false, [] or true were read as no import, or as file descriptor 1);

    • the 0.4.41 count correction above is marked as an erratum;

    • two tied merges of different kinds, one descending from the other, were "order unknown"; the first Low's rule orders them by ancestry.

• Review round 7 (1 Low, 2 nits), all fixed:

  • Low. Round 6's rule still depended on where a merge sat. A merge naming the CVE, whose first parent was an unrelated change and whose second parent the revert, read "fixed again" when merged in. Fast-forwarded, the same history read "reverted". One rule now places a merge naming the CVE, wherever it sits: after its first-parent side and before the commits it merged in. Ordinary commits are ordered by ancestry, and across landings by first-parent order. Merges naming the CVE that merge each other's sides can leave no latest commit; the result is then "order unknown", as the reading note says. CVE-2024-0018 to -0024 keep their answers. The seed adds CVE-2024-0025 for the reversed-parent merge.

  • Nits (tests):

    • the imported file's exact duplicate keys are now tested (round 6's fixture tested a case variant);

    • Unicode folding is now tested where it matters, in the imported block and at the top level, not only in the configuration's own block, which refuses unknown keys anyway.

    Each new test fails on its mutation of run.py.

• Review round 8 (1 Medium, 1 Low, 2 nits), all fixed:

  • Medium. Round 7's rule put a merge naming the CVE after its first-parent side and before what it merged in. A branch updated from the mainline ("Merge branch 'main' into fix-...") then gave a definite, wrong "reverted": its independent fix came before the mainline's revert, though the two are unordered. 0.4.41 said "order unknown". Rounds 5 to 8 each found a new shape that a finer merge rule got wrong. So CVE-12 now orders a tie only where the graph settles it:

    • across landings, by the first-parent order of the landing commits;

    • within one landing, the landing merge before what it brought, and the rest by ancestry;

    • a landing that brought a merge naming the CVE (other than the landing commit) has an unknown order inside it.

    This replaces the merge rules of rounds 5 to 7, and the relation can no longer have a cycle. CVE-2024-0018 to -0023 keep their answers. CVE-2024-0024 and -0025, sub-merges naming the CVE, now read "order unknown"; their expected results were set by hand first.

  • Low. The notes claimed that a history reads the same whether merged in or fast-forwarded, and described a cycle case. Both are gone. The notes now say a fast-forwarded history can read definitely where the merged-in one reads "order unknown", and that a merge naming the CVE inside a landing is what makes it unknown.

  • Nits:

    • run.py and the README say to run run.py and aveloxis-cve from the repository root, or to set an absolute $AVELOXIS_CVE_CONFIG for both. run.py finds its default configuration next to itself, while aveloxis-cve reads ./aveloxis-cve.json;

    • two run.py test cases that other checks already refused are relabelled or dropped.

• Review round 9 (2 Low, 1 nit), all fixed:

  • Low. Round 8's tangled rule was both untested and harmful. It was untested because sub-merges naming the CVE were already left unordered, so removing it changed no result. It was harmful because it made the usual GitHub shape read "order unknown": the mainline merged into a release branch, bringing pull-request merges whose titles name the CVE. 0.4.41 had that shape right. tangled is gone. Sub-merges naming the CVE stay unordered against the rest of their landing. That is still sound: the latest change is always among the unordered latest commits, so an answer that names one kind is right, and mixed kinds read "order unknown". The seed adds CVE-2024-0026 for that shape: a fix, its revert and a reapply, each landed by a pull-request merge naming the CVE, merged together into release-5.x. It reads "fixed again", and read "order unknown" with tangled.

  • Low. The notes and this entry did not say which shapes read "order unknown" where 0.4.41 gave an answer. The notes now name the case: a merge naming the CVE inside a landing, of a different kind than the latest change. (Corrected in round 10: 0.4.41 did not say "reverted" for every tie.)

  • Nit. The advice on where to run run.py also asks for an absolute aveloxis_config_path when $AVELOXIS_CVE_CONFIG is set, since each process resolves a relative one from its own working directory.

• Review round 10 (1 Medium, 3 Low, 1 nit), all fixed. No finding was a wrong definite answer.

  • Medium. The usual GitHub revert flow read "order unknown" where 0.4.41 correctly said "reverted". In that flow, a fix PR and a revert PR are each merged with a merge commit, and the mainline is then merged into a release branch. Both PR merges name the CVE through their titles. The revert PR's merge has no "This reverts commit" line, so it counts as a fix. Left unordered, both stood beside the revert as latest commits of the other kind. CVE-12 now orders a landing by one principle: a merge names the CVE but carries no change of its own, so it never outranks a change. The commits it merged in (ancestors not on its first-parent side), and the non-merge commits after it, are later than it. Non-merge commits are ordered by ancestry. Nothing else in a landing is ordered, so:

    • the latest change is always among the latest commits, and an answer naming one kind is right;

    • only demotions of merges are added, so there is no cycle;

    • this replaces the landing-merge rule, which it covers.

    The seed adds:

    • CVE-2024-0027, the GitHub revert flow, which now reads "reverted";

    • CVE-2024-0028, a reapply after a revert PR merge that is an undo, which now reads "fixed again".

    Both read "order unknown" before. CVE-2024-0025's merge brought in the revert, so it now reads "reverted", its fast-forwarded reading. CVE-2024-0018 to -0024 and -0026 keep their answers.

  • Low. A non-merge after a merge naming the CVE was not ordered after it, and no test covered that. The rule above orders it, and CVE-2024-0028 tests it.

  • Low. Round 9 said 0.4.41 "said 'reverted; not fixed again' for every tie". That was false. 0.4.41 ordered ties by ancestry with tied merge fixes left out. It said "reverted" whenever a merge naming the CVE was the reapply (CVE-2024-0023 to -0025, wrongly), and was right for a revert brought in with its fix PR's merge.

  • Low (predates 0.4.41). run.py sessions were not in UTC when PGTZ was set: libpq applies it after the PGOPTIONS time zone, while pgx ignores it. run.py now sets PGTZ=UTC, with a test.

  • Nit. A redundant condition in the old landing rule is gone with the rule.

0.4.41 (unreleased)

Copilot review 5850785132 of PR #3 (7 comments), each fixed red-first, and review round 6 of findings.md.

• The golden update refuses a partial result (High). go test -run TestAnalysisQueriesRun -update rewrote testdata/analysis_golden.txt from whichever subtests ran and passed, so a filtered run, or one with a failing query, dropped every other reviewed result. It now refuses unless every documented query (and series) produced a result and nothing failed (goldenText; TestGoldenTextRefusesPartialResults).

• run.py (1 High, 2 Medium):

  • It finds its configuration and docs next to itself, not in one developer's checkout path. $AVELOXIS_CVE_CONFIG overrides the configuration.

  • It reads the database block as aveloxis-cve does:

    • omitted host, port, sslmode and connect_timeout_seconds take the documented defaults;

    • a configuration without a database block uses its aveloxis_config_path's block;

    • an empty user, dbname or password leaves PGUSER, PGDATABASE and PGPASSWORD (or ~/.pgpass) in place instead of overwriting them.

  • Its sample :file_path comes from the current default branch's series, the pick the documented queries make. After a default-branch change, it took the newest window of any branch.

  • MAX_LINES can be set from the environment, and main runs only when the file is run as a script.

  • Tests: TestRunPyConnectionSettings loads run.py as a module and checks each case; TestRunPySamplesFollowTheDefaultBranch runs its sample SQL on the seed.

• LAB-12 lists every step (Medium). A step between two windows that count no files had no file rows, so the lateral join dropped it. A LEFT JOIN LATERAL keeps it, with deltas of 0 and no placeholder counted as an added file.

• LAB-13 keeps windows that count no files (Medium). Its inner joins dropped them, so handwritten_delta compared windows that were not adjacent. Left joins keep them, with totals of 0.

• CVE-12 orders a fix and an undo that landed together (Medium). When one merge brought the latest fix and the latest undo, their landing times tie, and the branch was always reported "reverted; not fixed again", even when a reapply came last. The query now orders the tied commits by ancestry. It walks parents only among the commits that landed at that time, since a commit between two of them cannot have landed earlier. A reapply descending from the revert is "fixed again"; a revert and an independent fix that neither descends from the other are order unknown: landed together.

• Seed:

  • a 45-day, 1-month series whose first two windows count no files; every :window_days query now also runs on it (<ID>@45/1);

  • release-5.x, where an octopus merge lands a fix, its revert and a reapply of CVE-2024-0018, and a fix, its revert and an independent fix of CVE-2024-0019;

  • a file only in devel's newer window, which run.py's old sample pick took.

• Docs:

  • docs/analysis/README.md documents run.py;

  • LAB-04 notes that a window that counts no files has no rows;

  • the new tests are in the development tripwire table.

• Review round 1 (1 Medium, 2 Low, 2 nits), all fixed:

  • Medium. run.py accepted an imported Aveloxis database block without user or dbname. It then connected with the inherited PGUSER and PGDATABASE, and sent any inherited PGPASSWORD to the imported host; aveloxis-cve refuses that configuration. run.py now refuses it too. It also refuses unknown database keys and an out-of-range port or timeout, keeps a default for a JSON null, and leaves PGHOST alone when it is empty.

  • Low. CVE-12 ordered tied landings by ancestry only. Two first-parent commits in the same second (a fix, then a merge bringing a revert of its cherry-pick) were reported "order unknown". It now orders tied commits first by the first-parent commits that landed them, and by ancestry only within one merge. The seed adds CVE-2024-0020 for this.

  • Low. The 45/1 series disagreed with the rest of the seed. Window 9 now counts 2 commits. Windows 8 and 9 have changed rows with NULL metrics, and are recorded as measured by an scc (3.6.0) that counted none of their files, with cache rows to match. Window 10's rows are modifications measured on both sides, with a new blob for src/a.c after c11.

  • Nits:

    • TestWithoutCode's comment is back above it;

    • run.py documents that it checks only the database block, and resolves a relative aveloxis_config_path from the working directory, as aveloxis-cve does.

• Review round 2 (1 Medium, 1 Low, 4 nits), all fixed:

  • Medium. CVE-12's tie ordering let a merge commit whose message names the CVE outrank the revert it brought. A GitHub "Merge pull request" of a revert counts as a fix, since its message names the CVE but has no "This reverts commit" line. Such a branch was reported "fixed again". A merge carries no change of its own, so tied fixes now leave merges out. The seed adds CVE-2024-0021 for this case.

  • Low. Snapshots hold only the files scc counted:

    • LAB-12 and LAB-06 now note that a file the previous measurement did not count shows as added;

    • LAB-07's content_changed is NULL, not false, when the snapshot has no row for the file at either end.

  • Nits:

    • CVE-12's notes name every source of a tie, including a first-parent clock behind its parent;

    • the seed's new hashes are hex;

    • the clean cherry-pick c44 shares c43's patch-id;

    • c16 adds src/devel_only.c, so the file devel's window holds has a commit;

    • run.py reads the labor block as aveloxis-cve does (JSON null keeps a default, each value must be >= 1);

    • run.py strips the brackets of an IPv6 host.

• Review round 3 (2 Low, 5 nits), all fixed:

  • Low. CVE-12 left merges out of tied fixes but not tied undos. A pull-request merge whose message says it reverts the fix (an undo) outranked the reapply it brought. Tied merge undos are now left out unless every tied undo is a merge. The seed adds CVE-2024-0022 for this case.

  • Low. An empty sslmode left an inherited PGSSLMODE in place in run.py. aveloxis-cve (pgx) uses prefer then and ignores PGSSLMODE, and run.py now does the same.

  • Nits:

    • CVE-12's notes say a later commit that only mentions the CVE counts as a reapply, and that a clock implausibly far ahead also ties a landing;

    • LAB-07 notes that content_changed with a row at one end only may reflect counting, not content;

    • run.py refuses unknown labor keys and values of the wrong JSON type (a string port, a boolean, a fraction), as Go's decoder does;

    • findings.md notes that Spark's CVE-12 result does not involve a tie, so 0.4.41 leaves it unchanged.

• findings.md, review round 6 (1 Low, 4 nits):

  • Low: a backport is a (CVE, branch name) pair, so a change naming several CVEs counts once per CVE. HBase's four Bootstrap CVEs are probably one change of 6 files on 30 branches.

  • Nits:

    • "dependency or build" is a class of touched files, not proof of an upgrade;

    • CVE-2021-44521's commits "at most" record Cassandra's fix;

    • CVE-2020-13949 also changes DemoClient.java;

    • "code-class CVEs";

    • a stray "in".

0.4.40 (unreleased)

Review round 4 of the findings.md refresh (2 Low, 3 nits), all fixed. No query or rule changed.

• Low. The backport bullet called all 126 of Cassandra's suppression-list backports "neither fixes nor upgrades". The Cassandra paragraph names CVE-2021-44521 (3 backports) as the likely exception, so the bullet now excepts it.

• Low. The DQ-04 open question described all 78 rows in the saved output as cherry-picks and "Blocked revisions" lists. They are:

  • 43 commits with pending cherry-pick references, 19 of which also have resolved links;

  • 35 commits with pending revert references: 13 Revert commits, 2 "Blocked revisions" lists and 20 others.

  The question also counts commits (1,808), not references.

• Nits:

  • restricting to the code class removes most dependency noise, not all of it (HBase's CVE-2020-13949);

  • "the saved output of the rerun";

  • the 0.4.39 round-3 note says CVE-2020-13949's files, not its backports, are regenerated sources.

Review round 5 (1 Medium, 2 Low, 4 nits), all fixed:

• Medium. The DQ-04 question said the 78 shown commits have unresolved references, and that 19 cherry-pick rows with links had only some unresolved. 11 of those 19 name one abbreviated SHA that is already linked, as does one revert row. They stay pending only because the SHA is abbreviated (data-quality.md says so). The question now asks why 1,808 commits stay pending, and splits the 19 into 11 and 8.

• Low. CVE-2021-44521 was "the likely exception" to Cassandra's suppressions not being fixes. Its commits touch only CHANGES.txt, NEWS.txt and the suppression list, with no code, so they record Cassandra's fix rather than make it. The text now says so.

• Low. CVE-2020-13949 was in the code class because of generated sources alone. 55 of its 63 files are generated, but it also changes ThriftServer.java and ThriftConnection.java. The text now says that, and that "most of its files shown" was all 63.

• Nits:

  • the 0.4.39 note says "in the code class", not "code fixes";

  • "the delay table (CVE-06/07)", so the changed backport-by-class table is not included;

  • the 80-line cut is named as run.py's MAX_LINES: 78 rows plus the two header lines, in the uncommitted analysis_results.txt;

  • the cherry-pick links counted are cherry-pick links.

0.4.39 (unreleased)

findings.md has been refreshed from the user's rerun of every query, as written in 0.4.38, on the same collection. No query or rule changed.

• Path classes (CVE-16/17). The revised path rules moved:

  • 37 of Cassandra's CVEs, all touching its scanner-suppression lists, from docs or config to dependency or build;

  • 5 of HBase's and 2 of Hive's from code to dependency or build;

  • 3 of Spark's net into code (26 code, 151 dependency or build, 10 docs or config).

• Backport split. Of 219 backported CVEs:

  • code: 70 CVEs, 176 backports;

  • dependency or build: 105 CVEs, 527 backports;

  • docs or config: 7 CVEs, 14 backports;

  • test only: 13 CVEs, 41 backports;

  • Spark's cut-off output: 24 CVEs, 54 backports, all dependency or docs.

  Few backports are in the code class (176 of 812, 22%), so the section now says so. 126 of the dependency-or-build backports are Cassandra's suppressions; without them, at least 401 are dependency or build changes (49% of all 812, 58% of the 686 that are not suppressions). HBase's four Bootstrap CVEs carry 120 of its 304 backports.

• Review round 1 (2 Low, 3 nits), all fixed. The opening line no longer says each section states its firmness. The rerun note says its class changes are CVE-16/17's, and that the LAB-13/14 generated-code figures are from the rerun. Cassandra's 37 all touch a suppression list, counting its second list under .build/owasp/, and its build.xml count is now scoped to the class it describes. The 80-line cut is the run script's.

• Review round 2 (2 Low, 4 nits), all fixed:

  • the backport headline no longer counts Cassandra's suppressions as dependency changes;

  • it explains why Spark's cut-off CVE-06 result loses no backports;

  • Cassandra's two suppression lists overlap on 2 CVEs;

  • Superset and Maven are added to the dependency-dominated repositories;

  • cve.md says Cassandra's .build/ also holds its suppression lists.

• Review round 3 (2 Low, 4 nits), all fixed:

  • Groovy (3 of 4) is added to the dependency-dominated repositories;

  • the 49% gives its denominator, alongside 58% of the non-suppression backports;

  • the code share is an upper bound (27 of HBase's code backports are CVE-2020-13949's, a Thrift upgrade whose files are mostly regenerated sources);

  • Cassandra's CVE-2021-44521 is its own CVE, so the suppressions note names it as the likely exception;

  • "at least one of its two lists";

  • DQ-04's sample is flagged as the 78 rows of the saved output, all Camel's.

• "Provisional" markers removed. The tables that use no path rule (data quality, delays, trends, the Aveloxis comparison) were checked against the rerun and are unchanged.

0.4.38 (unreleased)

Review rounds 13 to 18 of the analysis queries. Round 13's changes were not in the 0.4.37 commit (cfc4229), so they are recorded here.

• Review round 13 (1 Medium, 5 Low, 2 nits), all fixed:

  • Medium. findings.md said backport delays mostly measure dependency bumps. Joining CVE-06 with CVE-16 shows at least 75 of 219 backported CVEs are code fixes, so the section now gives the split.

  • Low:

    • the day range now agrees with the table;

    • scc versions may also differ in which files they recognise, which fits the file-count gaps;

    • per-person counts are lower bounds and the number of people an upper bound;

    • HBase's generated shares are lower bounds.

  • Tests. TestPathRules pins the resources/docs exception and the src/ anchor of the docs exclusion (src/test/resources/docs/, src/main/resources/tpl/docs/, websrc/docs/, src/xresources/docs/).

• Mutation sweep, final. After rounds 12 and 13 added rules and table rows, the sweep (TestPathRules and TestAnalysisQueriesRun, every alternative deleted, every anchor, character class and optional part narrowed) applied all 282 mutants. Only the 2 equivalent (?c) mutants survived.

Review round 14 (1 Low, 4 nits): no query or rule was wrong.

• Low. The backport split in findings.md gave docs or config as exactly 44 CVEs and left out the 13 test-only CVEs. Spark's partly shown output leaves 25 of its CVEs between dependency and docs, so the split is now a table of exact counts and ranges for every class:

  • code: 75 CVEs, 313 backports;

  • dependency or build: 62 to 87, and 263 to 318;

  • docs or config: 44 to 55, and 140 to 195;

  • test only: 13, and 41.

• Nits in findings.md:

  • Camel's parent/pom.xml count is explained by the actual precedence;

  • the per-person bounds assume correct resolved links;

  • the file-count gaps name only the repositories where they are marked.

• TestPathRules adds rows that catch widening edits as well as narrowing ones:

  • Docusaurus docs/src/theme/ (docs);

  • websrc/spec/ (test);

  • a.min.js.map (config);

  • case-insensitive Resources/Docs/ (docs) and SRC/.../Docs/ (code);

  • resources/docsite/docs/ (code, so the resources exception needs its /).

  The reviewer's 12 extra widening mutants all fail now.

Review round 15 (1 Low, 3 nits), all fixed:

• Low. Round 13's changes were filed under 0.4.37 although the 0.4.37 commit did not contain them. The 0.4.37 entry is restored to its committed text, and round 13 is recorded here.

• Nits in findings.md:

  • the backport ranges were tightened from CVE-17's totals and CVE-16's sort order; round 16 replaced them (below);

  • 27 of Camel's 29 docs CVEs are security-model entries, and the other 2 are advisory files;

  • the reverted fixes are one Spark CVE on 8 branches;

  • Maven's identical files are 99.6%.

Review round 16 (1 Low, 3 nits), all fixed:

• Low. The output still allowed tighter backport ranges than findings.md gave. Deducing classes from a truncated output's sort order was fragile, so the split now gives the exact classified counts:

  • code: 75 CVEs, 313 backports;

  • dependency or build: 62 and 263;

  • docs or config: 44 and 140;

  • test only: 13 and 41.

  Spark's 25 CVEs (55 backports) that the output did not classify get a row of their own.

• Nits:

  • HBase's lower bound reads 16.9%;

  • the commit gap with Aveloxis is measured from the window's end, and its end commit can be older, so Jenkins "may have" measured a different commit.

• Recorded here: round 13's findings.md edits that this entry did not list yet. They are the longer "How firm" list (end-to-end layouts, docs inside src/, gen_*/) and the full list of possible causes for the identity step's zero matches.

Review round 17 (2 nits), both fixed:

• this entry's intro names all its rounds;

• the possible causes of the identity step's zero matches include earlier steps winning first.

Review round 18 (2 nits), both fixed:

• findings.md says the run's output keeps the first 80 lines of each query, not 80 rows;

• this entry's round 16 and 17 items are in their own blocks.

0.4.37 (unreleased)

Review round 11 of the analysis queries (2 Low), and LAB-13's speed on real data.

• LAB-13 classifies each distinct path once (already in commit b1eaa19, recorded here). It matched the generated-file patterns against every file of every window. On apache/camel (234 windows) that took 327 s, and the 14 tracked repositories took 26 s to 327 s each. On a synthetic camel-sized series (230 windows of 40,000 files) the query now takes 3.3 s instead of 284 s, with the same result.

• Low. The src/ exclusion of round 10 turned real documentation into code. Examples: NiFi's src/main/resources/docs/.../additionalDetails.html, Superset's Docusaurus docs/src/, and Tomcat's webapps/docs/.../sample/src/. A docs/ path is now code only when a src/ tree comes before the docs/ directory and it is not a resources docs/. Go's src/go/doc/ stays code.

• Low. TestPathRules still let narrowings through. Its table had only root-level paths for most (^|/) anchors, so narrowing the test-directory rule to the root (which would turn every nested src/test/ helper into code) passed. The table now has a nested path for every anchor, and paths for every character class and optional part, including an uppercase extension after a test name (BarIT.JAVA) and the singular and plural forms of each docs?. The dead src/test alternative, already covered by test, is gone.

  • The reviewer's mutation sweep, rerun on the new rules, mutates both copies of each rule identically and runs TestPathRules and TestAnalysisQueriesRun. Its mutations delete every alternative and narrow every anchor, character class and optional part. All 280 mutants applied, and only 2 survived, both equivalent to the rule: under ~, the (?c) option is redundant, so dropping it or switching to ~* with it kept matches the same.

• The claims in 0.4.36's round 8 to 10 entries about full coverage were premature; this entry supersedes them.

• Review round 12 (1 Medium, 6 Low, 2 nits), all fixed. Most findings were in findings.md, which was rechecked against the run's output:

  • Medium: only 9 of 14 repositories agree with Aveloxis's repo_labor within 3%, not 12. The agreement section now tabulates all 14, with the scc version difference the operator reported (Aveloxis ran scc 3.7.0, aveloxis-cve 4.1.0).

  • Corrections:

    • Cassandra's "CVE fixes" are mostly scanner-suppression entries.

    • Aveloxis's dependency vulnerabilities are in aveloxis_data.repo_deps_vulnerabilities, not aveloxis_scan.

    • HBase's generated shares are restated.

    • Same-day dates do not mean the same commit (Jenkins).

    • The steady-growth claim is qualified by HBase's and Jenkins' single-month falls.

    • The pending-link split is a hypothesis.

    • No individual is named.

    • The backport table lists every repository.

  • LAB-13/14 match Thrift's gen_py/ (underscore), which HBase uses.

  • TestPathRules pins the depth of the src/.../docs/ exclusion (Flink's flink-docs/src/main/java/.../docs/ is code, Superset's docs/src/pages/apidocs/ is docs). The two mutants the review found surviving now fail.

  • CVE-16's text names the resources/docs/ exception.

• docs/analysis/findings.md (user request) records what the first collection of 14 repositories showed, with its open questions:

  • data quality and identity resolution;

  • which commits name CVEs: mostly dependency upgrades, Tomcat the exception;

  • backport delays, mostly same-day;

  • steady complexity growth everywhere;

  • generated and vendored code inflating totals, including copies in ordinary source directories that paths cannot catch;

  • agreement with Aveloxis's repo_labor: 9 of 14 within 3%, the rest likely scc 3.7.0 against 4.1.0.

0.4.36 (unreleased)

Review rounds 3 to 10 of the analysis queries, and new queries CVE-16, CVE-17, LAB-13 and LAB-14.

Review round 3 (0.4.35): 4 Low and 1 nit, all fixed.

• Clock-free backport classes (CVE-04, CVE-06, CVE-07). 0.4.35 compared landed_at across branches. A committer clock that ran fast on the default branch inflated its landing time, so a branch that had merged the default branch afterwards looked like a backport. A fix commit now counts as shared only when the commit that landed it on the default branch (its landed_via there) is also on this branch. This compares commits, not times. It covers:

  • a branch cut from, or merged with, the default branch (shared);

  • a cherry-pick later merged up, and a fix made on the branch first (both backports);

  • equal timestamps.

• CVE-01, DQ-01 and DQ-02 use the single default-branch pick. CVE-01's landed_on_default filtered on is_default alone, so a stale second default row's fix could appear as the default branch's. DQ-01's latest_window_end could report another branch's series. DQ-02's default_branch was the alphabetically first default name.

• CVE-09 and CVE-10 fold transitively. Every commit sharing a patch-id with a cherry-pick of a fix now joins the source's change. Before, a manual copy of a conflict-adjusted cherry-pick counted as a separate change.

• Test coverage of round 2's fixes. Window queries now also run against a second series: 30 days, stepped 2 months, rising only. That series catches LAB-03's sign filter; the 60/1 series cannot. The golden file holds both results (<ID>@30/2). The seed has a newer window on the stale default's branch name, which DQ-09 must not pick.

• Seed:

  • a fix only on the stale default row (renamed devel, so it sorts before main);

  • a branch that merged the default branch before the default's skew-inflated landing;

  • a manual copy of the conflict-adjusted cherry-pick;

  • outcome times after the last commit;

  • the archived copy of release-1.x as deleted and pushed again, which a force-push could not produce.

  Every changed golden value was checked by hand, and every fix's mutant is caught by its own query's subtest.

Review round 4 (2 Medium, 1 Low, 2 nits), all fixed:

• Medium. A merge back from the default branch hid a backport. Round 3's clock-free rule asked only whether the default branch's landing commit for a fix was anywhere on the branch. After a branch merged the default branch back (having been merged up into it before), every earlier backport on it became shares history. A fix commit is now shared only when the branch landed it through the same commit that brought it the default branch's landing commit (x.landed_via = bc.landed_via).

  • This is still clock-free.

  • The seed now merges main back into release-1.x and into release-0.9. release-0.9 has no archived copy, so CVE-06 and CVE-07, which count by branch name, see a misclassification.

• Medium (performance). CVE-09 and CVE-10 re-read the cve_fix_commits view of every repository once per fix commit. A reviewer measured 199 s on synthetic data where 0.4.35 took 41 s. The queries now read the repository's fix commits once (a materialized CTE), scoped to the repository. On 300k synthetic commits with 600 fix commits in cherry-pick chains, each takes 0.08 s.

• Low. The fold was two-level, not transitive. Fix commits of one CVE are now grouped as connected components of shared patch-ids and resolved cherry-pick links (a recursive CTE). A chain of conflict-adjusted cherry-picks (c2, c15, c20 in the seed) is one change.

• Nits: the seed's devel decoy window is now a state labor could produce (it ends before the last labor run, with two files at their real blobs), and the seed header comments are current.

Review round 5 (3 Low), all fixed:

• CVE-09 and CVE-10 collapse each patch-id group into one node before closing the cherry-pick graph. The closure over every pair of same-patch commits was cubic in the group's size: 1000 copies of one fix ran over 180 s, and now take 0.7 s. A 500-link chain of conflict-adjusted cherry-picks takes 3.3 s.

• CVE-04's backport class is now stated as the complement of shares history.

• CVE-04 documents a known limit. A git pull merge pushed to the default branch makes a branch cut at the earlier tip read as backport. That graph is the same as a backport merged up.

New queries (user request, after the first run on real data). In apache/camel most "CVE fixes" were dependency upgrades, and a generated Yarn file was the most complex file.

• CVE-16 and CVE-17 classify each CVE's fix by the paths its commits touched:

  • code: source outside test paths;

  • test only;

  • dependency or build: pom.xml, Gradle files, lockfiles, package.json, go.mod, ...;

  • docs or config only.

  CVE-17 counts the CVEs in each class.

• LAB-13 splits each window's complexity into generated or vendored files and hand-written code, by path patterns kept in one VALUES list. LAB-14 lists the most complex files and whether the patterns catch them.

• The seed has a fix of every class (a test known only by its directory, one known only by its name, code and its test in one change) and a generated file in the third window.

• The analysis guide says the CVE ids come only from commit messages, with no advisory database, and that many are dependency upgrades.

Review round 6 (1 High, 2 Medium, 4 Low), all fixed:

• High. CVE-16 and CVE-17 called any file ending in "it" or "test" a test. The test-name rule matched case-insensitively, so Commit.java, Submit.java, limit.c and Latest.java were tests. The capitalised suffixes (FooTest, BarIT) are now matched case-sensitively after a lowercase letter or digit. The seed fixes CVE-2024-0008 in src/Submit.java.

• Medium. LAB-13 and LAB-14 treated build-output directory names at any depth as generated. Examples are LLVM's lib/Target/, Bazel's .../build/ packages and Go's src/go/build/. target, build, out, dist, gen and deps now count only at the repository root, and the seed has a hand-written src/main/java/com/x/build/Builder.java.

• Medium. Template and UI source counted as docs or config. .html, .vue, .svelte, .jsp, .erb, .twig and .hbs are now code, except under doc/ or docs/.

• Low:

  • More build and dependency files: Makefile, CMakeLists.txt, meson.build, configure.ac, go.work, Gradle wrapper and convention files, buildSrc/, sbt project/*.scala, setup.py, requirements/, Dockerfile variants.

  • A dependency change now outranks a test change, so "bump pom.xml and adjust a test" is dependency or build (renamed from dependency or build only).

  • Generated-file patterns add redis's root deps/, Kubernetes' zz_generated, .pb.cc, .pb.h and .pb.gw.go, and drop the case-insensitive Generated name rule, which caught hand-written names.

  • LAB-14's reading covers false positives as well as misses.

  • The seed has a config-only fix and a merge naming a CVE (no files).

Review round 7 (1 Medium, 6 Low), all fixed:

• Medium. Thrift output counted as hand-written. LAB-13 and LAB-14 now treat src/gen/ and gen-java/, gen-javabean/, gen-cpp/, gen-py/ (and thrift-gen-java/ and the like) as generated. Hive commits about 1,600 such files and Accumulo about 100. The seed has core/src/main/thrift-gen-java/X.java.

• Low:

  • CVE-16/17's test-name rule now accepts an uppercase acronym before Test (HTTPTest.java) and needs a name before the suffix, so a production class named Test.java is code. IT still needs a lowercase letter or digit before it.

  • spec/ counts as a test directory only outside a src/ tree (RSpec, Jasmine), so Hive's production .../model/spec/ package is code.

  • Vendored code (vendor/, third_party/, node_modules/) counts as a dependency change, so a Go bump with a committed vendor/ is not a code fix.

  • Dockerfiles are matched after code, by name (Dockerfile, Containerfile, Dockerfile.*, *.dockerfile), so dockerfile.go is code.

  • More code extensions: .jelly (Jenkins views), .tag, .tagx, .gsp, .xsl, .xslt, .ejs, .pug, .jinja, .j2, .bat, .cmd, .ps1.

  • The seed now exercises the capitalised test-name rule (src/OverflowTest.java, outside any test directory), the docs/ rule (docs/conf.py), vendored code, a Dockerfile variant and dockerfile.go.

• TestDocsLinksResolve skips fenced code blocks and inline code, where GitHub renders no links. A regular expression such as [a-z](Test|IT) in a SQL block is not a link.

Review round 8 (5 Low, 1 nit), all fixed:

• CVE-16/17 count a checked-in minified library (*.min.js, *.min.css) as a dependency, so "upgrade jQuery for CVE-..." is not a code fix.

• CVE-16/17 recognise test modules with code under src/main/ as tests: itests/ (Hive), test-infra/ (Camel), test-scripts/, *-end-to-end-tests/ (Flink), *-e2e-test(s)/, integration-test(s)/.

• CVE-16/17 count Cassandra's .build/ (its dependency versions) as build.

• The seed gains cases for the round-7 and round-8 rules: src/HTTPTest.java, spec/models/user.rb, a class named Test.java, a production src/.../spec/ package, a minified jQuery, Flink's test-scripts/, .build/parent-pom-template.xml and a Thrift file under src/gen/. That still left some alternatives untested; round 10's TestPathRules covers them all.

• TestDocsLinksResolve fails on a code fence left open (which would hide every later link) and when it checked no link. TestWithoutCode pins what the link scan keeps and drops.

• LAB-13/14 add Thrift's other outputs (gen-nodejs/, gen-perl/, gen-netstd/, gen-csharp/, gen-erl/, gen-lua/).

Review round 9 (3 Low, 2 Info): no query was wrong. The test did not yet back round 8's claim that a deliberately broken version of each rule fails it.

• Seed coverage. More seed paths: every test-module directory, an IT-named class, a .min.css alone, and a gen-py/ file. Round 10 found alternatives still untested and moved per-alternative coverage to TestPathRules.

• Precedence. Each ranking of change and CVE classes is exercised:

  • a code fix with a pom.xml (code outranks build);

  • a CVE with a code change and a build change;

  • a CVE with a dependency change and a test change;

  • a CVE with a test change and a docs change.

  Each of 26 such mutations fails the test.

• Docs. CVE-16's table says build and minified files under docs/ count as build.

• Trade-offs accepted, with a comment at each rule:

  • The capitalised test-name rule takes a few production classes named like tests (Groovy's ASTTest, Spark's ANOVATest).

  • src/gen/ takes Hive's hand-written vectorization templates, which scc scores 0.

Review round 10 (1 Low, 2 nits, 2 Info): no query or golden value was wrong.

• Low. The CHANGELOG still overstated the seed's coverage of the path rules. Twelve alternatives could be deleted with the golden test passing. New TestPathRules reads the CVE-16/17 classifier and the LAB-13/14 generated-file list from the docs, and checks them against a table of paths. The table has one path per alternative (every build file, test layout, code extension and generated pattern) plus the traps the reviews found. The test also fails if the copies of a rule in the two queries that share it differ. Each of the round's 12 surviving mutations, and 7 more, fails it.

• Nits:

  • this section's intro names its whole scope;

  • the seed says its line counts are schematic.

• Info, taken:

  • docs/ counts as documentation only outside a src/ tree, so Go's own src/go/doc package is code;

  • the end-to-end layouts e2e/, e2e-test(s)/, end-to-end-tests/ and *.e2e-spec.ts count as tests.

0.4.35 (unreleased)

Review round 2 of the analysis queries and docs (0.4.34): 2 Medium, 8 Low, all fixed.

• Medium. A backport merged up into the default branch was no longer a backport. CVE-04, CVE-06 and CVE-07 treated a fix commit as shared with the default branch if the default branch held it at all. A cherry-pick later merged up into the default branch (git-flow, or a maintenance branch merged into master) became shares history and dropped out of the backport statistics. A fix commit now counts as shared only if it reached the default branch no later than this branch. A fix made on a branch first and merged up later is a backport with a negative delay.

• Medium. LAB-03 reported a rise as the largest fall on a steadily rising series. It now filters by sign: largest_rise and largest_fall are NULL when no step went that way.

• Low:

  • LAB-10, LAB-11 and X-01 take the default branch through the same single pick as every other query, so a stale second default row is not counted too.

  • LAB-09 breaks ties by path.

  • DQ-09 reads only the current default branch's series.

  • CVE-09 and CVE-10 fold a cherry-pick of a fix into its source even when conflicts changed its patch-id.

  • LAB-07's content_changed is NULL for the first window.

  • Corrected wording: LAB-11's correlation needs three windows; CVE-04's classes.

  • The seed's ahead and behind counts now follow its graph.

  • The query test pins DateStyle, IntervalStyle and extra_float_digits, so another cluster's defaults cannot change the rendered results.

• Test fix. TestAnalysisQueriesRun checked the golden file against the subtests that ran, so a run filtered to one query failed on every other result. That made every single-query mutation check since 0.4.34 meaningless. It now checks against the documented queries. Every earlier mutant was re-run against the full test, each counted only if its own query's subtest failed, and all are caught.

• Seed. It gains:

  • a backport merged up into the default branch;

  • a stale second default row;

  • a conflict-adjusted cherry-pick with its own patch-id.

  The golden results were regenerated, and every changed value was checked by hand.

0.4.34 (unreleased)

• Documentation set and analysis queries (user request). docs/ is now organised by reader, with docs/README.md as its index:

  • overview.md: purpose, research questions, pipeline, key concepts, design rules, components;

  • operations.md: install, run, recompute, stop, upgrade with migrate, troubleshooting, removal;

  • schema.md: the relational model of aveloxis_cve as Mermaid diagrams. It has an overview, attribute diagrams for commits and branches, rolling labor and operations, and the views' lineage. Declared foreign keys are solid lines; joins by value with no foreign key (repo_id, blobs, SHAs) are dotted. A chart shows what comes from Aveloxis.

  • development.md: build, test, rules, tripwires, schema changes, review;

  • analysis/: the research questions mapped to 41 SQL queries.

    • CVE-01 to CVE-15: fixes, propagation, backport delay against the default branch, missing fixes, reverted fixes, authors, footprint, GHSA-only advisories.

    • LAB-01 to LAB-12: window series with deltas, density, trend, languages, files, churn, and how each step's change splits into added, removed, modified and remeasured files.

    • X-01 to X-05: CVE fixes in their complexity context.

    • DQ-01 to DQ-09: data quality, including parity with Aveloxis's repo_labor.

• Tests.

  • TestAnalysisQueriesRun runs every documented query in a READ ONLY transaction against a migrated schema seeded by internal/db/testdata/analysis_seed.sql. It requires rows from each, and its full result to equal the hand-checked internal/db/testdata/analysis_golden.txt (-update rewrites it). The seed holds the cases that make naive queries double-count: a commit naming two CVEs, archived copies of a live branch, a fix completed on a branch by its own commit (review round 1).

  • TestAnalysisGuideMapsEveryQuery checks the question map links every query.

  • TestSchemaDiagramMatchesSchema checks schema.md against the migrated catalog: every table, column, type, key marker and nullability; every foreign key as a solid relationship, and no solid relationship that is not one; every view dependency.

  • TestDocsLinksResolve checks every relative link and anchor in docs/ and the README.

• docs/data.md now says it covers migrations 0001-0019 (it already documented 0019's stop_requests).

• Review round 1 (4 Medium, 8 Low, all fixed). Medium:

  • CVE-10 counted a commit that fixes several CVEs once per CVE;

  • CVE-06 and CVE-07 counted a branch again for each archived ref of it;

  • backport delay was measured from fix commits shared with the default branch, not from the branch's own fix;

  • the query test only required rows, now golden results.

  Low:

  • CVE-08's column name;

  • DQ-04 pending versus linked references;

  • the first-window rule with a stale first window (LAB-11, X-04);

  • LAB-01, CVE-04 and CVE-13 wording;

  • two operations.md claims;

  • a stale range in data.md;

  • seed parents and authors;

  • three gaps in the doc tests (unparsed diagram lines, indented code fences, SQL blocks outside a query section).

0.4.33 (unreleased)

• aveloxis-cve migrate (user request): the command for schema changes after the first installation, as in Aveloxis. init-db still installs.

  • migrate applies the embedded migrations not yet in the ledger (db.Upgrade), under the same polled lock as init-db.

  • It never installs. On a database without aveloxis_cve.schema_migrations it fails with db.ErrNotInitialized, says to run init-db first, and creates nothing, so a config naming the wrong database gets no schema. A schema without its ledger, such as one a DBA created for sql/roles.sql, counts as not installed. The check runs under the migrate lock.

  • init-db still installs, and still applies pending migrations on an installed schema; it now refuses a newer schema (below).

  • A failed lookup of the ledger is an error, never "not installed": a role without USAGE on aveloxis_cve gets the permission error (42501), not a hint to run init-db (review round 1; TestUpgradeReportsAFailedLedgerLookup).

  • migrate and init-db refuse a schema that a newer aveloxis-cve migrated (db.ErrSchemaNewer), apply nothing, and say to upgrade aveloxis-cve, as doctor does. Before, they printed "applied 0 migration(s)" at this binary's version and exited 0 (review round 1; TestMigrateRefusesASchemaNewerThanTheBinary, and TestDoctorFailsWhenSchemaIsNewerThanBinary now runs both commands).

  • migrate skips the install steps (CREATE SCHEMA, the ledger's CREATE TABLE IF NOT EXISTS) entirely. Before, a role with USAGE but not CREATE on aveloxis_cve failed on an up-to-date schema with "creating migration ledger: permission denied", and a schema dropped between the check and those steps would have been installed afresh. Now that case fails the ledger read (review round 2; TestUpgradeSkipsTheInstallSteps).

  • The hints for a schema older than the binary now say to run aveloxis-cve migrate: doctor's version check and the missing-stop_requests error (42P01). Doctor's hint for a database without the schema still says init-db.

  • Tests: TestUpgradeNeverInstalls, TestUpgradeAppliesPendingMigrations (internal/db), TestMigrateAppliesChangesAfterInitDB (internal/cli). TestStopRequestsOnAnOlderSchemaPointAtMigrate (renamed) and TestAFailedStopCheckFailsTheCommand now expect the migrate hint.

0.4.32 (unreleased)

• aveloxis-cve stop: graceful shutdown (user request). It records a stop request in the new table aveloxis_cve.stop_requests (migration 0019), prints its id and time, and returns without waiting for the commands it stops.

  • What it stops. Every sync, labor, resolve and run of 0.4.32 or later that is already running (connected) at that moment, on any machine using the database. Each finishes and records the repositories it is working on, starts no other, and exits 0 unless something failed.

  • What it reports. The rest are reported in one line, N repositories not started: stop requested (pipeline.ErrStopped). run also starts none of its later steps (run: stop requested; resolve and labor not started). resolve --all exits non-zero and names every repository to rerun, including any skipped as busy, since nothing redoes --all later.

  • How a command checks. Once connected, each command reads its stop marker, the highest request id recorded so far (store.StopMarker). Before each repository it checks for a later id (store.StopRequestedAfter), in the dispatch loops of SyncIDs and RunIDs (Syncer.Stop, Labor.Stop) and in the resolve loops of resolve and run. Ids rather than clock times, so a step of the database server's clock cannot hide a request (review round 1). A request recorded before a command read its marker does not stop it, and older binaries never check.

  • When the check fails. A failed check is not "no request". The command starts nothing more and fails, naming the check (pipeline.ErrStopCheck). run then starts neither resolve nor labor, even though a later check might succeed (review round 1). On a schema before 0019, the error says to run init-db (SQLSTATE 42P01; review round 1). A signal during a check between two resolves is reported as an interruption (run: interrupted, resolve: interrupted; review round 1).

  • Recorded one at a time. RequestStop locks the table (SHARE ROW EXCLUSIVE) before its INSERT. The lock conflicts with itself and with writers, never with the commands' reads; a stop can also wait briefly behind a VACUUM or ANALYZE of the table. So ids commit in the order they are taken, and a request with a smaller id can no longer commit after a command read a larger one as its marker and then fail to stop it (review round 2).

  • Signals. Ctrl-C and SIGTERM are unchanged: they interrupt at once. A signal during a stop check is an interruption whatever the check found, in resolve and run's resolve step (review round 2) as in sync and labor, which are now pinned by a test too (round 3). It no longer exits 0 as a stop.

• docs/data.md documents stop_requests, and the README lists stop.

• Tests:

  • internal/store: TestStopRequests (pins both sides of the marker boundary), TestStopRequestsReportAFailedLookup, TestStopRequestsOnAnOlderSchemaPointAtInitDB and TestStopRequestsAreRecordedOneAtATime (a request waits for one being recorded; a marker read does not).

  • internal/pipeline: TestNoRepositoryStartsAfterAStopRequest, TestTheStopCheckRunsBeforeEveryRepository, TestAFailedStopCheckStartsNothingMore and TestAStopCheckCutByTheSignalIsNotStarted.

  • internal/cli:

    • TestStopRecordsARequest, TestAStopBeforeACommandStartsDoesNotStopIt, TestCommandsStopBeforeTheirNextRepository and TestStopCheckFailsWithoutItsStartMarker;

    • TestResolveAllStoppedFails and TestResolveAllStoppedNamesBusySkips;

    • TestRunStoppedAfterSyncStartsNothingMore, TestRunStoppedDuringResolve, TestRunStoppedBeforeLabor and TestRunEndsAfterAFailedStopCheckDuringSync;

    • TestAFailedStopCheckFailsTheCommand, TestAnInterruptBetweenResolvesIsAnInterruption and TestASignalDuringAStopCheckThatFindsARequestIsAnInterruption.

  • internal/pipeline also has TestASignalDuringAStopCheckThatFindsARequestIsNotStarted (round 3).

  • Test seams: afterStart and wrapStopCheck.

0.4.31 (unreleased)

Fixes from review round 1 of repos add --url-file (0.4.30, 35a35f3).

• An interrupt stops --url-file while it waits for input (Medium). Run catches SIGINT and SIGTERM, so a read of stdin from a terminal or a stalled pipe, or the open of a FIFO with no writer, could not be stopped by Ctrl-C or kill: only EOF or SIGKILL ended it. Opening and reading now run on their own goroutine, and an interrupt ends the wait with an error that carries it (--url-file stdin: context canceled; nothing added; procx.Interrupted).

• --url-file is repeatable. Given twice, it used to read only the last file, silently. Every file is now read in order, and each error names its own file and line. --url-file - may appear only once, since stdin can be read only once.

• The line length limit is stated exactly and names the line. A line must be shorter than 65,536 bytes (bufio.MaxScanTokenSize), counting a trailing CR; 0.4.30's entry said "longer than 64 KiB". The error now names the line (--url-file urls.txt line 2: bufio.Scanner: token too long; nothing added).

• A line is trimmed exactly as --url is (space, tab, CR and LF, as source.RepoByURL's btrim), no longer with all Unicode whitespace. A URL with a trailing no-break space now fails the same way from a file as it does as --url; 0.4.30 accepted it from a file only.

• One trim set for --url and --url-file (review round 2). It was spelled twice, as a SQL literal in source.RepoByURL and a Go constant in the CLI, and no test tied the two together, so they could drift apart again unnoticed. Now source.URLTrim is its only definition: RepoByURL's queries take it as a parameter (btrim(..., $2)), and the file reader trims it too.

• A URL that is not found is quoted in the error (review round 2). The message trimmed all Unicode whitespace from the URL it showed, so a URL with a trailing no-break space was reported as not found while showing one Aveloxis does have. It now shows the URL trimmed as the lookup trims it, quoted (repository not found in aveloxis_data.repos: "https://github.com/a/one\u00a0"). This applies to --url as well.

• --url-file - given twice is refused before anything is read (review round 3). The check ran only after the first - had been read, so someone typing URLs at a terminal typed them all, pressed Ctrl-D, and only then was refused.

• Tests no longer hang on a regression, and pin more (review round 3):

  • the interrupt helper also ends when the code under test returns without reading, so such a regression fails the test instead of hanging until go test -timeout;

  • a CLI test checks that an unknown URL is named by its own file and line when several files are given;

  • TestRepoByURLTrimsExactlyURLTrim lists the four trimmed bytes itself instead of taking them from URLTrim, so shrinking URLTrim fails it.

• The interrupt tests pin that the interrupt, not the end of input, stopped the read (review round 4). They passed against a read that ran to the end of its input and only then checked for the interrupt, because each helper ended the input a second after the interrupt. They now fail when the call returns only after that.

  • The FIFO subtest ends its writer by opening the FIFO read-write, which never blocks, so it no longer hangs when the code under test never opens the FIFO.

  • The several-files check now puts the unknown URL in a file other than the last one, so a CLI that names the last file fails it.

  docs/data.md now says exactly which characters RepoByURL trims.

• Tests (internal/cli): TestReadURLFileTrimsOnlyWhatRepoByURLTrims, TestReadURLFileLineLengthLimit (pins both sides of the limit), TestLoadURLFiles (order, stdin twice), TestLoadURLFilesStopsWhenInterrupted (stdin and a FIFO), TestReposAddFromSeveralURLFiles, TestReposAddFromStdinStopsOnSIGINT (a real SIGINT through Run). The stdin interrupts come while the read is blocked (round 2); the FIFO's comes 100ms in. Each test also asserts that the call returned before its input ended a second later, so a read that cannot be interrupted fails the test instead of passing on the late end of input or hanging (round 4). These tests replace 0.4.30's TestReadURLFileRefusesAnOverlongLine and TestLoadURLFile. Tests (internal/source): TestRepoByURLTrimsExactlyURLTrim (every URLTrim byte is trimmed on either side, other whitespace is not, and the error quotes the URL).

0.4.30

• repos add --url-file <file> tracks every repository URL listed in a file, one per line, or read from standard input with --url-file - (user request). Blank lines, spaces around a URL, Windows line endings and a UTF-8 byte order mark are ignored. Each URL is looked up exactly as --url is (source.RepoByURL), and the flag combines with the other selectors. The add stays all-or-nothing: a URL Aveloxis does not have fails it with the file and line (--url-file urls.txt line 3: repository not found ...; nothing added). The file is read before the database is opened, so a file that is missing or unreadable, holds a line longer than 64 KiB, or holds no URL fails on its own and adds nothing.

• Tests (internal/cli): TestReadURLFileSkipsBlankLinesAndTrims, TestReadURLFileReportsAReadError, TestReadURLFileRefusesAnOverlongLine, TestLoadURLFile, TestReposAddFromAURLFile, TestReposAddFromAURLFileNamesTheLineOfAnUnknownURL, TestReposAddFromStdin, TestReposAddFromAnUnreadableURLFileAddsNothing, TestReposAddNamesURLFileWhenGivenNothing. Seam: stdinReader.

0.4.29

Fixes from Copilot review 5828551951 of PR #1 (on 3873f72, 0.4.28). Test harness only; the tool's behaviour is unchanged.

• testdb.New drops its scratch database whatever fails after creating it. It registered the drop together with closing the pool, after the pool was opened, so a failure in between (parsing the DSN, opening the pool) ended the test with its cvet_* database left behind, and such failures piled databases up across runs. The drop is now registered as soon as CREATE DATABASE returns, before anything else can fail, and closing the pool is its own cleanup, which runs first. A CREATE DATABASE that reports an error may still have made the database (its reply lost to a dropped connection), so it is dropped then too if it exists by the time the test ends, except when the name is already taken (SQLSTATE 42P04, another test's database). Declined (review round 1): waiting out a CREATE the server may still be running after the connection dropped; a timeout cancels the statement, and what a lost connection can leave is a cvet_* database.

• crypto/rand.Read never returns an error (it crashes the program instead), so the scratch database name and RandomName share one generator with no error path (review round 1).

• Tests (internal/testdb): TestNewDropsItsDatabaseWhenItsPoolFails, TestNewDropsADatabaseCreatedDespiteAnError, TestNewNeverDropsADatabaseItDidNotCreate, TestNewClosesItsPoolAndDropsItsDatabase. A fake testing.TB runs New on its own goroutine and runs its cleanups on demand, and again when the test ends; each seam registers the drop as soon as it learns the name, so a red run leaves no database either (review round 1). The pool's BeforeClose hook pins that the pool closes while the database still exists (round 1). Seams: scratchName, createDatabase and newPool.

0.4.28

Fixes from Copilot review 5819499534 of PR #1 (on 1b17341, 0.4.27).

Upgrade note: a 0.4.27 fetch left git's maintenance running detached, outside any lock, and 0.4.28's first sync of a repository removes git's lock and temporary files in its clone as leftovers. Before the first 0.4.28 sync or run, wait until no git runs on clone_dir: lsof +D <clone_dir>, run as root or as the user that runs aveloxis-cve (lsof silently leaves out other users' processes), lists no git process (it works on every git version; the detached process is git gc --auto on git before 2.47 and git maintenance run from 2.47, and it keeps its working directory in the clone; a large clone_dir takes a while to scan) (round 18). The same holds for a git gc run by hand: run it as the user that runs aveloxis-cve, under the clone's lock (docs/configuration.md, git.timeout_seconds; review round 17).

• A run that loses its repository lock keeps other runs off its clone until it stops. The per-repository lock is a database session lock: PostgreSQL releases it when the session ends (an idle timeout, a restart, a failover), while the process keeps running git. The 0.4.22 fence refuses that process's later database writes, but not its git. Until its next lock check, a fetch could go on writing into the clone (EnsureClone, Sync), or labor could go on reading its graph, trees and blobs, while a second run that took the freed lock fetched into the same clone. Now sync and labor also hold an exclusive flock on <clone_dir>/repo_<id>.git.lock for the repository's whole run (gitx.TryLockClone). The database never releases it, only Unlock or the process ending. The lock also lasts as long as any git its holder started: every git run on the clone gets the lock's file (procx.Spec.ExtraFiles), and a flock is released only once its last descriptor closes. So a process killed outright (SIGKILL, the OOM killer) leaves its still-running git holding the lock until that git exits, and no new run fetches beside it or removes its in-flight files (round 1 declined this; round 8 made it necessary, since the sync now removes git's leftover lock files). An orphaned git is no longer bounded by git.timeout_seconds, so a hung one keeps the repository busy until it exits; ErrCloneBusy's text and docs/configuration.md say so and name lsof <clone_dir>/repo_<id>.git.lock to find it (round 9). A first clone killed with its process leaves repo_<id>.git.tmp-* beside the clone; the next clone of that repository removes it (removeKilledClones, round 9; docs/data.md's removal steps name it, round 10). ErrCloneBusy names the lock file to run lsof on (round 10). A second run that takes the database lock but not the clone's reports ErrCloneBusy (an ErrBusy, so the CLI skips the repository as busy) and records nothing. A clone lock that cannot be taken at all (a clone_dir fault) is an error, carrying an interrupt like the database lock's, and is recorded as the run's outcome under the held database lock, so a lasting fault shows in status (review round 3). The lock file is empty and never removed (removing it while a run waits on it would let two runs lock two files); docs/configuration.md and docs/data.md say so. Concurrent runs of two installs sharing one clone_dir no longer overlap either, but sharing a clone_dir between installs is still unsupported (docs/configuration.md).

• git's automatic maintenance of a clone runs under its lock, bounded, and as a warning (review rounds 4 and 5). A git fetch ended by starting git maintenance run --auto, which git detaches by default into its own session: outside the fetch's process group, outside git.timeout_seconds, and outside the clone lock. So it went on repacking the clone after Sync returned and the lock was released, and in run a labor read could overlap it. Round 4 made it run inside the fetch, but then a timeout or a signal killed it with the fetch: that failed the sync, and left objects/maintenance.lock, which git never takes as stale, so maintenance on that clone stopped for good without a word (round 5). Now:

  • the fetch starts no maintenance (--no-auto-gc);

  • Sync then runs git gc --auto as its own call (Runner.maintain), in the foreground, under the clone lock and the git timeout. gc takes gc.pid, which a later gc takes as stale once its process is gone;

  • every git call built by Runner.spec passes -c gc.autoDetach=false -c maintenance.autoDetach=false (gitx.foreground; a git ignores keys it predates). StreamFacadeFiles builds its own spec, to keep Aveloxis's git configuration, and runs only git log, which starts no maintenance;

  • a maintenance that fails or times out is a warning (SyncResult.MaintenanceErr, RepoReport.MaintenanceWarning, printed by sync and run), not a sync failure, and is not recorded as last_sync_error. An interrupt during it still fails the sync as interrupted;

  • what a killed git leaves in the clone, which git itself never takes as stale, is removed at the start of every sync and after a failed maintenance, under the clone lock (removeLeftovers):

    • temporary pack files in objects/pack (tmp_*, .tmp-*; review round 6). A repack killed by the timeout leaves the pack it was writing, as large as the clone, which git removes only once two weeks old and only when a gc gets that far, so a repack that never fit the timeout left one more on every sync; and git reads a leftover .tmp-*.idx as a pack;

    • git lock files (*.lock) at the top of the clone, under refs/, and in reftable/, objects/, objects/info/, objects/info/commit-graphs/ and objects/pack/ (round 8). A gc killed mid-way leaves, for example, commit-graph.lock, which fails every later gc, or packed-refs.lock, which fails a fetch that prunes. Also packed-refs.new, the fixed-name temporary file of a packed-refs rewrite, opened O_EXCL: left by a killed git, it fails every later rewrite (a pruning fetch, an archiving update-ref, gc's pack-refs; round 19). A ref name cannot end in .lock, and the clone's own lock file is beside the clone, not in it;

    • git's own .keep markers in objects/pack (fetch-pack <pid> on <host>, receive-pack ...): a fetch killed after keeping its pack and before updating refs leaves one, and that pack is then never repacked. Any other .keep (an operator's) stays, and so does one that is not a regular file, which is never read (a FIFO would block); a .keep that cannot be read is an error (rounds 12 and 13);

    • names are read with os.ReadDir, never globbed. The round 6 version globbed the clone's path, so a clone_dir holding [ failed every sync, and one holding * matched other directories (round 8).

    A repack longer than git.timeout_seconds still never completes; docs/configuration.md says to raise the timeout or run git gc by hand.

  Tests:

  • TestSyncMaintenanceIsBoundedBestEffortAndRunsAgain: a pre-auto-gc hook that outlasts the timeout. The sync succeeds and reports the maintenance error, and the next sync runs maintenance again.

  • TestRemoveLeftoversReportsAnUnreadableRefsDirectory (through the readDir seam) and TestEnsureCloneRemovesItsKilledPredecessors (round 9; since round 10 under the clone's lock, whose file, a sibling too, must survive). TestEnsureCloneReportsKilledClonesItCannotRemove (the readDir and removeAll seams; round 10).

  • TestSyncRemovesStaleGitLockFiles (on a reftable clone and a files-backend one, made with --ref-format=files where git has it; since round 19 the files one packs its refs and has a leftover packed-refs.new while upstream deletes a branch), TestSyncRemovesLeftoversUnderAGlobLikeCloneRoot (clones[x, clones[1], cl*nes) and TestRemoveLeftoversReportsAnUnreadableDirectory (round 8).

  • TestSyncRemovesTemporaryPacksLeftBehind (each sync removes them before it fetches) and TestSyncInterruptedDuringMaintenanceFails (round 6). TestSyncMaintenanceIsBoundedBestEffortAndRunsAgain's hook also leaves a temporary pack, which the cleanup after the failed maintenance must remove; TestSyncWarnsOfLeftoversAFailedMaintenanceCouldNotRemove puts what that cleanup cannot remove in the warning (round 11).

  • TestEnsureCloneRemovesWhatAKilledCloneLeft (a clone whose deferred cleanup is skipped, as a SIGKILL skips it, through the removeAll seam: the next clone removes the very directory it left; killedClonePrefix names it in both places) and TestSyncRemovesKeepMarkersAKilledFetchLeft (round 12). TestRemoveLeftoversReportsAnUnreadableKeepAndSkipsNonFiles (a FIFO .keep, and through the openFile seam a .keep that cannot be opened, one that cannot be read, and one that vanishes before it is opened; rounds 13 and 14). The maintenance warning no longer promises a retry; its doc says when the next sync fails instead (a leftover it could not remove; round 12).

  • TestTryLockCloneCleansItsPath (round 11): repo_1.git/ locks repo_1.git.lock beside the clone (CloneLockPath, also used for ErrCloneBusy's lsof hint), not a .lock inside it, which the removals would delete while held.

  • TestSyncReportsAFailedMaintenanceAsAWarning (pipeline).

  • The one-line check in TestReportersClassifyInterruptions (CLI); since round 15 it also checks that the warning is printed when a later step of the same sync failed, and since round 16 that the default-branch warning is too (it was printed only on success).

  • TestRunnerKeepsMaintenanceInTheForeground pins the two -c pairs on the arguments, before the subcommand: no hook can tell foreground from detached, since git runs pre-auto-gc before it detaches (round 7). TestSyncFailsWhenATemporaryPackCannotBeRemoved (round 7): the sync fails before it fetches.

  • Each of these fails a test: dropping --no-auto-gc, emptying foreground, skipping maintain, failing the sync on it, warning on an interrupt, not passing or printing the warning, skipping either leftover removal, any one of its scans or its .tmp-* pattern, or dropping a removal or read error.

• Labor.RunRepo holds the labor gate. It ran the repository with no gate, so it could measure into and write the shared scc cache while labor --retry-uncounted held the gate exclusively and was rebuilding that cache. The CLI only uses RunIDs, which takes the gate, so this affected callers of the exported API (the tests). RunRepo is now RunIDs for one repository: it takes the gate shared (or uses Labor.Gate), reports busy while --retry-uncounted runs, and starts nothing once a signal has come. Every labor run now holds the gate, so held.Alive's check for a missing gate is gone.

• Tests:

  • TestCloneLockOutlivesItsHolderWhileItsGitRuns: a helper process (TestCloneLockHolderProcess) takes the lock and syncs, and is SIGKILLed while its gc's hook runs. The lock stays held until that orphaned git exits. Since round 13 it uses a real kill: the round 8 version released the lock in-process and raced under -race and TestGitCallsOnAHeldCloneGetItsLock (each way gitx builds a git call; round 8).

  • TestTryLockCloneIsExclusive, TestTryLockCloneReportsAnUnusableRoot, TestTryLockCloneReportsAnUnopenableLockFile and TestTryLockCloneReportsAFailedFlock (internal/gitx; the last swaps the flock seam: ENOLCK is an error, not busy, and EINTR is retried; round 7).

  • TestSyncKeepsItsCloneWhenItsLockIsLost and TestLaborKeepsItsCloneWhenItsLockIsLost end a run's lock session once it holds the lock, then start a sync of the same repository: it is refused with ErrCloneBusy and records no outcome, and the first run stops with ErrLockLost. TestLaborIsRefusedWhileASyncThatLostItsLockHoldsTheClone is the other way round: a labor run started while a sync that lost its lock holds the clone is refused, and writes no window, cache row or outcome (review round 2).

  • TestAFailedCloneLockIsAnErrorAndIsRecorded (a clone root under a regular file; rounds 1 and 3), TestCloneLockFailureAfterTheSignalCarriesTheCancellation (a new clone test-hook stage, between the two locks; round 3) and TestErrCloneBusyIsErrBusy (round 1).

  • TestLaborRunRepoWaitsOutARetryUncounted: while the gate is held exclusively, RunRepo reports busy and writes no window and no cache row.

  • TestLockFailureAfterTheSignalCarriesTheCancellation's labor half now signals as the lock is taken (as its sync half does), since RunRepo starts nothing after a signal.

0.4.27 (unreleased)

Fixes from review rounds 13-19 (of 0.4.26, committed as 826c32b). Tests and one comment only; no behaviour changes.

• The tie between doctor's index list and the lookup query checks what the plan does, not how the SQL is written. The 0.4.25 test read the SQL text, and from 0.4.26 it matched how the plan printed each lower(...). Each of review rounds 9-16 found another spelling that slipped through, and doctor stayed silent while the step read a table whole:

  • a wrapped lower() (IS TRUE, COALESCE, CASE, an OR);

  • upper(), ILIKE or a plain = on the key;

  • an alias on the key CTE, a sublink, or a materialized or inlined CTE over it;

  • a subquery computing lower();

  • a step inside the key CTE.

  TestResolveAuthorsReadsAveloxisOnlyThroughIndexes replaces that test, TestEmailIndexedMatchesTheQuery, and TestResolveAuthorsEmailStepsCanUseLowerEmailIndexes:

  • In its scratch database it creates the Aveloxis indexes the query is written to use, as Aveloxis 0.29.141 has them (aveloxisIndexes): idx_commits_repo_hash and the three lower(email) indexes in emailIndexed.

  • It plans the query generically (PREPARE, force_generic_plan), with seq scans, hash joins and merge joins off, over 20,000 analysed rows per table. So each step must look its rows up by key: on empty tables, reading a repository's commits by repo_id and then filtering looked cheapest.

  • It fails (aveloxisScans) when a scan of any Aveloxis schema's table has:

    • no index condition (a seq scan, or a whole-index scan);

    • a range condition;

    • a condition that does not fix every key column of its index by equality. The key columns are read from the catalog (aveloxisIndexKeys), so commits by repo_id alone, which reads the whole repository, fails (review round 17).

  • It also fails when a listed index is used by no step. Each email index is named by its table and column, and a duplicate emailIndexed entry is an error. In round 18's version two entries on one table shared a name, so one index was never created. That let a stale entry pass, and made a correct new step on the same table fail with a misleading message.

  • So a new email step on an unlisted column, or in a form no index serves, fails it whatever its spelling, and so does a stale entry in doctor's list.

  • The planPairs text parser and its helpers are removed.

• The empty-email guards are pinned (review round 18). Each email step skips an empty key (k.email <> ''), so an author with no email cannot resolve to a contributor whose email, alias or identity email is empty too. That would fabricate an identity. No test covered it. TestResolveAuthorsNeverMatchesAnEmptyEmail does, and removing any one of the three guards fails it. It also stores a whitespace-only identity email, so removing the trim of the author's email fails it too (review round 19).

• The test file is gofmt-clean. gofmt rewrites '' in a doc comment to a typographic quote, which also broke the ASCII rule.

• Tests:

  • TestAveloxisScans covers the accepted cases: index scans on their full key, and bitmap scans. It covers twelve refused ones:

    • a seq scan, a whole-index scan, a bitmap scan without a condition, a sample scan;

    • another Aveloxis schema;

    • a range: an index scan's range fixes no key by equality, and a bitmap scan's range recheck is refused as a range. The key-equality pattern is anchored on both sides, because >= key and <= key end in = key (review round 19). Two range cases whose right side spells the key are added;

    • part of a key, as an index scan and as a bitmap scan;

    • an index the catalog does not describe.

  • 37 query mutants each fail TestResolveAuthorsReadsAveloxisOnlyThroughIndexes: every bypass review rounds 9-19 found, plus the earlier ones. So do a stale emailIndexed entry (on its own table or on one already listed) and a duplicate entry. A correct new step on a listed table, with its entry, passes.

  • identity.go's comment now states the rule: every step looks its rows up through an Aveloxis index, and every email column is listed in emailIndexed.

0.4.26 (unreleased)

Fixes from review round 9 (of 0.4.25, committed as f86a815). Tests only; no behaviour changes.

• The tie between doctor's index list and the query reads PostgreSQL's plan, not the SQL text. The 0.4.25 text parser kept the last table bound to each alias, and recognised only one way of writing a binding. A new email step that reused an alias was therefore credited to an existing pair, and doctor stayed silent about its index. This happened whatever the binding's form: quoted names, a derived table, another schema (review rounds 9 and 10).

  • TestEmailIndexedMatchesTheQuery now prepares the query, plans it generically, and reads the pairs from EXPLAIN (VERBOSE, FORMAT JSON) (planPairs). The planner resolves every alias to its schema and table, renames reused ones, and prints each expression in one form. The plan is generic because a plan for fixed values could prune a step whose condition is false for them (round 11).

  • Every plan expression on the email key must be lower(alias.column) = key (either way round) on an aveloxis_data table, or the key's <> '' filter. Anything else that mentions the key or calls lower() fails the test (rounds 11 and 12). That covers upper(), ILIKE, a plain =, >=, an OR, a wrapped lower() and a COLLATE.

  • So a new email step fails the test unless emailIndexed lists its column and its predicate is one a lower(column) index serves.

  • The test runs twice: without the indexes, as Aveloxis before 0.29.141, and with them and seq scans off. The second pass is where the plan reads them (Index Cond).

  • Fifteen mutants each fail it:

    • a new step written in uppercase;

    • a reused alias with a quoted name, a quoted schema or a derived table;

    • another schema;

    • an unqualified table, which PostgreSQL refuses;

    • lower(btrim(...));

    • a step gated on a parameter;

    • btrim(lower(...)) and a COLLATE on a listed column;

    • upper(), ILIKE and a plain = on the key;

    • key >= lower(...);

    • an OR.

• A failure while reading the index check's rows is pinned as an error. pgx reports a query that fails while it runs (a statement timeout, a cancel mid-stream) at rows.Err, not at Query. The cancelled-context test reached only the Query error arm, so returning "nothing missing" from the other arm passed every test.

• Tests:

  • TestPlanPairs: duplicates count once, and the key's output columns and <> '' filter are accepted. Every other form of matching the key or calling lower() is an error, twelve cases in all.

  • TestEmailIndexedMatchesTheQuery is now DB-tier (it plans the query on the fixture).

  • TestMissingEmailIndexesReportsAFailureWhileReading: a DB whose rows fail mid-stream.

0.4.25 (unreleased)

Fixes from review round 6 (of 0.4.24, committed as 52d230a).

• A mistyped --url no longer shows its credentials. RepoByURL's errors redacted only well-formed URLs (scheme://user@...). "Not found" is what a typo gives, so a mistyped URL such as https//alice:token@host/x was printed whole. A single URL an operator gave is now redacted by one rule, procx.RedactURL: everything before its last @ is hidden, keeping a leading scheme. It serves both RepoByURL and gitx.ValidateURL's refusals, which used their own copy of it.

• doctor's index check is described as what it is. 0.4.24 said an index counts "only if the planner can use it". The rule is narrower: the index requested of Aveloxis (valid, non-partial, btree led by lower(column) or hash, in the column's collation). Some indexes the planner can use are reported missing, such as a partial index whose predicate the query implies, a BRIN index, or lower(column) as a later key. That is the conservative side: at worst a needless warning.

• The warning says who creates the indexes. Only Aveloxis creates indexes in its schemas, and Aveloxis 0.29.141 creates all three (idx_contributors_email_lower, idx_contributors_aliases_email_lower, idx_contributor_identities_email_lower, on exactly lower(column)). The warning tells the operator to upgrade Aveloxis to 0.29.141 or later, never to create the indexes by hand. docs/data.md says the same.

• One list ties doctor to the query. The plan test builds its indexes from emailIndexed, the list doctor checks. A new test requires that list to equal the (table, column) pairs resolveAuthorsSQL matches as lower(column), so a new email step cannot leave doctor silent. It reads every lower( call in any case, spacing, quoting or cast, resolves each alias to its aveloxis_data table, and fails on a call it cannot read. A column used twice counts once (lowerPairs, tested on its own). Mutants of a new uppercase step, an unqualified column and a step moved to another table each fail it.

• Tests:

  • TestRedactURL

  • RepoByURL with mistyped URLs

  • TestEmailIndexedMatchesTheQuery

  • TestMissingEmailIndexesReportsAFailedCheck: a cancelled context.

  • TestEmailIndexCheckReportsAFailedCheck: all three of doctor's outcomes, including the failed-check WARN.

  • A BRIN index in TestMissingEmailIndexes, which pins the access-method rule.

0.4.24 (unreleased)

• doctor warns when Aveloxis lacks the lower(email) indexes that author resolution needs: lower(cntrb_email) on contributors, lower(alias_email) on contributors_aliases, and lower(email) on contributor_identities (requested of the Aveloxis team on 2026-11-08). The warning names each missing one. source.MissingEmailIndexes reads only the catalog. An index counts only if the planner can use it for the query's predicate: valid (not a failed CREATE INDEX CONCURRENTLY), not partial, btree with lower(column) as its first key or hash, and in the column's collation. A failed check is reported as a warning, never read as "missing".

• repos add --url finds a repository by its verbatim URL when a twin exists (review round 5 of 0.4.23). Since 0.4.23 both sides were normalised before the first comparison. Aveloxis can hold .../b and .../b.git as separate rows, and then neither could be found by its own URL: both failed as "case variants". Now a verbatim match (surrounding whitespace aside) wins first, then the normal forms are compared exactly, then case-insensitively.

  • The two sides are normalised by one SQL expression (storedURL, givenURL), no longer by Go on one side and SQL on the other, which trimmed different whitespace.

  • The ambiguity error now says the matches differ "in case or a trailing / or .git".

  • The URL quoted in these errors has its credentials redacted.

• A refused URL's leading scheme is kept, and pinned by a test. The branch in redactRefused that could never run is removed.

• 0.4.23's notes on repairing data from earlier versions are removed: no deployment ran an earlier version.

• Tests:

  • TestMissingEmailIndexes: partial, invalid, other-collation and not-first-key indexes do not count; btree and hash do.

  • doctor's warning and its PASS line, end to end.

  • RepoByURL: twins, .git/, a literal ., ambiguity wording, redaction, and both sides normalised alike.

  • A refused URL keeps its scheme.

0.4.23 (unreleased)

Fixes from Copilot's review of PR #1 (review 5818468944, on 6f857b1), verified against the code first.

• sql/roles.sql refuses to create the role with an empty password. With cve_password empty (the documented command with AVELOXIS_CVE_DB_PASSWORD unset), PostgreSQL created the LOGIN role with no password: a notice, not an error. The result was a silent misconfiguration: password authentication can never log the role in, and trust or peer authentication ignores passwords either way. The script now stops before changing anything. An existing role is not re-created, so a re-run needs no password, and a DBA using peer or cert authentication creates the role first.

• URL credentials no longer reach error text. procx.Error printed git's arguments and stderr verbatim. A repository URL carrying a token or password (https://user:token@host/..., which gitx.ValidateURL accepts) could therefore end up in tracked_repos.last_sync_error and the CLI output when a clone or fetch failed. procx.Redact now replaces every URL's userinfo with <redacted>, in the arguments and in stderr. It redacts up to the last @ before the URL ends, so a password containing @, /, ? or # is hidden whole. A URL with no userinfo whose path contains @ is redacted up to that @ as well, and a credential with an unescaped space or quote is not covered. ValidateURL's refusals redact the URL they quote too. A refused URL is malformed, so everything before its last @ is hidden, keeping only a leading scheme; this can hide a host as well. The URL itself is still stored in tracked_repos.repo_git as Aveloxis has it, and repos list shows that column.

• Author resolution no longer materializes the live-contributor set on every call. The query's live CTE was referenced three times, so PostgreSQL built it whole for each 5,000-commit sync batch. Each step now joins contributors directly on cntrb_id. The gain is modest: on a scratch copy with 300,000 contributors, a 5,000-key batch took about 200 ms instead of 245 ms, and read the same number of buffers.

  • Waiting on Aveloxis: each call still reads contributors, contributors_aliases and contributor_identities whole. The email steps match lower(column), and Aveloxis indexes none of those expressions: plain cntrb_email and the UNIQUE alias_email cannot serve lower(), and contributor_identities.email has no index at all. aveloxis-cve creates nothing in Aveloxis's schemas, so these indexes were requested of the Aveloxis team on 2026-11-08:

    • lower(cntrb_email) on contributors

    • lower(alias_email) on contributors_aliases

    • lower(email) on contributor_identities

  • Ready for them: the query's predicates are those exact expressions. A test (below) creates the three indexes in its own scratch database and checks that the plan can use each one.

• A root commit's files are recorded whatever the operator's git config says. The commit stream (git log --numstat) relied on git's default log.showRoot=true. With log.showRoot=false in a global or system config, git printed no diff for the root commit, so sync stored every repository's first commit with no files (commit_files empty, files_changed 0). The stream now passes --root. Found while writing docs/data.md. The facade stream still runs with the user's configuration on purpose, as Aveloxis's does.

• Data usage and provenance documentation (docs/data.md). It covers every table, view and column of aveloxis_cve:

  • for each table, what a row is, its keys, and which command writes and deletes it;

  • for each column, its source and how its value is derived;

  • the Aveloxis objects read, and a matrix of which command writes which table;

  • the personal data stored, and how to remove all data or one repository's.

  TestDataDocCoversEverySchemaColumn (internal/db) migrates a scratch database and fails when the document misses a table, view or column, or documents one the schema does not have.

• repos add --url finds a repository whose stored URL ends in .git or /. source.RepoByURL removed a trailing / and .git from the operator's URL but compared it with Aveloxis's repo_git as stored. A repository that Aveloxis stores as https://host/a/b.git could not be found by --url at all, not even by its exact URL. Both sides are now normalised the same way: surrounding whitespace, then one trailing /, then one trailing .git. Found by the fact-check of docs/data.md.

• Test fixtures:

  • No global git config in the tests. Fixtures set GIT_CONFIG_GLOBAL and GIT_CONFIG_NOSYSTEM for the whole test (t.Setenv), not only for their own git calls; this includes NewReftable's support probe. The code under test runs git too, so a developer's global core.quotepath=false or diff.renames=false broke gitx's assertions.

  • Reftable fixtures skip on git older than 2.45. Before, they failed while building the fixture. The skip goes through the new testdb.SkipEnvironment, which applies RequireTool's rule: skip locally, fail under CI.

• README: the DB tier needs a superuser on a disposable cluster, not just CREATEDB, because the roles.sql tests require one. It creates and drops cluster-wide cvet_role_* login roles, and those roles must be able to log in with passwords. The DB tier's skip message now says the same.

• Already handled:

  • Trailing data after the config's first JSON value is already refused, by the whole-file json.Unmarshal that follows the decoder; Copilot's other finding missed it. A test now pins it (TestLoadRejectsTrailingData, which catches dropping that check), with a note at the site.

  • Pre-1970 landings: declined in 0.4.20, with the reason at the site; the thread is outdated.

• Tests:

  • TestRolesScriptRefusesAnEmptyPasswordForANewRole

  • TestErrorRedactsURLCredentials, including passwords that contain @, /, ? and #

  • TestCloneErrorsDoNotShowURLCredentials, including malformed refused URLs

  • TestResolveAuthorsMaterializesNoContributorSet: the plan scans no CTE but the call's own keys.

  • TestResolveAuthorsEmailStepsCanUseLowerEmailIndexes: a pin, which catches a predicate that is no longer lower(column).

  • TestFixturesIsolateTheTestFromGlobalGitConfig: global and system config

  • TestNewReftableSkipsWithoutReftableSupport: uses a fake git that refuses --ref-format.

  • TestLoadRejectsTrailingData

  • TestStreamCommitsListsRootFilesUnderLogShowRootFalse

  • TestRepoByURLIgnoresTrailingSlashAndGitOnBothSides

  • TestDataDocCoversEverySchemaColumn: the document drifting from the schema in either direction fails it.

  • Each failed before its fix, except the two pins.

0.4.22 (unreleased)

Closes the gap a code review of 0.4.21 found (its finding #1): a lock lost between a check that it is held and the write the check guards. No migration.

• Every write made under a lock proves the lock in its own transaction. Before, each check was a ping on the lock's own connection, outside the write. A session that ended just after the ping (an idle timeout, a server restart, a terminated backend) left the write to land anyway, possibly after another process had taken the lock and started on the same repository, or after labor --retry-uncounted had started clearing the shared cache. The write now fails with ErrLockLost and writes nothing. The fence has two halves (internal/store/fence.go):

  • Each write through a fenced store (store.Fenced) runs in a transaction that first takes each lock's fence key shared, for the transaction. It then checks in pg_locks that each lock is still granted to the session that took it. The session is identified by its pid and its backend_start, so a later session that reuses the pid does not pass. Single-statement writes now run in such a transaction when fenced; unfenced writes run as before.

  • Taking a lock (TryLockRepo, TryLaborGate) then waits for fence keys exclusively, so every fenced write still in flight of a previous holder it conflicts with ends first. A conflicting previous holder's write therefore either proved its lock before the lock was taken again, and lands before the new holder starts, or finds the lock granted to another session.

  • A lock has one fence key per holder mode. Writes under the repository lock or the exclusive gate (--retry-uncounted) take the exclusive holders' key; writes under the shared gate (labor runs) take the shared holders' key. An exclusive taker waits for both keys, a shared taker only for the exclusive holders' key, so labor runs never wait for each other's writes, nor, while waiting, hold back every other labor process's (PostgreSQL queues new shared requests behind a waiting exclusive one; found by review round 2 of this change).

  • The wait is bounded by fenceWaitTimeout (5 minutes, far above any one write), set as the wait's lock_timeout. Past it (a previous holder frozen inside a write: a suspended laptop, a paused VM), the taker releases the lock and reports it busy, as when another process holds it, only after that wait. A taker whose wait is cut short by its context, or fails on the server, also releases the lock.

  • Sync, resolve and every labor run use a fenced store from the moment they hold their locks: the repository lock, and for labor also the labor gate. --retry-uncounted's cache clear is fenced by the gate it holds exclusively. Each repository's outcome (last_sync_*, last_labor_*) is fenced too, by the repository's lock only (the gate guards the shared cache, not tracked_repos). When the repository's lock was lost the outcome is not recorded, since the repository may already be another holder's; the error is still reported by the command. A labor run that lost only the gate records it.

  • Two source tripwires guard against SQL that bypasses the fence by mistake (a write sent through the pool or a connection rather than a fenced store). Both fail closed: SQL they cannot read fails them. They do not see a store write method called on an unfenced store while a lock is held; that is left to the pipelines' rebinding to the fenced copy and their behavioural tests (declined at store.Fenced).

    • In internal/store (TestEveryStoreWriteGoesThroughTheFence), outside fence.go and the lock-connection code (tryAdvisory, waitFence, Alive, Unlock):

      • the pool may only be the receiver of a read whose SQL is a constant that does not write; it may not be passed on (pgx.BeginFunc), kept in a local, or used for Config to open a connection;

      • nothing may open or reach a connection (pgx.Connect, a lock's conn, Acquire, Conn, PgConn, Prepare, ExecParams);

      • Exec, Begin, CopyFrom and SendBatch are called only on tx, a fenced transaction, and never taken as method values.

    • In the rest of the module (TestOnlyStoreAndMigrationsWrite, an archtest), outside internal/store and internal/db, since the exported Store.Pool would otherwise let a write through unseen:

      • the SQL of every Exec, Query, QueryRow and Queue must be a constant expression (literals, their concatenations and the package's string constants, unless a local name shadows one) that does not write; its write pattern now also matches an UPDATE with an alias (the repository's own idiom, with or without digits) and MERGE;

      • nothing may copy, batch, reach a connection below the pool, or take those methods as values, and nothing imports another database route (database/sql, pgx's stdlib, lib/pq), whose calls the check does not read (found by review round 9 of this change);

      • nothing may open a transaction except internal/source's read-only one. The one exemption is source.ReposByQuery, which runs the operator's selection query inside a server-enforced read-only transaction.

      • The pool itself may still be passed to internal/source's readers.

    • Each tripwire has a test of the evasions review rounds 3 and 4 of this change found.

• A signal during a lock's ping no longer ends the lock's session. pgx closes a connection whose statement's context ends, so an interrupt landing during an Alive ping released the lock (for the labor gate, every worker's), and with the outcome now fenced the interrupted run could not record it. The ping now runs without the caller's cancel, bounded by its own 10s timeout, and a cancel is reported once the ping answers (found by review round 1 of this change). A caller cancelled while it waits its turn on the lock's connection (labor's workers share the gate's) returns without a ping of its own, so a signalled worker waits at most for the one ping in flight.

• The Alive checks stay. They stop a run before work whose writes would be refused (a fetch, an scc batch), and they keep the lock sessions from idling out.

• Cost: one extra round trip per fenced write, plus a transaction around the single-statement writes, and a short wait transaction each time a lock is taken. A taker only ever waits behind a previous holder that lost its lock in the middle of a write. A live holder is never waited for: while it holds the lock, the taker cannot take it (--retry-uncounted cannot take the gate while any labor run holds it shared, and the reverse). And that write then fails its check.

• Not covered: the clone directory. git work on a clone is still guarded only by the Alive checks and git's own ref locks.

• Tests:

  • store: a fenced write is refused once its lock's session is gone (a single statement, and one returning a row, as UpsertBranch writes; found by review round 12 of this change: the row-returning helper's fence was unpinned), while another session holds the lock, for a session with the same pid but another start (or the same start but another pid), for a session only waiting for the lock, and for the two-int advisory lock with the same halves; every lock of a fence is checked, whichever is lost; holders' writes never wait for each other, another holder's or the same holder's (found by review round 10 of this change: a write taking its key exclusively, which would serialize every labor write, passed every test); a write fenced by two locks (labor's) takes both fence keys, so either lock's next taker waits for it (found by review round 8 of this change: only the check of every lock was pinned, not its key); a write checks its lock only once it holds its fence key, so a lock lost while the write waits for the key (behind a taker's wait) is seen (found by review round 7 of this change: the order was not pinned);

  • store: a taker waits for a conflicting previous holder's open write transaction (repository lock after repository lock, exclusive gate after shared, shared after exclusive), and a shared gate taker does not wait for another shared holder's; a taker behind a write that stays open reports busy after fenceWaitTimeout and releases the lock; a busy attempt keeps no session (found by review round 11 of this change: its close was unpinned); a fenced write that fails gives its pool connection back and lets go of its fence key, so the next taker gets the lock at once (found by review round 13 of this change: the rollback after a failed single statement was unpinned); it does not keep the lock when its wait is cut short by its context (cancelled once it is seen waiting), or fails on the server (a statement_timeout: the connection stays open, so the lock is released explicitly);

  • store: a lock whose ping is interrupted (the connection's reads stalled, then the context cancelled once the ping is seen blocked) is still held, and a write it fences still succeeds; Alive cancelled while queued behind another ping returns at once;

  • cli: labor --retry-uncounted whose gate's session dies once it holds it (a test seam, afterRetryGate) fails and keeps the cache (found by review rounds 5 and 6 of this change: the clear could be unfenced with every test passing);

  • pipeline: with the lock killed right after a passing check (a new alive test-hook stage), sync stores no commit, resolve no resolution and labor marks no window (losing its repository lock or the labor gate). None records its outcome after losing the repository's lock; labor records it after losing only the gate.

  • Each test was checked against a mutant of the code it covers (no pid, start, key, granted or objsubid match; no fence key taken by the write; the write's fence key taken exclusively rather than shared; only the first or last lock's fence key taken; the check before the fence key; no wait on taking; only the first or last lock checked; unfenced single statements, or row-returning ones; each pipeline's fenced store removed, or labor's without the gate or without the lock; the labor outcome unfenced; the lock left open when the taker's wait fails; a busy attempt's connection left open; no rollback after a failed single statement or row write; each holder mode's key swapped, a shared taker waiting for both keys or an exclusive one for one; no lock_timeout on the wait, or its timeout reported as an error; the ping on the caller's context; no check of the context after the queue; a pipeline writing or opening a transaction through the pool; each tripwire check disabled in turn; the --retry-uncounted clear unfenced).

0.4.21

Fixes from review rounds 14 and 15 (round 14 of 0.4.20, committed as fb20691; round 15 of this change) and from a code review of 0.4.21.

• Labor workers no longer collide on the shared gate connection. Since 0.4.20 every worker checks the labor gate before each cache and window write, and all workers share the gate's one connection. A pgx connection serves one caller at a time, so with workers >= 2 (the default is 2) and two or more repositories, overlapping checks failed with conn busy, reported as labor gate: advisory lock lost, failing random repositories. Two pings writing to the wire together could also break the connection and really release the gate. RepoLock is now safe for concurrent use: Alive and Unlock hold a mutex. Alive and Unlock start their timeout after they take the mutex, so time spent behind another worker's ping is not taken from its own (a queued check on a slow server would otherwise read a healthy gate as lost).

• A run checks its locks before it first changes stored windows (InvalidateWindows, DeleteWindowsNotIn, ApplyRecompute), not only before each cache and window write. Those steps run after loading the graph, which can take long on a big repository. A run that has lost its repository lock by then no longer marks or deletes windows, or clears the branch's dirty mark, before it stops. A lock lost between the check and those writes is not detected until the next check, as for every check here: each is a ping on the lock's own connection, not part of the write's transaction.

• Tests:

  • two repositories on two workers with scc batches of one file, five rounds: no gate errors (it failed every time before);

  • a run that lost its locks before changing windows marks nothing and keeps the dirty mark;

  • Alive after waiting longer than the ping timeout for its turn still succeeds; eight goroutines calling Alive on one lock get no error (without the mutex it fails every time: conn busy, or now and then a pgx panic, watch already in progress, so the bug could also crash the process);

  • TestLaborStopsWhenItLosesTheGate now checks the second repository never started (it passed with the per-repository gate check disabled).

• The 0.4.20 entry said the locks are checked "before every cache and window write"; the window-changing steps above were the exception, and now are checked too.

0.4.20

Migration 0018. Fixes from Copilot's third review of PR #1 (review 5814265635, on dfac519).

• labor --recompute keeps the series until each replacement is written. It deleted the selected windows before loading the graph and measuring, so a broken clone, a git error or an scc failure afterwards left the series gone or partly replaced.

  • It now records a request and deletes nothing. store.MarkRecompute adds one to the request's count in labor_recompute_requests (migration 0018), per repository and window configuration, and marks no window. The command requests for every selected repository before any repository runs (Labor.MarkRecompute, which replaces the Labor.Recompute field). --retry-uncounted clears the cache first, then requests.

  • Each labor run applies a pending request under the repository's lock, before it reads which windows are done (store.ApplyRecompute). It marks that configuration's windows on the branch the run reads (needs_recompute, migration 0018) and records the request applied, in one transaction.

  • So the request needs no lock: a repository skipped as busy, or failing before it measures, is recomputed by a later run. A request made while a run is under way stays pending for the next run, since that run may have measured before it. And the marks always land on the branch labor reads: a default branch renamed between the request and the run cannot leave the former default's windows marked forever.

  • DoneWindows leaves marked windows out, so this run, and every later labor or run until each is replaced, computes them again. Each write replaces the stored window of the same key in its own transaction (WindowRow.Replace), so every window is always complete. Replace replaces a marked window only: writing an unmarked (done) window again is ErrWindowExists, as before, never a silent rewrite.

  • A run that fails part-way keeps the windows it did not reach, still marked, and the next plain run finishes them. store.DeleteRepoWindows is removed.

  • The two other places that deleted windows before measuring now mark them too: InvalidateWindows (windows a late push or force-push changed) and DeleteWindowsNotIn's count-scheme arm (windows of an older count scheme, as after an upgrade). Windows off the current plan's grid are still deleted, since no run would replace them.

  • --retry-uncounted now holds a global labor gate (store.TryLaborGate) exclusively for its whole run, and every labor run (Labor.RunIDs, also under run) holds it shared. The scc cache it clears is shared by every repository, so another labor process, perhaps still under the limit being repaired, could otherwise record files as not counted again behind it, through any repository. It refuses to start, clearing nothing, while another labor process is active; labor runs started meanwhile skip every repository as busy. A run checks the gate's session before each repository starts, and with the repository's lock before every cache and window write, and stops if either is gone (ErrLockLost, now "advisory lock lost"), so a run that lost the gate writes nothing more to the shared cache. Using the gate's session that often also keeps it from idling out (idle_session_timeout) over a long repository. A signal before or while the gate is taken reports every repository not started, as the summary line. A repository busy with a sync or resolve is skipped and finished by the next labor run, from the cleared cache.

  • A test recomputes with an scc that reports its version but fails every scan, after clearing the whole blob cache so scc must run. It checks the series is unchanged, then that a plain run with scc working recomputes every window.

• resolve takes the repository's lock, as sync and labor do; it writes commits.cntrb_id and labor dirty marks.

  • While another aveloxis-cve process holds the lock, Syncer.Resolve returns ErrBusy and writes nothing.

  • Like sync and labor, it checks the lock is still held before each write. If the lock's session died, it stops with ErrLockLost.

  • The resolve command reports such a repository as skipped and goes on. run's resolve step reports resolve skipped, without failing.

  • resolve --all then fails, naming the skipped repositories and saying to run it again: nothing redoes --all later (sync and run resolve only unchecked or unresolved authors).

• Epoch and pre-1970 commits (Copilot asked to allow pre-1970 landings):

  • A commit dated exactly at the epoch (git accepts @0 +0000) is a real landing. A history made only of epoch-dated commits used to get no windows, and now does. An epoch landing anchors the windows only when nothing lands after the epoch: a placeholder root dated at the epoch, as some VCS conversions write, followed by real history still anchors on the first real landing, not on decades of windows of the placeholder tree. DefaultBranchState also counts an epoch landing only when the commit that landed it was committed at the epoch, which excludes the synthetic landing a root with an implausible clock gets.

  • A default branch on which nothing with a plausible date has landed (a single root whose clock was years ahead) is skipped with its own reason (store.ErrNoDatedLanding). It used to be reported as not synced, with "run sync first".

  • A hand-written commit object with a negative time no longer stops the repository's sync. git rev-list --timestamp prints its wrapped unsigned value, which LoadGraph rejected as a malformed stream; it now reads as an invalid time, as git log already did.

  • Declined, with the reason at the site: anything before the epoch. git keeps times unsigned, and git 2.55 refuses pre-1970 dates in git commit, so no real commit lands then; dropping the bound would let the synthetic landing anchor the windows.

• Docs: the README's table row for repo_labor_windows names needs_recompute (filter NOT needs_recompute for current windows only).

• Tests:

  • a replace is atomic: a failing write keeps the stored window;

  • a repository busy during --recompute is recomputed by the next plain run;

  • a request marks no window, and a run applies it once to the branch it reads and its configuration only (not a former default branch, not another configuration); a request made during a run is applied by the next run; a Replace over an unmarked window is ErrWindowExists; resolve --all fails when it skips a busy repository;

  • dirty windows and older-scheme windows keep the series through a failed run, and the next run recomputes all of them;

  • a recompute keeps a configuration whose window ends coincide (step_months 1 and 3);

  • an epoch-dated history gets windows; neither the synthetic epoch landing (also alone: no windows) nor an epoch-dated placeholder root before real history anchors them;

  • --retry-uncounted refuses to start beside another labor process and clears nothing; labor skips everything while --retry-uncounted runs; a run stops starting repositories once its gate is lost, and stops writing within a repository; the gate survives an idle-session timeout shorter than a long repository; a signal before the gate starts nothing; a history without a plausible date is skipped with its reason;

  • LoadGraph takes a negative-time commit as an invalid time.

0.4.19 (unreleased)

Fixes from the review of 0.4.18 (committed as 1b98a71), and run now resolves authors.

• run is sync, resolve, then labor. Sync already resolves the authors of the commits it ingests. run now also retries, for each repository whose sync succeeded, the commits whose author Aveloxis could not match before (resolve, unresolved only), as Aveloxis's contributor data improves.

  • Resolve runs before labor. A newly resolved author changes a window's contributor counts (and marks it dirty), and resolving after labor would have left that run's windows stale until the next run.

  • A failed resolve is reported and makes run fail; labor still runs. An interrupt during resolve ends run as an interrupt, with no failure lines.

  • Tests cover all three behaviours.

• Facade file names are scrubbed as Aveloxis scrubs them: one U+FFFD per invalid UTF-8 byte (Aveloxis's db.safeUTF8), not one per run of them. This is only possible with core.quotepath=false. Names that differed only in how many invalid bytes they held collided on the table's key, so a row was silently dropped and another stored under a name Aveloxis does not have.

• Tests:

  • the type change (a symlink replaced by a file: one numstat line, two patch sections);

  • the 64 KiB line cap, at its exact boundary (a difference at content byte 65534 is a change, at 65535 a reformat);

  • a sync that needs several facade batches (the batch size is now a variable a test can lower);

  • a non-UTF-8 name pair.

• Docs: the sync package doc, sync's help text and the README name the facade step.

0.4.18 (unreleased)

Migration 0017. Commits counted exactly as Aveloxis counts them, for every branch (user decision of 2026-11-07: alongside our own counts).

• New table commit_facade_files. It holds one row per (commit, file) with Aveloxis's aveloxis_data.commits semantics:

  • the file name as git's numstat prints it, renames as old => new or dir/{a => b};

  • added and removed lines adjusted by Aveloxis's whitespace walker (Augur's rules): a blank added line is whitespace, and a reformat of a line longer than 8 runes is whitespace and one fewer removed;

  • a whitespace column;

  • binary files as 0/0;

  • one (no files) row for a merge or an empty commit.

• Every commit on every branch gets rows. Joining branch_commits gives Aveloxis-identical numbers per branch. commit_files is unchanged: its per-path, no-rename counts are what the labor windows need to line up with snapshot trees.

• How it's computed. gitx.StreamFacadeFiles runs Aveloxis's walker command (git log --numstat -p with its flags) once over a batch of commits, and ports its parser.

  • git runs with the user's configuration and git's defaults, as Aveloxis's does, not with the runner's core.quotepath=off, so non-ASCII names are C-quoted exactly as Aveloxis stores them.

  • Patch sections pair with numstat names by position; Aveloxis also checks each header against its name.

  • Sync counts every commit not yet counted (commits.facade_checked), in batches, after patch-ids.

  • Migration 0017 adds the table and the flag, so the next sync counts every existing commit.

• Verified against Aveloxis. Run on aveloxis-cve's own clones over the commits Aveloxis has for repos 123 (redis), 132, 281 and 377, it reproduces aveloxis_data.commits exactly: all 42,640 rows on all five columns.

0.4.17 (unreleased)

Fixes from the review of 0.4.16 (committed as caf5325).

• Shared blobs need few open files. 0.4.16 opened a scratch file for every slot of a blob at once, before copying, so a tree with many distinct-named copies of one blob (empty fixtures, say) could exhaust file descriptors and fail labor every time. Each blob now streams to its first slot, and that file is copied to the other slots one at a time, so at most two files are open. A test runs labor with the descriptor limit just above what the process holds.

• The shared-blob test also checks that every copy of a blob measures like the first, so a copy written wrong (empty, say) fails it.

• Comments that still described the removed patch-copy machinery are corrected.

• The copies to a blob's further slots (copySlots) check the context before each copy. They read a local file, which an interrupt's cut of git's pipe does not reach, so otherwise an interrupt waited for every copy of a large blob (a reviewer measured 637 ms for a 256 MiB blob under 8 names).

0.4.16 (unreleased)

Migration 0016. Fixes from the sixth review round of 0.4.14-0.4.15 and from Copilot's second review of PR #1 (review 5813423952, on a57ae17).

• Revert chains follow named commits and cherry-pick trailers only (decision of 2026-11-07). 0.4.14-0.4.15 also followed patch-id matches, to catch reverts of unnamed manual backports (Copilot's first finding).

  • Four review rounds each found a realistic history, made with ordinary git commands, where that looped and reported undoes, and the vulnerable commits themselves, as fixes. The last was a squashed backport, then cherry-pick -x of a fix-by-revert, then git revert. 0.4.13 had handled it.

  • Patch-id matches have no direction, so a fix made by reverting the vulnerable commit and its undo admit two consistent readings.

  • Revert and cherry-pick links always name a commit that existed before the commit naming it (its hash is in the message), so the chains kept are acyclic by construction.

  • Removed: the commit_patch_copies table, commits.patch_copies_checked, sync's patch-copy step, store.PatchCopyCandidates/RecordPatchCopies and branches.Ancestry.

  • Migration 0016 drops the table and column and rebuilds commit_cve_undoes for every repository. That also clears any copies and undoes stored under the 0.4.14-0.4.15 rules.

  • Kept from 0.4.14: a revert naming a CVE of an unnamed cherry-pick of its fix is an undo, and a revert of an unnamed cherry-pick of an undo is a reapply.

  • Known gap, as before 0.4.14: a revert naming a CVE of an unnamed backport matched only by patch-id (no trailer) counts as a fix. So does the commit that undoes it in turn, counted as an undo. A test pins this so that changing it is deliberate.

  • Migration 0015 is unchanged from 02b7c64.

  • The store/0016 agreement fixture includes a cherry-pick that itself names the CVE, picked from an undo, so drift in that filter between RefreshUndoes and 0016 fails it.

• sql/roles.sql refuses a role with SUPERUSER, CREATEROLE, REPLICATION or BYPASSRLS. None is a privilege a revoke can remove. On PostgreSQL 15 and earlier CREATEROLE can grant itself other roles, and a replication connection can copy the whole cluster, tokens and password hashes included.

• Blobs stream to disk (Copilot). gitx.CatBlobs handed each blob to its callback whole, so one multi-gigabyte tracked binary could exhaust the sidecar's memory before scc saw it.

  • The callback now gets the size and a reader limited to the blob; anything it leaves unread is drained.

  • Labor writes each blob straight to its scratch file(s). A test holds streaming a 64 MiB blob to under 16 MiB allocated.

  • Control files are read up to controlFileCap (16 MiB), and only whole lines: a rule the cap splits is dropped, never applied in part, so *.log can't become *.

• A timeout during Stream's final drain is reported (Copilot). When the consumer returned early, the process exited 0 and a detached descendant held stdout until the deadline, the cut-off drain read as success. It now returns the context's error.

• Test robustness (Copilot): the grace-sweep test checks for a running process with ps (present and not a zombie), not kill(pid, 0), which succeeds on a zombie a container's init has not yet reaped.

0.4.15 (unreleased)

Completes a 0.4.14 fix (committed as 02b7c64, whose TestRolesScriptRevokesGrantsOtherRolesMade fails on the column case below).

• sql/roles.sql: column grants made by a column-only grantor. A grantor holding only column privileges with grant option is refused a table-level REVOKE ("permission denied for table"). The whole script then aborted and the grant survived. Such grants are now revoked column by column, as their grantor: REVOKE ALL (cols) ON TABLE ... CASCADE, lending USAGE on the schema when the grantor lacks it.

• The grantor test's cleanup no longer leaks scratch roles when the script fails. A column privilege a grantor granted survives DROP OWNED, so the cleanup drops the scratch schemas' objects first, then the three roles' owned objects together.

0.4.14 (unreleased)

Fixes from the Copilot review of PR #1 (review 5809845545, on d235b90), verified against 5f7d5d3. Adds migration 0015 (table commit_patch_copies, column commits.patch_copies_checked).

• A revert of an inherited fix is not a fix. Store.RefreshUndoes treated a revert as an undo only when the reverted commit's own cve_ids named the CVE. cve_fix_commits also credits a fix to an unnamed backport: one that shares a patch-id with a commit naming the CVE, or one cherry-picked from it. So a revert of such a backport whose message named the CVE was reported as a fix.

  • A hop of a revert chain may now revert either a commit naming the CVE, or an unnamed copy of one in the same repository. In the second case the chain continues from the named commit, so the copy carries its parity. A revert of an unnamed backport of a fix is an undo; a revert of an unnamed copy of an undo is a reapply. A named commit never takes a copy's path.

  • A copy is either cherry-picked from the named commit, or a patch-copy: an unnamed commit with the same patch-id that is not an ancestor of the named one.

    • That makes a backport, an embargoed fix committed unnamed on a maintenance branch before the named one on main, and an unnamed reland on the same line all copies.

    • An ancestor is not a copy. A fix made by reverting the vulnerable commit, then undone by reapplying it, would otherwise make the vulnerable commit a "copy" of the undo and loop the chain.

    • The test is ancestry, never commit time or ingest order, so commits made in the same second are judged the same way.

    • Sync decides patch-copies from the commit graph after it computes patch-ids, checking each commit once (patch_copies_checked), and records them in commit_patch_copies.

    • It marks exactly the commits it evaluated, so one patch-id'd in between waits for the next sync.

    • It pairs only commits that could earn credit; a copy already naming all its source's CVEs is skipped.

    • branches.Ancestry computes generation numbers once. It then walks once per named source, never below the oldest generation asked about, reusing one visited set, so memory is O(commits) bits whatever the number of pairs. An earlier draft cached a full-history bitset per commit: about 800 MB for 4,000 commits on a 1.5M-commit graph.

    • Migration 0015 adds the table and the flag. Each repository's next sync fills them and refreshes its undoes. Until then, and for a disabled repository, undo chains count only named and cherry-picked credit. (The view still credits patch-id matches, as it did before.)

  • The recursion keeps each state once (UNION), so the work stays bounded on any history, and a chain stops at depth 64.

  • Sync refreshes the undoes again whenever it checks commits for patch-copies.

  • Migration 0015 rebuilds commit_cve_undoes under the new rule, including for disabled repositories. A test holds it equal to RefreshUndoes run per repository.

• SCP-style URLs must be remote. gitx.ValidateURL accepted anything containing @ and : without ://, so /srv/private@host:repo.git passed with local sources disabled, though git reads it as a local path. The part before the first colon must now be exactly user@host, with a non-empty user and host and no /, which is git's own rule.

• sql/roles.sql narrows on a re-run. It granted SELECT on the analytics schemas but never revoked earlier, broader grants. It now first revokes everything the role holds in the four Aveloxis schemas, then grants SELECT:

  • table, column and sequence privileges;

  • CREATE and other schema privileges;

  • default privileges any role set for future tables, sequences or schemas, per schema or global.

  • A superuser's REVOKE acts as the owner and removes only the owner's grants, so the script also revokes, with CASCADE, as every other grantor found in the table, column, sequence and schema ACLs. A grantor that has lost USAGE on the schema is lent it for its revoke and loses it again afterwards.

  • It refuses to run, naming them, while the role owns anything in those schemas or is a member of another role. Neither is a privilege a revoke on the role can remove (a membership carries the other role's privileges). Reassigning objects and ending memberships are left to the DBA.

• procx.Stream cannot hang on a held pipe. It read stdout to EOF before waiting on the process. A descendant holding stdout after the process exited, including a detached one the group kill misses, could block it past the timeout, and with no timeout indefinitely.

  • It now waits while it reads.

  • When the context ends, it stops reading at once.

  • Once the process has exited, a Read blocked on the empty pipe for streamGrace (5s) means a descendant holds it. Stream then stops reading, sweeps the group, and reports exec.ErrWaitDelay.

    • The grace counts only while a Read is blocked. It starts from the later of the Read's start and the process's exit, and afresh for each Read.

    • So a slow consumer keeps every byte still buffered in the pipe, a descendant may keep writing after the exit as long as each gap is shorter than the grace, and a consumer busy after EOF is never told the pipe was held.

  • If Wait already gave up on stderr, Stream sweeps the group at once.

  • A failure the cut-off caused is reported as the held pipe, whether or not the consumer's error wraps the read error (the git parsers' do not). A consumer's own earlier failure still wins.

  • Tests use the test binary as a detached helper.

• gopls hints: five WriteString(a + b) calls in internal/gitx (StreamCommits twice, CatBlobs, PatchIDs, Existing) now write the parts separately.

• Tooling note: the local Go is now 1.27.3. The gopls MCP server's run_govulncheck, which forces GOTOOLCHAIN=local, failed (exit 3) under the old local Go 1.26.5 on five standard-library vulnerabilities. It now reports none.

0.4.13 (unreleased)

Fix from the review of 0.4.12 (committed as efb8ddb). Test only; no behaviour change.

• TestLinksHopLimitMatchesRealSCC failed on a correct osLinkLimit in two cases:

  • the test host's TMPDIR was reached through a link to a link, or through a link whose target holds one (0.4.12's count of the links in its path took one per linked component);

  • TMPDIR was relative (the count started at /).

  It now runs scc in its temp directory's resolved real path, where no link uses up a hop, and checks osLinkLimit exactly. EvalSymlinks runs first, so a .. after a link leaves from the link's target, as the kernel does. A relative result is joined to the resolved working directory. It passes with:

  • the default TMPDIR, and a link-free one;

  • TMPDIRs reached through one or two links;

  • a .. after a linked component;

  • relative TMPDIRs.

  checkoutPathLinks stays pinned by TestLinksHopLimit's table.

0.4.12 (unreleased)

Fixes from the review of 0.4.11 (committed as bf9857c):

• macOS hop limit is 31, not 32. labor.MaxLinkHops is the OS limit less the links in the checkout's own path. Aveloxis always clones under os.TempDir()/aveloxis-analysis, and on macOS that is /var/folders/..., where /var -> /private/var uses up one hop. Linux stays at 40.

  • The two parts have their own functions: osLinkLimit (the kernel's MAXSYMLINKS) and checkoutPathLinks (1 on macOS, 0 on Linux). A table test pins both hosts' values.

  • TestLinksHopLimitMatchesRealSCC counts the links in its own temp directory's path and checks osLinkLimit against real scc there, so it holds whatever the test host's TMPDIR.

  • An Aveloxis whose TMPDIR path holds a different number of links than the default follows a different number of hops. This is documented, not emulated.

  • labor.CountScheme is now scc-walk3, so windows of 0.4.11 are recomputed from the cache.

• New TestLinksResolve cases:

  • a .. after a linked directory that is not at the root (tdeep -> a/b, then ../tdeep/../rules is a/rules);

  • a .. leaving the root, whose target would exist if it were clamped;

  • a cycle of linked directories. Each case fails its mutant, which all passed before. The cycle mutant hangs.

• Declined at the site (labor.Links.Resolve), with reasons:

  • case-insensitive matching of link targets on a case-insensitive host filesystem;

  • links into the checkout's own .git/.

0.4.11 (unreleased)

Fixes from the review of 0.4.10 (committed as 473be6b):

• A symlinked control file is followed. In a checkout, scc reads a .gitignore, .ignore or .sccignore that is a symlink through the link. 0.4.10 never saw it, because LsTree drops symlinks, so it counted files Aveloxis skips.

  • walked now reads such files through labor.Links, which resolves one path component at a time, as the OS does:

    • it follows chains and linked directories, and applies .. to the directory reached;

    • a component before a .., a trailing / or a /. must be a directory;

    • it follows at most labor.MaxLinkHops links, the host's limit (32 on macOS, 40 on Linux).

  • A missing, directory or non-file target, or a cycle, reads as no rules, as gocodewalker's failed read does.

  • An absolute target, or one outside the snapshot, also reads as no rules. In a checkout it would read a file on Aveloxis's host, which aveloxis-cve cannot see.

  • Tests hold this, and the hop limit, to real scc on a git clone on the test host. A symlink in Aveloxis's own checkout path (macOS's /var) uses up one hop and is not emulated.

• A symlinked .gitmodules is not followed (labor.FollowedLink). git refuses to check out such a tree, so Aveloxis has no checkout and no repo_labor for it, and there is nothing to match.

  • labor.CountScheme is now scc-walk2, so windows computed by 0.4.10 are recomputed from the cache.

• WriteWindow refuses a window with no count scheme. DeleteWindowsNotIn would otherwise recompute it on every run.

• New tests:

  • A walk-skipped file has no start or end metrics in a window where it is changed and ignored at both ends.

  • An edited .gitignore applies its new rules from the next snapshot on, because control files are cached by blob, not path.

  • Migration 0014 marks pre-0014 windows tracked and leaves the column without a default.

• README: the sentence about changed files without metrics now names walk-skipped files. It read as if walked files had none.

0.4.10 (unreleased)

Migration 0014. Rolling labor now counts the same files as Aveloxis's repo_labor.

• Ignore files are honoured, as in a checkout. Aveloxis runs scc -f json --by-file <checkout>, and scc's own walk then skips:

  • tracked files that a .gitignore, .ignore or .sccignore names;

  • .gitmodules paths;

  • directories named .git, .hg or .svn.

• What this fixes. aveloxis-cve counted every tracked file, so on redis it counted the tracked but ignored deps/jemalloc/configure (14,378 code lines, about 4.6%), which Aveloxis skips.

• How. labor.Skipped applies scc 3.7.0's walk to each snapshot's tree without a checkout, using gocodewalker's matcher at the version scc 3.7.0 is built with. A test checks it against the real scc binary on a fixture with one case per rule. A walk-skipped file is not counted in the snapshot. It still appears among the window's changed files, without metrics, like a file scc does not classify.

• Recompute. Windows record a count_scheme. The next labor run recomputes windows stored under 0.4.9's scheme (tracked). Each file is still measured alone, so the scc blob cache stays valid and the recompute rescans nothing.

• New dependencies: github.com/boyter/gocodewalker (its go-gitignore package) and github.com/danwakefield/fnmatch.

0.4.9 (unreleased)

• New run command: sync, then labor for each repository whose sync succeeded. A repository whose sync failed gets no labor run (its clone may be missing or behind the database); a failed sync or a failed labor run makes the command fail. A not-ready or busy repository is skipped.

• govulncheck: golang.org/x/text v0.43.0 (fixes GO-2026-5970, GO-2026-6629) and toolchain go1.26.10 (crypto/tls, crypto/x509, encoding/asn1, encoding/xml fixes), both as Aveloxis pins them. govulncheck ./... reports none.

• An interrupt (SIGINT/SIGTERM) ends sync, labor, run and resolve with one cancellation error (non-zero exit), and run does not start labor after an interrupted sync. A repository whose error the interrupt caused is reported as interrupted, not FAILED; a failure that happened before the interrupt is still reported as FAILED.

• A sync, labor or resolve run of one repository that ends with an error while the context is done returns an error carrying the cancellation. The two exceptions are deliberate: a busy repository (another process holds its lock) is a definite answer, and a failure to record the outcome happens on a context the interrupt cannot cancel. Some pgx write paths return a bare net error (write failed: i/o timeout) when cancelled, which was reported as FAILED.

• After an interrupt, sync, labor and run start no further repository. The ones never started are reported in one line (N repositories not started: interrupted, or 1 repository ...), not one line each, and are not failures. A repository skipped as not ready, not synced or busy is still reported as skipped when a later interrupt also reached its error.

• One interrupt rule, procx.CarryInterrupt (with its predicate procx.Interrupted, which the CLI uses to report interrupted), used by procx.Stream and at every repository's exit: an error returned while the context is done (cancelled or timed out) is prefixed with the context's error unless it already carries it. procx.Stream applies it to a consumer error, whose input may have been cut short; a real parse failure without a cancel is returned as itself, as before.

• Both combined errors read on one line (context canceled: <cause>), and sync, labor and run print every repository's error on one line, so each per-repository line starts with repo N: .

• golang.org/x/sync v0.24.0 (required by go mod tidy with the bump above; Aveloxis pins the same).

0.4.8 (unreleased)

• labor --recompute deletes only the current branch and window settings' windows, and only after the clone and scc checks pass. Other settings' series are kept, and a repository that cannot run keeps its windows.

• labor --retry-uncounted recomputes every enabled repository and refuses --repo-id. The cache it clears is shared, so recomputing only some repositories left the others measured both ways.

• The migration-0013 test now pins the rebuild as well as the delete.

0.4.7 (unreleased)

Fixes from the review of 0.4.6 (migration 0013):

• labor --recompute drops the selected repositories' windows and computes them all again. labor --retry-uncounted also clears the files the current scc version recorded as not counted, so they are measured again. This is the recovery after a poisoned measurement; deleting cache rows alone left the already-computed windows as they were.

• Migration 0013 rebuilds commit_cve_undoes from scratch. 0012 only inserted, so a stale row from an interrupted 0.4.5 sync could survive.

0.4.6 (unreleased)

Fixes from the review of 0.4.5 (migration 0012):

• Migration 0012 backfills commit_cve_undoes. 0011 created it empty, so an upgraded database reported reverts of fixes as fixes until each repository's next sync, and permanently for disabled repositories.

• A test pins that RefreshUndoes touches only its own repository.

• Docs: a Go crash in scc (panic or runtime out-of-memory) exits with status 2, which counts as a deterministic failure. docs/configuration.md says so. (The recovery it gave did not recompute existing windows; 0.4.7 replaces it with labor --retry-uncounted.)

0.4.5 (unreleased)

Fixes from the review of 0.4.4 (migration 0011):

• Landing times depend only on ancestors. An implausible committer time (invalid, or later than the commit's own ingest + 1 day) now lands with the commit beneath it (0 for a root). A commit's landing therefore never depends on commits added above it later. Incremental sync and full recompute always agree, and landings never run backwards along a branch. The price: a commit whose clock runs more than a day ahead lands at the previous commit's time. store.MembershipScheme is 3, so 0.4.3/0.4.4 memberships are recomputed by the next sync.

• Broken scc is detected with a control file. Before a file is recorded as not counted, scc must measure a known-good control file; if it cannot, scc is broken and the run fails without recording anything, whatever the batch size. A wrapper's exit code of 128 or more (137 = SIGKILL/OOM) counts as a signal, never as a deterministic failure.

• Revert parity is computed per repository at sync time (Store.RefreshUndoes into commit_cve_undoes). cve_fix_commits no longer recurses over every repository's reverts on each query.

• Doc fixes: resolveLinks returns the links it newly inserted.

0.4.4 (unreleased)

Fixes from the review of 0.4.3 (migrations 0009, 0010):

• Stable plausibility. A committer time counts when it is valid and not later than the commit's OWN ingest time plus one day. (The replacement rule for implausible times was replaced in 0.4.5; this release's rule did not keep incremental and full in agreement.)

• Revert chains of any depth. A commit naming a CVE is a fix when its revert-chain depth back to the original fix is even: the fix 0, a reapply 2, ... Odd depth (a revert, or a revert of a reapply) undoes it.

• scc poison is deterministic only. A file is recorded as not counted only when scc exits with an error on it twice. A crash, a signal or a timeout fails the run, and nothing transient is cached for good. (0.4.5 replaced this release's majority guard with a control-file check.)

• Precise labor invalidation. labor_dirty_since is the earliest landing a membership write actually changed: deleted rows, inserted rows, and both the old and the new landing of updated rows. An identical recompute no longer marks every window stale.

• Scheme check. labor refuses a membership of an older scheme until sync has recomputed it.

• Cheaper link resolution. Commits that name no SHA are checked at insert, with partial indexes on the pending ones. Abbreviated links are rewritten only when they change.

• CVE view pushdown. The view's CTEs are NOT MATERIALIZED, so a repo or CVE filter reaches the commits scan instead of scanning every repository.

• is_revert is recomputed with the same rule as cve.IsRevert.

• RepoLock.Alive reports the caller's deadline as itself.

0.4.3 (unreleased)

Fixes from the L10 review of 0.4.2 (migrations 0007, 0008):

• Implausible clocks no longer poison landing times.

  • A committer time is used only when it is valid (years 1..9999) and plausible (0.4.4 refines both the bound and the replacement). One future-dated commit (a wrong clock) used to drag every later landing to its date.

  • The commit graph uses the same range check (gitx.ValidUnix) as commit headers.

  • An invalid tip time is stored as a NULL head_committed_at.

  • labor reads the landing times sync stored, so the two always agree.

• Revert-aware CVE matching.

  • A commit naming a CVE counts as a fix unless it reverts a commit that names the same CVE and is itself a fix (0.4.4 generalises this to revert chains of any depth).

  • A revert that is the fix (a revert of the vulnerable commit, as the kernel CNA assigns) counts as a fix.

  • A reapply (Reapply "...", a revert of the revert) counts as a fix.

  • The reverted commits are resolved like cherry-picks, into commit_reverts.

• Cherry-pick and revert links. Abbreviated SHAs are re-resolved on every sync, so a link whose abbreviation later becomes ambiguous is dropped. A source ingested later is picked up.

• labor changes:

  • windows that are no longer in the plan are deleted (a re-rooted history, a moved anchor);

  • resolve marks windows stale when contributor counts change;

  • over-long scratch file names are shortened, keeping their extension;

  • a batch that makes scc fail is bisected down to the failing file, which is recorded as not counted (reported as poison files; 0.4.4 restricts this to deterministic failures).

• New knob: scc.batch_files (default 2000).

• Membership scheme. Migration 0007 adds membership_scheme. Memberships of an older scheme are recomputed in full by the next sync, and their tips are kept, so in-place archive renames still work.

• RepoLock.Alive reports a cancelled context as such, not as a lost lock.

0.4.2 (unreleased)

Fixes from the Phase 3-4 review (migration 0006):

• Windows recompute when history changes under them. A sync records the earliest landing it added or changed (branch_repos.labor_dirty_since). labor then drops and recomputes every window ending at or after it. This covers a late push of an old-dated commit and a force-pushed default branch.

• Landing times never run backwards along a branch: they are the running maximum of committer time along the first-parent chain. Window snapshots use the same times. A malformed root date no longer anchors windows in 1970.

• scc sees each file alone (scratch/<n>/<name>, .sccignore disabled), so its result depends only on name and content, which is exactly the cache key. Case-variant names, a file beside a same-named directory and .sccignore files no longer mix up or block results. The cache scheme is versioned (+isolated1), so stale rows are never reused.

• Paths are matched by their stored name. A non-UTF-8 path counts as added, not X.

• scc runs in batches (default 2000 files), cached after each batch, so a timed-out first snapshot keeps its progress. Version 0.4.3 made the batch size configurable.

• Cherry-pick trailers are resolved at sync time to the one commit they name (commit_cherry_picks). This replaces a whole-repo LIKE scan in the view, which took 74 s on a 200k-commit repo. An ambiguous abbreviation is never guessed.

• Reverts (Revert "..." / "This reverts commit") are flagged is_revert. Version 0.4.3 refines how they count toward fixes.

• New index: branch_commits (branch_id, landed_at).

• Robustness:

  • the per-repository lock is checked before every write phase (ErrLockLost);

  • a half-finished ref update cannot archive the same tip twice;

  • commit dates outside years 1..9999 are flagged dates_invalid (0.4.3 applies the same check to graph times);

  • under CI a missing tool fails a test instead of skipping it.

0.4.1 (unreleased)

Fixes from the Phase 2 review (migration 0005):

• Syncs that could wedge permanently or hang:

  • Moved tags. An upstream re-tag failed every sync; tags are no longer fetched. Failures now keep git's stderr (no --quiet).

  • Branch renames. Renaming release to release/1.0, or a/b to a, failed every sync. Ref reconciliation now runs as two ordered transactions.

  • Malformed commits. One malformed commit failed ingest. Dates are read as unix seconds, so an out-of-range timezone is harmless. Name fields are separated by newlines, so a \x1f in a name is harmless. A missing date is stored as dates_invalid.

  • Lock deadlock. The per-repository lock held a pooled connection and deadlocked small pools. It now uses its own connection, outcome stamps are time-bounded, and the pool is sized from workers. A pool_max_conns below workers is refused.

  • Awkward branch names. Branch names that are not valid UTF-8, or that differ only in case, failed git fetch on macOS and other case-insensitive filesystems. New clones use --ref-format=reftable when git supports it (2.45+).

• Membership after a "foxtrot" merge (the old tip merged in rather than on the first-parent chain) depended on sync history. The incremental path is now taken only when the old tip is on the new tip's first-parent chain.

• Archived branches are never deleted from the database. A re-created clone gets its archive refs back where the objects still exist, and a missed archive rename is reconciled from the refs.

• Efficiency and correctness:

  • branch stats are recomputed only when a tip moves;

  • patch-ids stream from diff-tree into patch-id;

  • archive names are unique within a second;

  • refs are matched to rows by their stored (sanitised) name;

  • sync resolves identities only for commits it has not checked, while resolve re-checks the unresolved ones.

• gitx now runs git through internal/procx.

• Go sources must be ASCII (tripwire).

0.4.0 (unreleased)

Phase 4, the CVE layer (migration 0004, views only, inside aveloxis_cve):

• cve_fix_commits: commits that fix a CVE, matched by their message, by a cherry-pick trailer naming a tagged commit, or by patch-id. The patch-id match catches manual backports whose message does not name the CVE.

• cve_fix_propagation: per CVE and branch (live or archived), the first landed_at of a fix commit and the lag behind the first branch.

• commit_equivalence: commits sharing a patch-id.

• New command: cves --repo-id N [--cve ID].

0.3.0 (unreleased)

Phase 3, rolling complexity (migration 0003):

• labor computes sliding windows on the forge's default branch: labor.window_days (60) long, advanced every labor.step_months (1) months from an anchor of the first commit plus window_days. Month ends clamp (Jan 31 + 1 month = Feb 29), and only complete windows are computed.

• Each window stores:

  • a full scc snapshot at the window's end (repo_labor_rolling, with Aveloxis's repo_labor columns);

  • totals (repo_labor_windows);

  • the files changed by commits that landed in it, with complexity at start and end (repo_labor_rolling_changed).

• Boundaries: windows are (start, end]; the first window is [first, end].

• scc results are cached by (blob, file name, scc version), so each file content is measured once. Unchanged windows reuse the previous snapshot. Runs are resumable.

• scc runs with --no-gitignore --no-ignore. On a real repository the latest window matched Aveloxis's repo_labor exactly (163 files, 16,941 code lines, complexity 277).

• New package internal/procx: the one way aveloxis-cve runs subprocesses (process group, timeout, bounded stderr, typed errors).

0.2.0 (unreleased)

Phase 2, branches and commits:

• sync runs on every tracked repo that is ready in Aveloxis.

  • It uses aveloxis-cve's own bare clone under clone_dir, cloned from the forge. Aveloxis's clones are never used.

  • Every network git call runs with http.followRedirects=false, as in Aveloxis.

  • Branches deleted upstream, and tips replaced by a force-push, are kept under refs/cve-archive/<deleted|forced>/<stamp>/<branch>, never pruned.

  • A remote that suddenly advertises no branches archives nothing.

  • The default branch is the forge's, read on every sync with git ls-remote --symref origin HEAD. If that fails, the previous default is kept with a warning, as in Aveloxis.

• New tables:

  • branch_repos;

  • commits (with CVE/GHSA ids, cherry-pick trailers and git patch-id --stable);

  • commit_files;

  • branch_commits: full membership, with on_first_parent, landed_via/landed_at (when the change reached that branch) and first_observed_at.

• New view: branch_commit_files.

• Membership is computed incrementally on a fast-forward and in full otherwise. An archived branch keeps its row, so first_observed_at survives.

• Identity resolution reads Aveloxis's commit links, contributor emails, active aliases and identities. An ambiguous match stays NULL, and merged contributors are never chosen.

• New commands: sync, resolve [--all], status.

• Second-round review fixes:

  • roles.sql sets ON_ERROR_STOP itself, runs as one transaction and reassigns types and domains;

  • doctor fails when the read-only role can read a credential table;

  • the overlap check compares real directories;

  • the tripwires use one shared SQL lexer and read whole concatenations.

0.1.1 (unreleased)

Fixes from the Phase 1 adversarial review:

• sql/roles.sql no longer exposes Aveloxis secrets.

  • aveloxis_ops is granted per table (schema_meta, collection_queue), with no schema-wide grant and no default privileges. That schema holds forge OAuth tokens, password hashes, session tokens and API keys.

  • aveloxis_data.chaoss_user (password hashes) is revoked.

  • The script refuses to run as anything but a superuser, and does so before changing anything.

  • It hands any aveloxis_cve object another role created over to the sidecar role.

  • Re-running it narrows grants from older versions.

• init-db polls pg_try_advisory_lock instead of waiting in pg_advisory_lock; a waiter's snapshot stalls CREATE INDEX CONCURRENTLY anywhere in the database. The lock key is hashtextextended('aveloxis_cve.migrate', 0), and the wait honours cancellation.

• Config:

  • refuses a clone_dir that equals or nests with Aveloxis's repo_clone_dir, with symlinks resolved;

  • an imported Aveloxis database block must set user and dbname.

• repos add --query refuses ids Aveloxis does not have, and an empty selection.

• repos enable and disable are all-or-nothing.

• doctor fails when the schema is newer than the binary.

• Tripwires:

  • INSERT detection is whitespace-tolerant and checked per statement;

  • ON CONFLICT ON CONSTRAINT is accepted;

  • quoted literals and comments can neither satisfy nor hide a check;

  • constant string concatenations are folded before scanning.

0.1.0 (unreleased)

Phase 1, the skeleton:

• Own JSON config. The Aveloxis config import is optional and reads only its database block and repo_clone_dir. Defaults live in one layer, unknown keys are refused, and docs/configuration.md and the example JSON are tripwired against the code.

• Schema aveloxis_cve with embedded migrations, a ledger (schema_migrations) and a session advisory lock. Migration 0001 creates tracked_repos.

• internal/source, the single read gateway into Aveloxis. It has a column probe (typed SchemaDriftError), readiness from collection_queue.last_collected, and repository selection by id, URL, group or read-only query.

• sql/roles.sql: an idempotent read-only role that owns aveloxis_cve, with default privileges for tables Aveloxis creates later.

• CLI: version, init-db, doctor, repos add|list|enable|disable.

• Architecture tripwires:

  • no Aveloxis Go imports;

  • only internal/source names Aveloxis schemas, and it only reads;

  • every INSERT names a conflict arbiter;

  • no structural test skips.