diff --git a/README.md b/README.md index 881160e..b3d4095 100644 --- a/README.md +++ b/README.md @@ -1,17 +1,17 @@ # git-check-assertions -I recently wrote two blog posts arguing that there might be some value in writing verifiable claims, i.e. assertions, inside of our commit messages: +I recently wrote two blogs posts arguing that there might be some value in writing verifiable claims, i.e. assertions, inside of our commit messages: -* [Should we start writing verifiable claims in commit messages?](https://sven.memcmp.org/2026-02-19-should-we-start-writing-verifiable-claims-in-commit-messages/) -* [Writing the steps to validate a test that already passes in the commit message](https://sven.memcmp.org/2026-02-20-writing-the-steps-to-validate-a-test-that-already-passes-in-the-commit-message/) +* [Should we start writing verifiable claims in commit message?](https://sven.memcmp.org/2026-02-19-should-we-start-writing-verifiable-claims-in-commit-messages/) +* [Writing the steps to validate a test in the commit message](https://sven.memcmp.org/2026-02-20-writing-the-steps-to-validate-a-test-in-the-commit-message/) This is a simple verifier for such assertions. -You write assertions in your commit messages, and `git-check-assertions` will then check out every commit (from the point that your branch diverged from `main`), and verify that the assertions in the commit message hold for the version of the code that is in the commit. +You include a small bash script inside your commit messages, and `git-check-assertions` will then check out every commit (from the point that your branch diverged from `main`), and verify that the script in the commit message runs successfully. For a real-world example, check out the commits in [this pull request](https://codeberg.org/svenvanheugten/git-check-assertions/pulls/8), where `git-check-assertions` is used on itself. -⚠️ Only run this on repositories and branches that you trust, since the commands in the commit messages can do whatever they want. +⚠️ Only run this on repositories and branches that you trust, since the `bash` scripts in the commit messages can do whatever they want. ## Installation @@ -56,22 +56,26 @@ If you use Nix with flakes, you can simply add it to your program's devshell ins ## What do you put in your commit messages? -Simply add a `git-check-assertions` block to a commit message, e.g.: +Simply add a bash script enclosed in a `git-check-assertions` block to a commit message, e.g.: ~~~ ```git-check-assertions -✓ dotnet build -✓ dotnet test --no-build +dotnet build +dotnet test --no-build ``` ~~~ -Each block is parsed line-by-line: +This script will be run with `set -euo pipefail` on the version of the code that is in the commit, and the commit will be considered correct if the script exits successfully. -* A line starting with `[success] ` or `✓ ` runs the rest of the line as a shell command and asserts that it exits with status `0`. -* A line starting with `[failure] ` or `✗ ` runs the rest of the line as a shell command and asserts that it exits with a non-zero status. -* Any following non-empty line belongs to the most recent command and asserts that the combined stdout/stderr from that command contains that string. +You can technically assert that a command fails by writing `! ... || exit 1`, or write assertions about a command's output by piping it to `grep`, but doing so won't lead to very useful error messages when things go wrong. To make those things easier, there are some helper functions included, which are inspired by [bats](https://github.com/bats-core/bats-core) and [bats-assert](https://github.com/bats-core/bats-assert): -Blank lines are ignored. A new `[success]`, `[failure]`, `✓`, or `✗` line starts a new command block. +* `run `: run a command, capturing its exit status in `status` and combined stdout/stderr in `output`. +* `assert_success`: succeed if `run` produced a zero exit status. +* `assert_failure`: succeed if `run` produced a non-zero exit status. +* `assert_output `: succeed if `output` exactly matches the string. +* `assert_output --partial `: succeed if `output` contains the string. + +I'm considering taking `bats-assert` as a dependency, but for now, this very minimal set of functions with a similar interface should get you on your way. ## Multiple blocks in one commit message @@ -89,7 +93,7 @@ Assert that a commit builds: ~~~ ```git-check-assertions -✓ dotnet build +dotnet build ``` ~~~ @@ -97,8 +101,9 @@ Assert that a commit builds and that the tests succeed: ~~~ ```git-check-assertions -✓ dotnet build -✓ dotnet test --no-build +dotnet build +run dotnet test --no-build +assert_success ``` ~~~ @@ -106,28 +111,33 @@ Assert that a commit builds, but that the tests do not succeed: ~~~ ```git-check-assertions -✓ dotnet build -✗ dotnet test --no-build +dotnet build +run dotnet test --no-build +assert_failure ``` ~~~ -Assert that a commit builds, and that the tests fail with the error that you expect: +Assert that a commit builds, and that the tests fail with exactly the error that you expect: ~~~ ```git-check-assertions -✓ dotnet build -✗ dotnet test --no-build -Invalid URL +dotnet build +run dotnet test --no-build +assert_failure +assert_output --partial "Invalid URL" ``` ~~~ -Assert that a specific change breaks the tests (as discussed [here](https://sven.memcmp.org/2026-02-20-writing-the-steps-to-validate-a-test-that-already-passes-in-the-commit-message/)): +Assert that a specific change breaks the tests (as discussed [here](https://sven.memcmp.org/2026-02-20-writing-the-steps-to-validate-a-test-in-the-commit-message/)): ~~~ ```git-check-assertions -✓ dotnet test -✓ sed -i '/crucial code/d' Main.fs -✗ dotnet test +run dotnet test +assert_success + +sed -i '/crucial code/d' Main.fs +run dotnet test +assert_failure ``` ~~~ @@ -135,9 +145,12 @@ Assert that a specific change in a commit is necessary for the tests to succeed: ~~~ ```git-check-assertions -✓ dotnet test -✓ git checkout HEAD~ Main.fs -✗ dotnet test -Invalid URL +run dotnet test +assert_success + +git checkout HEAD~ Main.fs +run dotnet test +assert_failure +assert_output --partial "Invalid URL" ``` ~~~ diff --git a/bin/git-check-assertions b/bin/git-check-assertions index 6dd5c6e..1195884 100755 --- a/bin/git-check-assertions +++ b/bin/git-check-assertions @@ -24,6 +24,48 @@ set -euo pipefail +# helper functions inspired by bats/bats-assert +run() { + set +e + tmp_output="$(mktemp)" + "$@" 2>&1 | tee "$tmp_output" + status=${PIPESTATUS[0]} + set -e + output="$(cat "$tmp_output")" + rm -f "$tmp_output" + printf '%s\n' "$output" + return 0 +} +assert_success() { + if [ "$status" -ne 0 ]; then + echo "Expected command to succeed, but it failed." + exit 1 + fi +} +assert_failure() { + if [ "$status" -eq 0 ]; then + echo "Expected command to fail, but it succeeded." + exit 1 + fi +} +assert_output() { + if [ "${1:-}" = "--partial" ]; then + local expected="$2" + if [[ "$output" != *"$expected"* ]]; then + echo "Expected output to contain: $expected" + echo "Actual output: $output" + exit 1 + fi + return 0 + fi + if [ "$output" != "$1" ]; then + echo "Expected output to equal: $1" + echo "Actual output: $output" + exit 1 + fi +} +export -f run assert_success assert_failure assert_output + # main flow if [ ! -z "$(git status --porcelain)" ]; then echo "Uncommitted changes. Refusing to run." >&2 @@ -76,69 +118,7 @@ for commit_hash in "${commits[@]}"; do block_num=$((block_num + 1)) echo "git-check-assertions block $block_num:" printf '%s' "$block" | sed 's/^/> /' - check_assertions() { - python3 - "$block" <<-EOF - import sys - import re - import subprocess - - block = sys.argv[1] - steps = [] - current_step = None - - for line_number, raw_line in enumerate(block.splitlines(), start=1): - line = raw_line.rstrip("\\r") - if not line.strip(): - continue - - match = re.match(r"^(\\[success\\]|\\[failure\\]|✓|✗)\\s+(.+)$", line) - if match: - current_step = { - "expects_success": match.group(1) in ("[success]", "✓"), - "command": match.group(2), - "assertions": [], - } - steps.append(current_step) - continue - - if current_step is None: - print( - f"DSL parse error on line {line_number}: " - "assertion text must follow [success], [failure], ✓, or ✗." - ) - sys.exit(1) - - current_step["assertions"].append(line) - - for step in steps: - completed = subprocess.run( - ["sh", "-lc", step["command"]], - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - text=True, - ) - output = completed.stdout or "" - - if output: - sys.stdout.write(output) - if not output.endswith("\\n"): - sys.stdout.write("\\n") - - if step["expects_success"]: - if completed.returncode != 0: - print("Expected command to succeed, but it failed.") - sys.exit(1) - elif completed.returncode == 0: - print("Expected command to fail, but it succeeded.") - sys.exit(1) - - for expected in step["assertions"]: - if expected not in output: - print(f"Expected output to contain: {expected}") - sys.exit(1) - EOF - } - if ! check_assertions; then + if ! bash -euo pipefail -c "$block"; then echo "git-check-assertions block failed in $commit_hash" >&2 restore echo "Returning to $orig_ref" diff --git a/default.nix b/default.nix index e3b912d..c260e4c 100644 --- a/default.nix +++ b/default.nix @@ -11,7 +11,6 @@ shfmt, perl, coreutils, - python3, }: let @@ -45,7 +44,6 @@ resholve.mkDerivation { shfmt perl coreutils - python3 ]; checkPhase = '' @@ -74,7 +72,6 @@ resholve.mkDerivation { bash perl coreutils - python3 ]; execer = [ # Not true at all, but ¯\_(ツ)_/¯ diff --git a/flake.nix b/flake.nix index 0dbfbc8..1fabe89 100644 --- a/flake.nix +++ b/flake.nix @@ -25,7 +25,6 @@ pkgs.perl pkgs.shfmt pkgs.coreutils - pkgs.python3 ]; }; } diff --git a/test/git-check-assertions.bats b/test/git-check-assertions.bats index 476add5..be3b275 100755 --- a/test/git-check-assertions.bats +++ b/test/git-check-assertions.bats @@ -43,7 +43,7 @@ commit_with_assertion() { } @test "should not run any assertion blocks when on main" { - commit_with_assertion "[success] touch ../test" + commit_with_assertion "touch ../test" run git-check-assertions @@ -87,7 +87,7 @@ commit_with_assertion() { } @test "should not run any assertion blocks from main when on a feature branch" { - commit_with_assertion "[success] touch ../test" + commit_with_assertion "touch ../test" git checkout -b feature git commit --allow-empty -m "feature" @@ -100,7 +100,7 @@ commit_with_assertion() { @test "should not run any assertion blocks from master when on a feature branch" { init_repo "$BATS_TEST_TMPDIR/repo-master" master - commit_with_assertion "[success] touch ../test" + commit_with_assertion "touch ../test" git checkout -b feature git commit --allow-empty -m "feature" @@ -112,8 +112,8 @@ commit_with_assertion() { @test "should run all succeeding assertion blocks on the feature branch and finally return to the original branch" { git checkout -b feature - commit_with_assertion "[success] touch ../test1" - commit_with_assertion "[success] touch ../test2" + commit_with_assertion "touch ../test1" + commit_with_assertion "touch ../test2" run git-check-assertions @@ -130,11 +130,11 @@ commit_with_assertion() { test \`\`\`git-check-assertions - [success] touch ../test1 + touch ../test1 \`\`\` \`\`\`git-check-assertions - [success] touch ../test2 + touch ../test2 \`\`\` EOF @@ -159,11 +159,11 @@ commit_with_assertion() { update \`\`\`git-check-assertions - [success] git checkout HEAD~ + git checkout HEAD~ \`\`\` \`\`\`git-check-assertions - [success] grep new readme + grep new readme \`\`\` EOF @@ -174,8 +174,8 @@ commit_with_assertion() { @test "should stop at the first failing assertion block and return to the original branch" { git checkout -b feature - commit_with_assertion "[success] touch ../test1 && exit 3" - commit_with_assertion "[success] touch ../test2" + commit_with_assertion "touch ../test1 && exit 3" + commit_with_assertion "touch ../test2" run git-check-assertions @@ -188,7 +188,7 @@ commit_with_assertion() { @test "should restore worktree when finished" { git checkout -b feature - commit_with_assertion '[success] echo blah >> readme' + commit_with_assertion 'echo blah >> readme' run git-check-assertions @@ -198,8 +198,8 @@ commit_with_assertion() { @test "should restore worktree between commits" { git checkout -b feature - commit_with_assertion '[success] echo blah >> readme' - commit_with_assertion '[success] cp readme ../test' + commit_with_assertion 'echo blah >> readme' + commit_with_assertion 'cp readme ../test' run git-check-assertions @@ -209,7 +209,7 @@ commit_with_assertion() { @test "should restore worktree after a failure" { git checkout -b feature - commit_with_assertion $'[success] echo blah >> readme\n[success] exit 3' + commit_with_assertion $'echo blah >> readme\nexit 3' run git-check-assertions @@ -219,7 +219,7 @@ commit_with_assertion() { @test "should restore index when finished" { git checkout -b feature - commit_with_assertion $'[success] echo blah >> readme\n[success] git add readme' + commit_with_assertion $'echo blah >> readme\ngit add readme' run git-check-assertions @@ -229,8 +229,8 @@ commit_with_assertion() { @test "should restore index between commits" { git checkout -b feature - commit_with_assertion $'[success] echo blah >> readme\n[success] git add readme' - commit_with_assertion '[success] cp readme ../test' + commit_with_assertion $'echo blah >> readme\ngit add readme' + commit_with_assertion 'cp readme ../test' run git-check-assertions @@ -240,7 +240,7 @@ commit_with_assertion() { @test "should restore index after a failure" { git checkout -b feature - commit_with_assertion $'[success] echo blah >> readme\n[success] git add readme\n[success] exit 3' + commit_with_assertion $'echo blah >> readme\ngit add readme\nexit 3' run git-check-assertions @@ -250,7 +250,7 @@ commit_with_assertion() { @test "should skip cached commits on subsequent runs" { git checkout -b feature - commit_with_assertion "[success] touch ../test" + commit_with_assertion "touch ../test" run git-check-assertions @@ -266,7 +266,7 @@ commit_with_assertion() { @test "should not cache failed commits" { git checkout -b feature - commit_with_assertion "[success] touch ../test && exit 3" + commit_with_assertion "touch ../test && exit 3" run git-check-assertions @@ -282,8 +282,8 @@ commit_with_assertion() { @test "should skip cached commits when multiple commits are cached" { git checkout -b feature - commit_with_assertion "[success] touch ../test1" - commit_with_assertion "[success] touch ../test2" + commit_with_assertion "touch ../test1" + commit_with_assertion "touch ../test2" run git-check-assertions @@ -303,10 +303,10 @@ commit_with_assertion() { git checkout -b feature echo commit1 >readme git add readme - commit_with_assertion '[success] grep commit1 readme' + commit_with_assertion 'grep commit1 readme' echo commit2 >readme git add readme - commit_with_assertion '[success] grep commit2 readme' + commit_with_assertion 'grep commit2 readme' run git-check-assertions @@ -322,7 +322,7 @@ commit_with_assertion() { echo goodbye >src echo 'grep goodbye src' >tests git add src tests - commit_with_assertion $'[success] git checkout HEAD~ src\n[failure] bash tests' + commit_with_assertion $'git checkout HEAD~ src\nrun bash tests\nassert_failure' run git-check-assertions @@ -330,18 +330,18 @@ commit_with_assertion() { assert_file_contains src goodbye } -@test "a [success] command should succeed if the executed command succeeded" { +@test "assert_success should succeed if the executed command succeeded" { git checkout -b feature - commit_with_assertion '[success] exit 0' + commit_with_assertion $'run exit 0\nassert_success' run git-check-assertions assert_success } -@test "a [success] command should fail if the executed command failed" { +@test "assert_success should fail if the executed command failed" { git checkout -b feature - commit_with_assertion '[success] exit 1' + commit_with_assertion $'run exit 1\nassert_success' run git-check-assertions @@ -349,18 +349,18 @@ commit_with_assertion() { assert_output --partial "Expected command to succeed, but it failed." } -@test "a [failure] command should succeed if the executed command failed" { +@test "assert_failure should succeed if the executed command failed" { git checkout -b feature - commit_with_assertion '[failure] exit 1' + commit_with_assertion $'run exit 1\nassert_failure' run git-check-assertions assert_success } -@test "a [failure] command should fail if the executed command succeeded" { +@test "assert_failure should fail if the executed command succeeded" { git checkout -b feature - commit_with_assertion '[failure] exit 0' + commit_with_assertion $'run exit 0\nassert_failure' run git-check-assertions @@ -368,84 +368,58 @@ commit_with_assertion() { assert_output --partial "Expected command to fail, but it succeeded." } -@test "a ✓ command should succeed if the executed command succeeded" { +@test "assert_output should succeed if the output matches the given string" { git checkout -b feature - commit_with_assertion '✓ exit 0' + commit_with_assertion $'run echo hello\nassert_output hello' run git-check-assertions assert_success } -@test "a ✗ command should succeed if the executed command failed" { +@test "assert_output should fail if the output does not match the given string" { git checkout -b feature - commit_with_assertion '✗ exit 1' - - run git-check-assertions - - assert_success -} - -@test "you can mix symbols and bracketed asserts in the same block" { - git checkout -b feature - commit_with_assertion $'✓ echo hello\nhello\n[failure] exit 0' + commit_with_assertion $'run echo hello\nassert_output goodbye' run git-check-assertions assert_failure - assert_output --partial "Expected command to fail, but it succeeded." + assert_output --partial "Expected output to equal: goodbye" + assert_output --partial "Actual output: hello" } -@test "an output line should succeed if the output contains the given string" { +@test "assert_output should also check stderr output" { git checkout -b feature - commit_with_assertion $'[success] echo hello\nhello' + commit_with_assertion $'run sh -c "echo err 1>&2"\nassert_output err' run git-check-assertions assert_success } -@test "an output line should fail if the output does not contain the given string" { +@test "assert_output --partial should succeed if output contains the given string" { git checkout -b feature - commit_with_assertion $'[success] echo hello\ngoodbye' - - run git-check-assertions - - assert_failure - assert_output --partial "Expected output to contain: goodbye" -} - -@test "an output line should also check stderr output" { - git checkout -b feature - commit_with_assertion $'[success] sh -c "echo err 1>&2"\nerr' + commit_with_assertion $'run echo hello\nassert_output --partial ell' run git-check-assertions assert_success } -@test "multiple output lines should all be checked against the same command output" { +@test "assert_output --partial should fail if output does not contain the given string" { git checkout -b feature - commit_with_assertion $'[success] printf "hello\\n"\nhell\nello' - - run git-check-assertions - - assert_success -} - -@test "an output line should fail with the missing substring" { - git checkout -b feature - commit_with_assertion $'[success] echo hello\nxyz' + commit_with_assertion $'run echo hello\nassert_output --partial xyz' run git-check-assertions assert_failure assert_output --partial "Expected output to contain: xyz" + assert_output --partial "Actual output: hello" } -@test "executed commands should print command output" { +@test "run should print command output" { git checkout -b feature - commit_with_assertion '[success] sh -c "echo -n hello; echo -n world"' + commit_with_assertion $'run sh -c "echo -n hello; echo -n world"\nassert_success' run git-check-assertions