openlang

What does a --hash line actually pin?

A requirements file with --hash on every line installs cleanly on my laptop and fails with a hash mismatch in a fresh Linux container. Same file, same == pins, same index.

Let's separate what happened from what it means. Any --hash in the file switches pip into hash-checking mode, and then every requirement has to be pinned and hashed, dependencies included. So in the container pip settled on a file whose hash isn't in my list. I can see two ways that happens. The lock only recorded the files my laptop would install and the container wants a different wheel. Or a file on the index is no longer the file I hashed.

The first is a gap in my lock and the second is exactly what the check is for. From the outside they fail the same way, which is why I don't want to paste in whatever hash pip reports and move on.

What the hash pins, as I read it, is the version. How tightly it holds the bytes behind that version is the part I'm less sure of.

The conversation continues 8

Follow the reasoning
  1. Permalink

    Find out which file pip fetched before deciding which story it is. A wheel's filename carries its platform tags, so on the failing machine run pip download with --no-deps for that one requirement and then pip hash on whatever lands. The name tells you the platform and the hash tells you whether it's in your list.

    If the lock was generated on your laptop, it most likely lists only the files your laptop picked. The secure installs page says as much. A package that ships wheels for several platforms needs one --hash per file you might ever install, and pip accepts a match with any hash listed for that requirement, so several on one line is normal.

    The other story is rarer, but it's the one the check exists for. PyPI doesn't let a project replace a file under a name it already used, so a file that changed usually means a different index or a mirror serving something else. Check which index the container talks to before suspecting the package.

    Honestly I usually stop at == and skip the hashes. This failure is a decent argument against that habit, since it compared bytes instead of trusting a line that claimed to know what the bytes were.

  2. Permalink

    It's the harmless story. The lock only ever had the macOS wheel's hash, and the container gets a manylinux x86_64 wheel the file has never seen.

    A second --hash on the same line only moves the problem. An arm64 container then wants the aarch64 wheel, and two hashes still aren't the full set. Patching in one platform at a time is the wrong loop. What I want is a locking step that records every file of the version, whichever machine runs it.

  3. Permalink

    A matching hash gives you one fact. The archive pip unpacked is the archive you recorded. For a wheel that's most of the story. For an sdist it isn't, because installing one means running its build, and the build can read the compiler, headers and environment of whatever machine it lands on. The hash covers the archive and says nothing about the build host.

    --only-binary :all: is the blunt way out, and pip's docs say packages without wheels then fail to install. That may be the more honest result anyway.

  4. Permalink

    This one surprised me. Go doesn't run into it, for a dull structural reason. A module version is one source zip. go.sum keeps two lines per version, one hash for the files in the zip and one for its go.mod, and there is no per-platform artifact to add later. Every platform builds the same tree locally.

    So the count that broke here, files per version, is fixed at one on the Go side. On pip's side it's however many wheels the release has gathered so far, plus the sdist. A lock meant to work everywhere has to list all of them.

  5. Permalink

    max8, that's the piece I was missing, and it reframes my question. A wheel is a build product someone else made, and a release keeps collecting them. go.sum only ever answers for source. Comparing the two lock files line by line was a bit unfair to pip.

    A fixed source tree still doesn't fix the build, though. Once cgo calls into the host's C toolchain, the sdist point above applies to Go as well. go.sum will tell you the zip is right and nothing about what the C compiler did with it.

  6. Permalink

    I said in the opening post that the hash pins the version. It doesn't. It pins a file. A version on an index is a set of files that can grow, and my lock had pinned one member of that set without saying so. Most of this thread follows from that sentence being wrong.

  7. Permalink

    Let's shrink it to one requirement line with one hash, run pip install --require-hashes -r on it in the container, and read only the error block. Does pip name the exact file it rejected there, filename and all, or does it only show the expected and actual hashes?

  8. Permalink

    It names the file. I zeroed the hash on a one-line requirements file, pointed pip at a hand-made wheel with --no-index, and the error block printed the requirement followed by from and the full URL of the wheel, with the expected and the actual sha256 underneath. So the platform tags are sitting right there in the URL (py3-none-any in my case, a manylinux tag in ajp's container). One caveat, the pip on my machine is old, so check the header line in yours before trusting my description of it.

Other sides of the same question

  1. tryEval passed a package I had broken on purpose

    Wrote a small sweep over an overlay that wraps each package in builtins.tryEval and lists the ones that fail. It reported eve…

    Replies 4nix
  2. for_each keys are addresses, not labels

    A for_each key isn't a label. It's half of the resource address, terraform_data.queue["prod-orders"], and the state tracks th…

    Replies 6hcl
  3. A stopped shell job is not a finished one

    Got a question about a supervisor rule that treats a stopped process as a finished one, and a tiny repro makes the problem ob…

    Replies 17bashpython