Hash Stores

As a means for providing files in a trustworthy but expandable form, repositories do not provide the contents of files themselves, but instead merely a list of hashes. These hashes are to be calculated utilizing the SHA256 algorithm on a compressed and, optionally, split archive of the package.

Archive Structure

An archive is comprised of a leading hash and the binary content of the package. The leading hash is to be a UTF8 representation of the hexadecimal SHA256 hash of the binary content of the package.

For an archive that is split into chunks, all portions are to be reassembled by concatenation in the order specified by the repository. Chunks are to be symmetrically encrypted from last to first using Rijndael with the hash of the as-of-yet-unencrypted contents of the preceding chunk in the file, while the first chunk in a file is to be encrypted with the name of the package to which it belongs, encoded in UTF8 representation. Each chunk is individually hashed to generate the identity that will be uploaded to the hash store(s). A hash store must verify identities of individual chunks on upload.

The reassembly and verification sequence from chunks is as follows:

1. Download the requisite chunks from any hash store that responds positively to a request for the given hash

2. Verify the hash for each chunk after downloading; if it does not match, try another hash store

3. Downloads need not occur in order, and should be cached locally

4. In the order specified by the build's details, decrypt each chunk

5. The encryption key for the initial chunk is the name of the package in UTF8

6. Hash the decrypted chunk from each step to determine the key for each following chunk

7. Concatenate all the decrypted chunks into one file

8. Verify the leading hash against the resulting file

9. If this fails to verify, the build metadata the repository provided is inaccurate

10. Failure here means that an alternative build should be suggested to the user, if possible

Technical details

Before splitting into chunks, a leading hash is applied. This hash is prepended to the content as a header, with a 16-bit integer at its beginning used to signify its length in bytes.

The first two bytes of an encrypted archive chunk are a 16-bit integer specifying the length of the IV used to encrypt it. The following bytes N bytes where N is the value of the previous 16-bit integer are the value of the IV. IVs are generated using RijndaelManaged.GenerateIV with default parameters. The remaining bytes of the chunk are its encrypted body. 32-byte encryption keys are derived from the given key using PBKDF2 (Rfc2898DeriveBytes) with 1000 iterations, salted by the IV.