Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CompressingReader: compressed data reader stream (#210) #211

Merged
merged 4 commits into from
Dec 8, 2023

Conversation

oakad
Copy link
Contributor

@oakad oakad commented Nov 13, 2023

In Go, http.Request requires an io.ReadCloser for an upload body. To facilitate compression of data being uploaded we want to wrap a source reader (such an io.File) with a proposed
lz4.CompressingReader, whereupon the said reader can be set directly into http.Request Body field.

This benefits not only the plain http.Request, but also the various network file storage APIs which build on top of Go's http client.

In Go, `http.Request` requires an `io.ReadCloser` for an upload body. To
facilitate compression of data being uploaded we want to wrap a
source reader (such an `io.File`) with a proposed
`lz4.CompressingReader`, whereupon the said reader can be set directly
into `http.Request` Body field.

This benefits not only the plain `http.Request`, but also the various
network file storage APIs which build on top of Go's http client.

Signed-off-by: Alex Dubov <oakad@yahoo.com>
@oakad oakad mentioned this pull request Nov 13, 2023
Out buffer may have some stale overflow data on Reset.

Signed-off-by: Alex Dubov <oakad@yahoo.com>
It is possible for out buffer to have no overflow data and yet
the read offset will be not zero (if byte slices passed by the reader
just happened to align perfectly).

Signed-off-by: Alex Dubov <oakad@yahoo.com>
@pierrec
Copy link
Owner

pierrec commented Dec 7, 2023

CI reports some errors in your test file. Could you have a look?

@oakad
Copy link
Contributor Author

oakad commented Dec 8, 2023

./compressing_reader_test.go:38:17: undefined: os.ReadFile
./compressing_reader_test.go:42:10: undefined: io.NopCloser
./compressing_reader_test.go:50:18: undefined: io.ReadAll
./compressing_reader_test.go:57:17: undefined: io.ReadAll

Looks like your go version is much older than mine. Technically, should not be a problem to go back to io/ioutil for these.

Signed-off-by: Alex Dubov <oakad@yahoo.com>
@pierrec pierrec merged commit e974631 into pierrec:v4 Dec 8, 2023
0 of 9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants