New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

example batch scripts for efficient parallel (de)compression of files #10

Open

aculich opened this issue Dec 16, 2014 · 1 comment

Labels

enhancement help wanted

Milestone

Self-sufficiency ...

Contributor

aculich commented Dec 16, 2014

Requested that LBL install efficient parallel (de)compression software on Savio and provide example SLURM batch scripts.

I specifically requested 7z (7zip), but open to alternative recommendations, such as pigz.

pigz - A parallel implementation of gzip for modern multi-processor, multi-core machines.
7z (7zip) with LZMA2 support. LZMA2 is a simple container format that can include both uncompressed data and LZMA data, possibly with multiple different LZMA encoding parameters. LZMA2 supports arbitrarily scalable multithreaded compression and decompression and efficient compression of data which is partially incompressible.

aculich added the enhancement label

aculich added the help wanted label

aculich added this to the Self-sufficiency for Savio-specific HPC basics milestone

Contributor Author

aculich commented Feb 4, 2015

Need to upload fully-working example based on fragment from Yong

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment