Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skip handling of known noisy/invalid binary files #3706

Open
rgmz opened this issue Dec 2, 2024 · 0 comments
Open

Skip handling of known noisy/invalid binary files #3706

rgmz opened this issue Dec 2, 2024 · 0 comments
Labels
enhancement

Comments

@rgmz
Copy link
Contributor

rgmz commented Dec 2, 2024 •
edited
Loading

There are many known bad test files which produce a lot of noise. It would be nice if these were skipped, as they're known to be invalid/useless.

Examples

github.com/microsoft/go-winio/archive/tar/testdata/

    2024-11-16T21:40:54-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/cosmos/ethermint_abci.git", "commit": "07dad5f", "path": "vendor/github.com/microsoft/go-winio/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 2147483648}
    2024-11-16T21:40:54-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/cosmos/ethermint_abci.git", "commit": "07dad5f", "path": "vendor/github.com/microsoft/go-winio/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 2147483648}

xz test files?

    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-2.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-3.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-4.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-5.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-6.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-7.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-lzma2-8.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-1.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-2.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error unarchiving chunk.	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c652", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-stream_flags-3.xz", "timeout": 60, "mime": "application/x-xz", "timeout": 60, "error": "error creating reader for decompressor with format: .xz unable to detect MIME type: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-vli-1.xz", "error": "failed to create file reader to handle file: error identifying archive: matching zip: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-1-vli-2.xz", "error": "failed to create file reader to handle file: error identifying archive: matching tar: xz: data is corrupt"}
    2024-11-15T12:15:57-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/aws/audit-plugin-for-mysql.git", "commit": "2c8c65297865d9f8da501761f46e2a34e29af603", "path": "storage/tokudb/PerconaFT/third_party/xz-4.999.9beta/tests/files/bad-2-compressed_data_padding.xz", "error": "failed to create file reader to handle file: error identifying archive: matching rar: xz: data is corrupt"}

libbogus

2024-11-14T09:47:59-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "d17dee5", "path": "test/tools/llvm-readobj/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:02-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "8a83602", "path": "test/tools/llvm-objdump/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:02-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "cec5045", "path": "test/tools/llvm-readobj/Inputs/library.lib", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus11.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"~97&\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus12.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"#55!\": invalid syntax"}
2024-11-14T09:48:03-05:00	error	trufflehog	error processing AR files	{"repo": "https://github.com/apple/swift-llvm.git", "commit": "e5d1578", "path": "test/tools/llvm-objdump/Inputs/libbogus14.a", "timeout": 60, "mime": "application/x-archive", "timeout": 60, "timeout": 60, "error": "error reading AR payload: strconv.Atoi: parsing \"1foobar273\": invalid syntax"}

github.com/pierrec/lz4/fuzz/corpus

2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/226780b32ba8f87ec614fdb376aa0884011c4ca9-17", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/234cc427d9be32470f3c2e11a6bc16567f558e55-22", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2486a84bf0f161f45b050d9c19ea9e35f5def864-8", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/276580343a14eec04143e89a778dae3e14df472c-17", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/27fb5dc4016dc640e55a60719a222c38c604fa6b-2", "error": "failed to create file reader to handle file: error identifying archive: matching zip: bzip2: corrupted input: mismatching stream checksum"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2a33d8514fb512aa20b0a56800cd3e12f3952b6b-26", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2c2a5947341d76797a7e2299f39d01e3aebb2eb8-19", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2cc2308b75a2e8f7eafcf69370767e5fce314892-13", "error": "failed to create file reader to handle file: error identifying archive: matching zip: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2cdafdadb156e2759c389b6b8edf6a402034886c-26", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2de93224b5f0db491ced1ec491a9f41d71820671-11", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid block size"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2e8487cf61feda70c0d74f12bfb5b692b684f82a-9", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2f0ee9cf4bb951a37efc6460d5709442bc3de54e-6", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid frame checksum: got 2cc5d05; expected 206e6168"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/2fad20024167a500cdb8df5334a614f113efae00-20", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/300579a548d96d64c9da8470efa15e787f1a36f1-28", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:26-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/31c6c22708d346ed9e936fa7e77c8d9ab6da8d1e-33", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:27-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/344d38ec2ec90cb617e809439938b4cbf3b11f02-10", "error": "failed to create file reader to handle file: error identifying archive: matching rar: lz4: invalid source or destination buffer too short"}
2024-12-01T23:27:27-05:00	error	trufflehog	error handling binary file	{"repo": "https://github.com/pismo-foss/zipkin-go-opentracing.git", "commit": "6a5cc3c2821285801748ce60af6c64bef49138de", "path": "vendor/github.com/pierrec/lz4/fuzz/corpus/352631eab692c4a2c378b231fb3407ebcc0c3039-33", "error": "failed to create file reader to handle file: error identifying archive: matching 7z: lz4: invalid source or destination buffer too short"}

Git

2024-12-02T18:10:59-05:00	info-2	trufflehog	skipping file: size exceeds max allowed	{"repo": "https://github.com/Shopify/git.git", "commit": "e51217e", "path": "t/t5000/huge-and-future.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "huge", "size": 68719476737, "size": 68719476737, "limit": 2147483648}

Misc

2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "9d3d370", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/gnu-sparse-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "gnu-sparse", "size": 60000000000, "size": 60000000000, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/pax-sparse-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "pax-sparse", "size": 60000000000, "size": 60000000000, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "e0ab505", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "7ae9561", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "7ae9561", "path": "src/archive/tar/testdata/writer-big.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit": 21474
83648}
2024-12-04T02:20:47Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "c991d2a", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:20:51Z    info-2  trufflehog      skipping file: size exceeds max allowed {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "2d3cd51", "path": "src/archive/tar/testdata/writer-big-long.tar", "timeout": 60, "mime": "application/x-tar", "timeout": 60, "filename": "16gig.txt", "size": 17179869184, "size": 17179869184, "limit":
2147483648}
2024-12-04T02:21:02Z    error   trufflehog      error handling binary file      {"repo": "https://github.com/IBM/go-with-wakeup-profile.git", "commit": "139fad21b937dc0b405d243e3abe33429890947f", "path": "src/compress/gzip/testdata/issue6550.gz", "error": "failed to create file reader to handle file: error identifying archive: matching tar: gzip: invalid heade
r"}

Related

This relates to #1517 in spirit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
enhancement
Milestone
No milestone
Development

No branches or pull requests
1 participant
@rgmz