working with a 30gb file

[text2sql]

question from a novice user here. sorry. how do you work with the main opinions dump? any unarchiver that i use is crashing midway even though i have more than 150 gb free space. unarchiving in python takes forever. and i can't import bz2 directly into pgadmin.
what do you normally do to quickly read an opinions file? thank you!

[mlissner]

Hi, we're working on some scripts for this over in #3174. You might look at those, but I'm not sure how to guide you, honestly. We use Mac and Linux utilities and those seem to work fine. I don't know anything about pgadmin, but it strikes me as too-high-level of a tool.

[text2sql]

Thank you. It says "You must place all uncompressed bulk files in the same directory and set

environment variable BULK_DIR, BULK_DB_HOST, BULK_DB_USER, BULK_DB_PASSWORD" , does it mean I don't need to unarchive .bz2 file?

[mlissner]

Sorry, I'm not in a position to help you debug this, but I think if you're using that script you need to download the bulk files, put them in the same directory, set those variables, and then it should work. Note that that PR is unreviewed though, so it's possible that it has bugs or other issues.

[mlissner]

You closed the discussion. Did you get it?

[text2sql]

no, but that's ok.