Docker Error KeyError

[ameygat]

I have run docker for first time and I get keyerror, it seems code is trying to get postgress user and database, So is it needed to be created on base system ?
There were no instructions to setup DB on https://github.com/isaacmg/fb_scraper/wiki/Docker-Image

variables.list:

FB_ID=myappid
FB_KEY=mysecreate
IDS=cnn,paddlesoft,msnbc
# Include only if you want to scrape comments
COMMENTS=1
# Include below ONLY if you want to use Kafka.
USE_KAFKA=1
KAFKA_PORT=localhost:9092

Error:
docker run --env-file variables.list paddlesoft/fb_scraper Traceback (most recent call last): File "threaded_proc.py", line 6, in <module> from fb_scrapper import scrape_groups_pages File "/fb_scraper/fb_scrapper.py", line 2, in <module> from fb_posts import FB_SCRAPE File "/fb_scraper/fb_posts.py", line 11, in <module> from save_pg import save_post_pg File "/fb_scraper/save_pg.py", line 3, in <module> db = Database(os.environ['db'], user=os.environ['pg_user'], password=os.environ['pg_password'], host=os.environ['pg_host'], database=os.environ['pg_db']) File "/opt/conda/lib/python3.6/os.py", line 669, in __getitem__ raise KeyError(key) from None KeyError: 'db'

[isaacmg]

Yeah I think the newer version by default uses PostgreSQL to save scrapping times as shelving is unstable. So I think you need to set the following
db = Database(os.environ['db'], user=os.environ['pg_user'], password=os.environ['pg_password'], host=os.environ['pg_host'], database=os.environ['pg_db'])
I might create a parameter in the future where you can choose whether to use PostgreSQL too. Also if you want to make PR to add the parameter that would probably be the quickest way.

[ameygat]

But is there already a postgress added in to the Docker or we need to setup that database on the host ?

[isaacmg]

At the moment Postgres is not part of the Dockerfile as it would take up too much memory. The way I was using is as part of a docker-compose with a separate container for PostgreSQL and another for FB scraper. Another easy option is to just set up a Heroku app as those come with a free PostgreSQL database. Just added some documentation on this to the wiki page as well.