Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install Councilmatic in pupa settings file #35

Merged
merged 1 commit into from
Dec 4, 2024
Merged

Conversation

hancush
Copy link
Collaborator

@hancush hancush commented Dec 3, 2024

Overview

A long time ago, in a galaxy far away, we used one scraper repository for many jurisdictional scrapers. These scrapers shared a pupa settings file that needed to account for some jurisdictions scraping into a plain OCD database, while others scraped into a Councilmatic database. Thus, we introduced a SHARED_DB environment variable that, if true, would include councilmatic_core in installed apps.

The script we use to run scrapes sets the SHARED_DB variable, but it was not set for the pupa clean DAG, so the councilmatic_core app was not installed. This resulted in the deletion of top-level entities like organizations and people behaving badly, i.e., not cascading as configured in the Councilmatic app.

This PR does away with the conditional behavior in the settings file. The Metro scrapers are always running against a shared DB.

Connects datamade/django-councilmatic#293

Notes

Once this is merged and the main tag builds, we should be able to run the cleanup DAG on the staging dashboard without error.

Testing Instructions

Ran locally against the staging database to confirm this works!

docker run -it -e DATABASE_URL=${STAGING_DATABASE_URI} scrapers-lametro /bin/bash
# inside container
pupa clean

@hancush hancush requested a review from antidipyramid December 3, 2024 22:33
@hancush hancush merged commit 7670619 into main Dec 4, 2024
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants