Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better error handling for scraper #222

Open
ericyoondotcom opened this issue Feb 5, 2024 · 1 comment
Open

Better error handling for scraper #222

ericyoondotcom opened this issue Feb 5, 2024 · 1 comment

Comments

@ericyoondotcom
Copy link
Contributor

ericyoondotcom commented Feb 5, 2024

When the scraper encounters any sort of error (e.g. a blank HTTP response), the whole script hangs. Add some exception handling so the script can continue and just skip over that one page.

Also, from #223

because the different parts of the scraper mostly run asynchronously, errors are not being properly thrown from the appropriate thread. This leads to errors that can be difficult to debug, because the apparent cause of a crash will be the fact that one thread doesn't return a students list for example, when in fact the actual root cause is something more specific that happened in that thread several thousand lines of logs ago.

@ErikBoesen
Copy link
Collaborator

To clarify, I believe this is in reference to the Departmental portion of the scraper specifically.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants