Skip to content

Latest commit

 

History

History
129 lines (86 loc) · 3.86 KB

README.md

File metadata and controls

129 lines (86 loc) · 3.86 KB
Raw

Toronto City Hall API

This repo aims to experiment with using the Grape framework to create an API for Toronto City Hall data gathered via the Open Civic Data scrapers-ca project.

WARNING: I provide no guarantees on the docs being entirely up-to-date. Please don't spend more than a few minutes getting it to work without speaking with me. I'm on the Freenode IRC network as "patcon". [Sign into IRC]

Requirements

  1. Set up the scrapers-ca repo.

  2. Run the Toronto scraper to populate the pupa database:

     pupa update ca_on_toronto

  3. Follow these steps:

bundle install
bundle exec rackup

You should now be able to access the API at

http://localhost:9292/v0/

Documentation

This API is Swagger-compliant. With the service running, the base URL will redirect to the docs:

http://localhost:9292/v0

There are 20 results per page, and you may paginate by adding the page query string.

No guarantees of stability while the API is labelled as v0.

Live Demo

A relatively up-to-date demo API is running at:

https://tocityhall-api.herokuapp.com/v0

It is using data scraped from my in-progress improvements to the Toronto OCD dataset:

https://github.com/patcon/scrapers-ca/tree/toronto-committee-orgs

The Swagger docs can be viewed here:

https://tocityhall-api.herokuapp.com/v0

Related Resources

There are a few wonderful resources related to the current hosted OCD API. I'm very appreciative of the prior art, but am interested in what I consider the more RESTful approach of APIs built on Grape.

Heroku

Clean scrape to Heroku database

cd path/to/this/app
# Set the local DATABASE_URL envvar to that of the Heroku DB
heroku config --shell | grep DATABASE_URL | source /dev/stdin

cd path/to/scrapers-ca
# Get Toronto-specific listing of division IDs, and set envvar for usage
wget https://gist.githubusercontent.com/patcon/2aecb6dae0a87c405bea/raw/25badea7220d7ddc4c80eb578683d10efce7c086/country-ca-toronto.csv
OCD_DIVISION_CSV=country-ca-toronto.csv

export DATABASE_URL OCD_DIVISION_CSV

# WARNING: This will drop all databases in the Heroku app database.
pupa dbinit --reset ca

# Run all Toronto scrapers, or optionally run specific scrapers
pupa update ca_on_toronto # [scraper1 scraper2, ...]

# Unset envvars to make sure we're not acting on Heroku DB later
unset DATABASE_URL OCD_DIVISION_CSV

Deployment

We are using Heroku Pipelines for deployment.

Heroku Pipeline screenshot

This means:

  • Code is auto-deployed from master to the staging app.
  • Code is promoted on-demand from the staging app to the production app.
  • Staging and production have their own databases.
  • Review apps are automatically created for each pull request from a branch in the main repo (ie. not forks). These apps are connected to the staging database. [Example]

The above process only migrates code. To migrate the database from staging to production:

bundle exec rake tocityhall:migrate_stage_to_prod

Warning: This is a destructive action on production database. It will prompt for confirmation.

Contributors

See the contributors graph.