Endpoint to match agencies for submission

[josh-chamberlain]

Context

• related to CKAN general scraper scrapers#246
• related to Muckrock tools scrapers#245

We have tools which generate batches of Data Sources to submit to our db. The biggest hurdle is typically that agencies out in the world aren't named the exact same thing as they are in our database; we need an agency_described which matches our database. There are 3 potential things that can happen:

• * happy: the name is exactly the same in our database, and we get a single match. hooray!
• ~ different: the agency exists in our database by a different name.
• + new: the agency does not exist in our database and we need to add it.
• x unknown: we don't even have an external name for the agency; we have no idea what to guess. that's out of scope but begun here: Identify agencies data-source-identification#15

Requirements

• make an agency-match endpoint which we can use to find proper names for our agencies
• It should accept a single request with
  • external name
  • county
  • state
  • locality
• It should return
  • agency name: great for CSV submissions
  • agency id: good for hitting the API
• It should return one of 3 possible entries:
  • exact match, if one confident match is detected; this covers the happy and different paths
  • possible match, an exact match is not found, but there are one or more agencies within the same location which might match
  • no match, No agencies found for that location and/or meet the threshold for being a possible match.
• Determine if this should be an endpoint vs. an internal tool used by the scrapers. For example, a read-only SQLite database of agencies (which could minimize endpoint calls).
  • determined that this should be an endpoint because of the broad usage
• Agencies may have the same name across different states or counties. we should disambiguate using counties or states.

Open questions

How would we do this internally?

We have several possible ways of approaching this, depending on our preferences and how thorough we want to be. This includes:

• Exact matching -- Probably will do this regardless as a means of first-past-the-post checking
• Fuzzy string matching -- things like Levenshtein distance, Jaro-Winkler similarity, or Cosine similarity
• Phonetic matching
• N-gram matching
• Machine learning-based solutions

[maxachis]

Looking at this now!

@josh-chamberlain When we say it should accept a batch, are we thinking of providing that batch via csv, json, or both?

[maxachis]

In terms of implementation, here's my initial thinking:

Steps

1. Get all agencies whose location id match the location id that is derived from the state/county/locality
   a. This assumes exact matching.
2. Of values retrieved, perform fuzzy matching on names. Exact matches are labeled as such, while possible matches (i.e. those where the similarity is above some threshold) are included. Anything below the threshold is not returned.

Possible Outcomes

A. Exact match
B. Possible matches
C. No matches

Other Considerations

• This will be premised on locations being exactly correct. Misspellings or alternative names (e.g., "St." instead of "Saint") won't be considered.
• If successful, this could be extended to other components, such as data sources.