SOLR fails silently and does not restart

[fgregg]

This causes our councilmatics to stop functioning. We need to address this this week.

[reginafcompton]

Related to: Metro-Records/la-metro-councilmatic#232

[reginafcompton]

It looks like a conventional strategy for restarting failed processes on Ubuntu entails the use of a config file (e.g. etc/init/solr.conf/) and Upstart.

[reginafcompton]

N.B. To prevent Solr crashes, we might also think about limiting the query results (since that tends to be what sucks up server memory and brings down solr). However, this may involve removing the Haystack dependency. @jeancochrane has a solution for limiting query results in SFM, which is worth considering.

[reginafcompton]

@evz and I did some work on this. Next steps:

• we can limit the Jetty heap memory by passing in arguments to JETTY_OPTIONS in jetty.conf. I'll add such args and put this file under version control in the Metro repo.

• subscriptions via the RSS feed cause the solr outages, i.e., if someone subscribes to a search that returns thousands of bills, the feed iterates over every bill to build out the RSS (likely here). This process requires a fair amount of memory. Reducing the object size in django-councilmatic resolves this memory problem. Happily, the city-specific repos do not complicate feeds: nyc, chi, metro.

• if we want to restart jetty, then it seems like we'd need to provide a directive in the jetty upstart file (/etc/init.d/jetty).

I will prioritize these tasks in the order above.

[hancush]

Haystack defaults to silent failures. It can easily be configured to raise these exceptions (and use a certain default batch size, plus other options here). See: Metro-Records/la-metro-councilmatic#641