Add missing index on 'submissions.created_at' #559
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
leeky
approved these changes
Dec 5, 2019
| @@ -0,0 +1,7 @@ | |||
| class AddIndexOnSubmissionsCreatedAt < ActiveRecord::Migration[5.2] | |||
| def change | |||
| add_index :submissions, | |||
There was a problem hiding this comment.
Is it worth adding the index concurrently, particularly since we're still inside a submission window - e.g. https://semaphoreci.com/blog/2017/06/21/faster-rails-indexing-large-database-tables.html ?
In July 2018 it was noted that the `submissions.created_at` field ought to be indexed as it was now used in a query: #23 This is `Task#latest_submission` (f46dd46) which is used by the 'no_business' endpoint (`/v1/tasks/#{task.id}/no_business`) As the query selecting the most recent submission uses: `has_one :latest_submission, -> { order(created_at: :desc) }` I've opted for an index which is 'sorted' to suit. This work was requested in order to improve the problematic daily export procedure invoked via `DataWarehouseExport.generate!` which runs out of memory and times out regularly. This unindexed field (`submissions.created_at`) isn't actually involved in any of the 4 "extractions" which run. We believe that the 2 problem extractions are - `Export::Invoices::Extract` - `Export::Contracts::Extract` and that improvements to these may be brought about by adding the following indexes: - `tasks.updated_at` - `submissions.updated_at` - `submission_entries.updated_at` The following PR will add these indexes. But we actually believe that the main problem is Postgres memory: - the index on SubmissionEntry no longer fits in memory: we are going to look into increasing the memory allocation in Postgres - the `submission_entries.*` glob in the SELECT statement may include more data than is required: we will explore whether a sub-set of the SubmissionEntry fields can be returned by the DB Zendesk : https://dxw.zendesk.com/agent/tickets/10463 Trello : https://trello.com/c/GeaLyAWy/1162
edavey
force-pushed
the
add-index-to-submissions-created-at
branch
from
6b18d26 to
ee9e5a1
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In July 2018 it was noted that the
submissions.created_atfield ought to be indexed as it was now used in a query:
#23
This is
Task#latest_submission(f46dd46)
which is used by the 'no_business' endpoint
(
/v1/tasks/#{task.id}/no_business)As the query selecting the most recent submission uses:
has_one :latest_submission, -> { order(created_at: :desc) }I've opted for an index which is 'sorted' to suit.
This work was requested in order to improve the problematic
daily export procedure invoked via
DataWarehouseExport.generate!which runs out of memoryand times out regularly.
This unindexed field (
submissions.created_at) isn'tactually involved in any of the 4 "extractions" which run.
We believe that the 2 problem extractions are
Export::Invoices::ExtractExport::Contracts::Extractand that improvements to these may be brought about by
adding the following indexes:
tasks.updated_atsubmissions.updated_atsubmission_entries.updated_atThe following PR will add these indexes.