Skip to content

Commit

Permalink
Add missing index on 'submissions.created_at'
Browse files Browse the repository at this point in the history
In July 2018 it was noted that the `submissions.created_at`
field ought to be indexed as it was now used in a query:

#23

This is `Task#latest_submission`
(f46dd46)
which is used by the 'no_business' endpoint
(`/v1/tasks/#{task.id}/no_business`)

As the query selecting the most recent submission uses:

`has_one :latest_submission, -> { order(created_at: :desc) }`

I've opted for an index which is 'sorted' to suit.

This work was requested in order to improve the problematic
daily export procedure invoked via
`DataWarehouseExport.generate!` which runs out of memory
and times out regularly.

This unindexed field (`submissions.created_at`) isn't
actually involved in any of the 4 "extractions" which run.
We believe that the 2 problem extractions are

- `Export::Invoices::Extract`
- `Export::Contracts::Extract`

and that improvements to these may be brought about by
adding the following indexes:

- `tasks.updated_at`
- `submissions.updated_at`
- `submission_entries.updated_at`

The following PR will add these indexes.

But we actually believe that the main problem is Postgres
memory:

- the index on SubmissionEntry no longer fits in memory: we
  are going to look into increasing the memory allocation in
  Postgres

- the `submission_entries.*` glob in the SELECT statement
  may include more data than is required: we will
  explore whether a sub-set of the SubmissionEntry fields can
  be returned by the DB

Zendesk
: https://dxw.zendesk.com/agent/tickets/10463

Trello
: https://trello.com/c/GeaLyAWy/1162
  • Loading branch information
edavey committed Dec 4, 2019
1 parent bf0de8d commit f67f64f
Show file tree
Hide file tree
Showing 2 changed files with 9 additions and 1 deletion.
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
class AddIndexOnSubmissionsCreatedAt < ActiveRecord::Migration[5.2]
def change
add_index :submissions,
:created_at,
order: { created_at: :desc }
end
end
3 changes: 2 additions & 1 deletion db/schema.rb
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
#
# It's strongly recommended that you check this file into your version control system.

ActiveRecord::Schema.define(version: 2019_10_23_072354) do
ActiveRecord::Schema.define(version: 2019_12_04_094848) do

# These are extensions that must be enabled in order to support this database
enable_extension "citext"
Expand Down Expand Up @@ -172,6 +172,7 @@
t.decimal "management_charge_total", precision: 18, scale: 4
t.decimal "invoice_total", precision: 18, scale: 4
t.index ["aasm_state"], name: "index_submissions_on_aasm_state"
t.index ["created_at"], name: "index_submissions_on_created_at", order: :desc
t.index ["created_by_id"], name: "index_submissions_on_created_by_id"
t.index ["framework_id"], name: "index_submissions_on_framework_id"
t.index ["submitted_by_id"], name: "index_submissions_on_submitted_by_id"
Expand Down

0 comments on commit f67f64f

Please sign in to comment.