Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add total package count and improved pager #1038

Merged
merged 1 commit into from
Oct 11, 2024

Conversation

djjuhasz
Copy link
Collaborator

@djjuhasz djjuhasz commented Oct 3, 2024

Fixes #988: The package list shows the total number of packages found, before paging.

Other changes:

  • The package list subtitle now indicates which packages are currently displayed as well as the total number of results, e.g. "Showing 1 - 20 of 228"
  • The pager is only shown if there are more results than the page limit, i.e. if 27 results are returned an the page limit is 20
  • The pager includes individual page links for up to 11 pages
  • If more than 11 pages of results are returned, "First" and "Last" page links are shown
  • The pager page list includes ellipses to indicate when their are more pages then the indivdual page links shown. I.e. if 15 pages of results are returned and the current page is page 1, then an list item containing an ellipsis is shown after the list item for page 11.

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 3, 2024

Total result display:
image

Updated pager:
image

@djjuhasz djjuhasz force-pushed the dev/issue-988-update-dashboard-paging branch from 5ffcfd6 to 6b91f06 Compare October 3, 2024 18:26
Copy link

codecov bot commented Oct 3, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 55.38%. Comparing base (f6bf851) to head (7f26650).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1038   +/-   ##
=======================================
  Coverage   55.38%   55.38%           
=======================================
  Files         103      103           
  Lines        6021     6021           
=======================================
  Hits         3335     3335           
  Misses       2434     2434           
  Partials      252      252           

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 3, 2024

Pager with ellipses and "First" / "Last" page links:
image

@djjuhasz djjuhasz force-pushed the dev/issue-988-update-dashboard-paging branch 2 times, most recently from 7ea9144 to a30763e Compare October 3, 2024 18:45
@djjuhasz djjuhasz changed the title Add total result count and improved pager Add total package count and improved pager Oct 3, 2024
@djjuhasz djjuhasz requested a review from jraddaoui October 3, 2024 18:57
@djjuhasz djjuhasz force-pushed the dev/issue-988-update-dashboard-paging branch 3 times, most recently from 1e6bfa8 to daab09e Compare October 3, 2024 23:10
Copy link
Collaborator

@jraddaoui jraddaoui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @djjuhasz, and sorry for the late feedback!

You went with a fully fledged pagination, I like the implementation in general but I have three main concerns:

Responsiveness

Mostly covered in my comments below, I'd like to address that in this PR if we go with this full pagination.

Re-usability

We should implement this as a component that could be used in other parts of the dashboard (or easily migrated to other UIs). This can happen in another PR, where we can also think about the following point ...

Performance

While I like to have the option to do full pagination in the API, it was a request in another project, using offset and limit. I think we should try to use cursor based pagination when possible ourselves, specially if you are showing that last page link, that could be heavy on MySQL. With better filtering and sorting options, the next/previous pagination would be better/enough in most cases (like the Github commits list). Or use scroll based pagination in other places (like the Gitlab commits list).

dashboard/src/pages/packages/index.vue Show resolved Hide resolved
dashboard/src/pages/packages/index.vue Outdated Show resolved Hide resolved
dashboard/src/pages/packages/index.vue Outdated Show resolved Hide resolved
dashboard/src/stores/package.ts Outdated Show resolved Hide resolved
dashboard/src/stores/package.ts Show resolved Hide resolved
dashboard/src/pages/packages/index.vue Outdated Show resolved Hide resolved
dashboard/src/pages/packages/index.vue Outdated Show resolved Hide resolved
@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 8, 2024

@jraddaoui with regards to performance, I disagree about using cursor based paging instead of offset & limit for the following reasons:

  1. I've never noticed a significant performance hit using offset & limit for paging a MySQL query. Offset and limit are supported by the MySQL query language, and presumably are well optimized by MySQL. There's no noticeable impact on performance in this pager when jumping to the last page of a 10,000 package record set.
  2. I don't anticipate clients using Enduro to have hundreds of thousands of packages in the system, and even if they do we don't know if that will have a significant performance impact. In a very large dataset, I think sorting and searching will likely to have a much higher performance impact that offset paging.
  3. I don't think we should be pre-optimizing for performance, especially considering the additional complexity of implementing cursor based paging (see below).
  4. Cursor-based paging is significantly more complicated to implement, especially if you are sorting data on a non-unique field (e.g. name). You may have hundreds of "Untitled" packages (or similar) in a record set, and with cursor-based paging it's complicated to go to the next page of that result set. Cutting out cursor-based paging from the query filtering code made the code significantly simpler.
  5. Cursor-based paging provides less functionality than offset paging — you can't jump to an arbitrary point in the result set.

@jraddaoui
Copy link
Collaborator

@fiver-watson here!

@jraddaoui
Copy link
Collaborator

Running locally, with a clean database, just created 1M packages with the genpkgs script:

mysql> SELECT * FROM package LIMIT 20 OFFSET 999980;
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id      | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
|  999981 | DPJ-SIP-d9ec93a8-cf32-4851-9861-d7ec6c7ae43f.tar | processing-workflow-e9e66b85-7b01-4527-a60e-9d4f98501231 | 926abf67-d48a-4e80-b5a8-f14e8dcae3e9 | aca325a5-5c77-450e-8c8f-ac2ee318bb69 | 79d21fae-1b33-4947-a09a-e6a077be8a4d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
.
| 1000000 | DPJ-SIP-0038482a-5a62-4862-ac1f-73a1697cfbe5.tar | processing-workflow-130cb0dc-8cf2-4c79-853f-9e0d6c3ad0f9 | a36bb94b-5659-4eed-91a4-ad911558fee0 | 47eecf98-4f89-41c1-a2b9-33e6a247df47 | ee4bd777-37da-484c-a9b6-625f6592219d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.30 sec)

mysql> SELECT * FROM package WHERE id>999980;
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id      | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
|  999981 | DPJ-SIP-d9ec93a8-cf32-4851-9861-d7ec6c7ae43f.tar | processing-workflow-e9e66b85-7b01-4527-a60e-9d4f98501231 | 926abf67-d48a-4e80-b5a8-f14e8dcae3e9 | aca325a5-5c77-450e-8c8f-ac2ee318bb69 | 79d21fae-1b33-4947-a09a-e6a077be8a4d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
.
| 1000000 | DPJ-SIP-0038482a-5a62-4862-ac1f-73a1697cfbe5.tar | processing-workflow-130cb0dc-8cf2-4c79-853f-9e0d6c3ad0f9 | a36bb94b-5659-4eed-91a4-ad911558fee0 | 47eecf98-4f89-41c1-a2b9-33e6a247df47 | ee4bd777-37da-484c-a9b6-625f6592219d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+---------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.01 sec)

@djjuhasz djjuhasz force-pushed the dev/issue-988-update-dashboard-paging branch 2 times, most recently from 83c123b to 2297a7f Compare October 9, 2024 20:53
@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 9, 2024

Also tested with 1M packages and an index on the name column:

mysql> SELECT * FROM package order by name desc limit 20 offset 999980;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 793806 | DPJ-SIP-00014fd4-d54f-473b-b0ed-ca2b61847b19.tar | processing-workflow-f8a55c36-4f7d-40d2-be23-68900cfce89c | 8d07ad2b-a9be-4a8b-b548-e3507334380d | 9fed38cd-dd6a-4c80-97df-815bd949f705 | a6b85072-6786-4a4e-8657-3f965029861d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 307987 | DPJ-SIP-00000779-abb1-4ced-aeee-598e6bf53364.tar | processing-workflow-4f89598c-0919-4485-b2a5-c2bc415e41eb | e069a871-e8cb-4085-93dc-7c7735f10f35 | ccaa41d5-86bb-48dd-af0f-594243e03f08 | addbf45d-758c-43cb-9a7a-7d2536214cb0 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (1.27 sec)
mysql> SELECT * FROM package where name <= "DPJ-SIP-00014fd4-d54f-473b-b0ed-ca2b61847b19.tar" order by name desc limit 20;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 793806 | DPJ-SIP-00014fd4-d54f-473b-b0ed-ca2b61847b19.tar | processing-workflow-f8a55c36-4f7d-40d2-be23-68900cfce89c | 8d07ad2b-a9be-4a8b-b548-e3507334380d | 9fed38cd-dd6a-4c80-97df-815bd949f705 | a6b85072-6786-4a4e-8657-3f965029861d |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 307987 | DPJ-SIP-00000779-abb1-4ced-aeee-598e6bf53364.tar | processing-workflow-4f89598c-0919-4485-b2a5-c2bc415e41eb | e069a871-e8cb-4085-93dc-7c7735f10f35 | ccaa41d5-86bb-48dd-af0f-594243e03f08 | addbf45d-758c-43cb-9a7a-7d2536214cb0 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.01 sec)

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 9, 2024

1M packages, offset of 10 000:

mysql> SELECT * FROM package order by name desc limit 20 offset 10000;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
|  11773 | DPJ-SIP-fd66526b-ad01-455c-ae9e-d9b4c0dd0ff5.tar | processing-workflow-36ed04c5-11b1-4bc8-ae76-a3a8b136a5d1 | b1caf82d-28e4-49de-9c72-c02bedb7ecb4 | f40e7d0c-2e3c-4981-ba24-5eceb309470a | 433d3889-e9da-461a-9d32-21938ec5e446 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 223885 | DPJ-SIP-fd659c44-3fe2-4019-a211-4c44d6e87207.tar | processing-workflow-5cd3b960-06e0-4959-85fd-f68fd44ec054 | c98772dd-bef6-4b85-9b78-05f9e4b94e6f | 53471c2f-32ae-4bc5-a9e5-890288b37bc2 | 5f070e35-504b-4336-b333-efc6cedfabe8 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.85 sec)
mysql> SELECT * FROM package where name <= "DPJ-SIP-fd66526b-ad01-455c-ae9e-d9b4c0dd0ff5.tar" order by name desc limit 20;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
|  11773 | DPJ-SIP-fd66526b-ad01-455c-ae9e-d9b4c0dd0ff5.tar | processing-workflow-36ed04c5-11b1-4bc8-ae76-a3a8b136a5d1 | b1caf82d-28e4-49de-9c72-c02bedb7ecb4 | f40e7d0c-2e3c-4981-ba24-5eceb309470a | 433d3889-e9da-461a-9d32-21938ec5e446 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 223885 | DPJ-SIP-fd659c44-3fe2-4019-a211-4c44d6e87207.tar | processing-workflow-5cd3b960-06e0-4959-85fd-f68fd44ec054 | c98772dd-bef6-4b85-9b78-05f9e4b94e6f | 53471c2f-32ae-4bc5-a9e5-890288b37bc2 | 5f070e35-504b-4336-b333-efc6cedfabe8 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.44 sec)

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 9, 2024

1M packages, offset of 1000:

mysql> SELECT * FROM package order by name desc limit 20 offset 1000;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 967333 | DPJ-SIP-ffbcc9ef-9e4f-4199-83ae-8ce328f91c21.tar | processing-workflow-ee9a7d39-5e97-4e56-a99d-122067ec9795 | 97294e6b-a372-424a-997e-95db61973fab | a9347c73-a9cc-4aed-b790-55d28bd0db0d | c134e987-186c-425f-9629-8d0bb9b40fe1 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 
.
.
| 426927 | DPJ-SIP-ffbbaf45-9312-470b-a849-cb27f1f19061.tar | processing-workflow-da825d9e-b8a8-47c4-93d3-da4f603279e0 | 625eceb3-95a6-4639-84e7-b98fb6820226 | 22993a1b-6e21-47c1-b3ee-8a6b6db89b4b | 967c2e5f-f715-477f-b4fa-4d4ff784d54b |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.64 sec)
mysql> SELECT * FROM package where name <= "DPJ-SIP-ffbcc9ef-9e4f-4199-83ae-8ce328f91c21.tar" order by name desc limit 20;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 967333 | DPJ-SIP-ffbcc9ef-9e4f-4199-83ae-8ce328f91c21.tar | processing-workflow-ee9a7d39-5e97-4e56-a99d-122067ec9795 | 97294e6b-a372-424a-997e-95db61973fab | a9347c73-a9cc-4aed-b790-55d28bd0db0d | c134e987-186c-425f-9629-8d0bb9b40fe1 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 426927 | DPJ-SIP-ffbbaf45-9312-470b-a849-cb27f1f19061.tar | processing-workflow-da825d9e-b8a8-47c4-93d3-da4f603279e0 | 625eceb3-95a6-4639-84e7-b98fb6820226 | 22993a1b-6e21-47c1-b3ee-8a6b6db89b4b | 967c2e5f-f715-477f-b4fa-4d4ff784d54b |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.42 sec)

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 9, 2024

1M packages, offset of 100:

mysql> SELECT * FROM package order by name desc limit 20 offset 100;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 976958 | DPJ-SIP-fff967fa-f44b-4f2e-b4ef-63a10fa05687.tar | processing-workflow-7c7d97d8-b4a7-4d61-af33-d4658b723aa6 | 44679ea8-96b2-4247-be39-6b8a5ded1e27 | 84f6d6ce-0988-4530-8cdb-390600eb058b | 472226a8-cda9-4e75-99da-56188fb82ec8 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
.
.
| 556471 | DPJ-SIP-fff7f017-59ad-47ee-9a59-bd1b3903f8d1.tar | processing-workflow-8cc8e528-b8db-4f69-bbb7-984888f1b987 | 06c40294-3508-4192-990b-338aaf1e19cf | abb9182b-f977-4940-a042-715f58a45dda | c39c4bab-ce25-46cb-9d45-9b1592a7b552 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.42 sec)
mysql> SELECT * FROM package where name <= "DPJ-SIP-fff967fa-f44b-4f2e-b4ef-63a10fa05687.tar" order by name desc limit 20;
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| id     | name                                             | workflow_id                                              | run_id                               | aip_id                               | location_id                          | status | created_at                 | started_at                 | completed_at               |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
| 976958 | DPJ-SIP-fff967fa-f44b-4f2e-b4ef-63a10fa05687.tar | processing-workflow-7c7d97d8-b4a7-4d61-af33-d4658b723aa6 | 44679ea8-96b2-4247-be39-6b8a5ded1e27 | 84f6d6ce-0988-4530-8cdb-390600eb058b | 472226a8-cda9-4e75-99da-56188fb82ec8 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 
.
.
| 556471 | DPJ-SIP-fff7f017-59ad-47ee-9a59-bd1b3903f8d1.tar | processing-workflow-8cc8e528-b8db-4f69-bbb7-984888f1b987 | 06c40294-3508-4192-990b-338aaf1e19cf | abb9182b-f977-4940-a042-715f58a45dda | c39c4bab-ce25-46cb-9d45-9b1592a7b552 |      2 | 2019-11-21 17:36:10.000000 | 2019-11-21 17:36:11.000000 | 2019-11-21 17:42:12.000000 |
+--------+--------------------------------------------------+----------------------------------------------------------+--------------------------------------+--------------------------------------+--------------------------------------+--------+----------------------------+----------------------------+----------------------------+
20 rows in set (0.43 sec)

@djjuhasz
Copy link
Collaborator Author

djjuhasz commented Oct 10, 2024

To summarize my tests with offset vs. cursor searches on a set of one million packages sorted by name (descending) and various offsets:

Observations:

  1. For an offset of 100, offset paging and cursor paging were roughly equivalent (~0.4s). It appears in these cases almost all the search time is used to sort the results
  2. For an offset of 1000, offset paging is slightly slower (~0.6s) than cursor paging (~0.4s)
  3. For an offset of 10 000, offset paging is twice as slow (~0.8s) as cursor paging (~0.4s)
  4. For an offset of 999 980, offset paging time is (~1.2s) and cursor paging is nearly instantaneous (~0.01s)!

Trends:

  1. For paging with an offset of 100 packages, the MySQL search times for offset paging and cursor paging are equivalent — the difference is less than the time variation when running the same query multiple times.
  2. Paging times for offset paging appear to grow logarithmically as the offset increases.
  3. Paging times for cursor paging are constant for most offsets, but then drop to zero near the last page. 😕

Conclusions:

  1. For paging through the first 50 pages (offset < 1000) users will probably not notice a difference between offset and cursor paging even with 1 million packages.
  2. It seems unlikely that most users would incrementally page through more than 10 or 20 pages, and in this range offset paging and cursor paging are pretty much the same performance wise.
  3. Jumping to the last page of a very large dataset with offset paging is slow enough that it would definitely be noticeable (1.2s), but if this is a problem we could just remove the "last" page link from the pager.
  4. I think the reduced complexity and additional functionality of offset paging more than makes up for the reduced performance, especially in most real-world paging scenarios

@jraddaoui
Copy link
Collaborator

Yeah, that was my understanding too, I am okay removing the last page link.

@djjuhasz
Copy link
Collaborator Author

@jraddaoui as we agreed in the scrum today, I'll leave the first/last page links on the pager for now.

Fixes #988: The package list shows the total number of packages found,
before paging.

Other changes:
- Show text indicating which packages are currently displayed as well
  as the total number of results, e.g. "Showing 1 - 20 of 228"
- Show the pager only when there are more results than the page
  limit, e.g. if 27 results are returned an the page limit is 20
- Include individual page links for up to 7 pages
- Show "first" and "last" page links when more than 7 pages of results
  are returned
- Show an ellipsis before or after the page links to indicated there are
  more pages than the page links show
- Hide page links and ellipses on narrow screens
@djjuhasz djjuhasz force-pushed the dev/issue-988-update-dashboard-paging branch from c9d03f2 to 7f26650 Compare October 11, 2024 00:02
@djjuhasz djjuhasz merged commit 566cfca into main Oct 11, 2024
15 checks passed
@djjuhasz djjuhasz deleted the dev/issue-988-update-dashboard-paging branch October 11, 2024 00:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Problem: Packages count only ever counts packages listed on current page
2 participants