Cancel tasks

[curquiza]

Because of GitHub bug, I cannot transfer the discussion, but here is the original one: meilisearch/meilisearch#2156

The feature request is about canceling the tasks.

Technical notes from @MarinPostma:

See, while in appearance this seems like a very simple feature, it becomes tricky when we add that in a concurrent/distributed model, and the design of this feature still requires some more thinking to be sure we're not missing anything

[FutureFutureTo]

Hmm, okay, well how about the ability to just wipe everything and rebuild?

The pending corrupted task queue seems to prevent any action at all from happening on my end.

[gmourier]

Hi @FutureFutureTo

Do you expect to cancel enqueued tasks? What about processing one?

Also, do you think you would need to delete old tasks to clear the number of tasks?

Thanks

[FutureFutureTo]

yea, id basically want the ability to remove all tasks. even old ones if needed.

my specific reason is this:
I am testing different implementations of reindexing based upon outside parameters. if i by chance hit an error (still testing), the task queue becomes completely backed up and i have no escape but remove and reinstall meilisearch, which is very time consuming and frustrating when developing.

ie.

1. I create 4 indexes: products, users, categories, tags.
2. I run a job to update all of the products nightly, but my job gets stuck and fires 1500 times on one product (by accident).
3. Even though this is clearly bad code on my part, I have no option now but to remove and reinstall meilisearch as it is just stuck queuing for an indeterminate amount of time.

I understand that the index is backing up due to my own fault, but there should definitely be an escape hatch in case something messes up.

at the very least, if I could just stop all tasks, and just re-add the index, this would be help with development time

[curquiza]

Looks like some users want to clear the queue due to indexation time issue: meilisearch/meilisearch#2186

[elite6adger]

+100 upvote please, since we are looking at Meilisearch and not being able to clear/clean/cancel a task queue is a real time sinker.

Dealing with many clients with different data requirements and data transformation needs pre-search indexing, it is not uncommon after the initial setup to do some testing and have to load a real sample (75K documents out of 800K documents per index on live) to test well. Walking away and having to come back and notice things are stalled or more commonly there was a logic mistake (property mapping via scout, etc.) makes it painful to destroy, rebuild and restart.

[Ovyerus]

Would love to have this. Accidentally implemented Meilisearch updates in a weird way which didn't account for our actual traffic, instead of batching them (totally on us), which has left us with ~96k tasks that is taking Meili quite a while to work through. Would be super nice to just be able to cancel all of these and re-update all of them manually in the same operation.

[MarinPostma]

You could take a look at auto-batching, this may solve your problem: meilisearch/meilisearch#2070

[FutureFutureTo]

thanks! will try this out!

[irevoire]

I would like a feature where, when you push a new task to delete all documents or delete an index meilisearch automatically skip all the others tasks up to this point. (if it's a delete all documents task, obviously, we'll need to do some tricks. Like first delete all documents, and then apply all settings update up to this point or something like that)

That would make it even easier to clean your index when you make a mistake without having to understand complex routes 👍

[gmourier]

I like the idea! Would it mark the previous tasks as canceled, for example, to keep a history of what happened?

[gmourier]

Thinking about it, I think it deserves its discussion; it's more of an "auto-scheduling" subject.

[bidoubiwa]

Is this in discussion for the upcoming iteration?

[gmourier]

@irevoire is it related to the thinking you had about batching tasks in a smarter way with the new task scheduler? I'm not 100% sure.

[irevoire]

Yeah it's not as smart as what I proposed, but if you have only document addition and settings before your document clear / indexDeletion it'll work

[ppamorim]

@curquiza Is there some work done on that? I could have a look.

[gmourier]

Hi @ppamorim 👋

Several solutions are emerging.

(We are not talking about deleting old succeeded or failed tasks but about solving the problem of people being blocked by having too many enqueued tasks)

• Introduce a dedicated endpoint that allows canceling tasks. At first, this endpoint would be used to cancel enqueued tasks; it should be considered extensible to accommodate the canceling of the task being processed.
• Implement a scheduling mechanism as mentioned here. The scheduler could cancel?/skip? all the tasks concerning an index placed before in the queue when it encounters a deletion for this index.

I think that the two solutions are not exclusive. A user might not want to delete an index to cancel enqueued tasks.

What do you think? Do you have another idea?

[mpwsh]

What would be the best way to work around this limitation?
We have a big queue that was generated due to bad timing on batching and created multiple tasks with a small amount of documents.
This was already fixed on our end and batches look better now, but we have around 5000 tasks enqueued that are taking around 1 hour per task to process.
Is there a way to keep the indexed documents and remove pending ones from the task queue safely?

[gmourier]

Hello @mpwsh!

Unfortunately, there is no effective workaround today without removing the data.ms. We are working on this and it should be out for v0.29 if all goes well! At the moment we are thinking about providing an endpoint to cancel tasks (one or more precisely or all the ones that are enqueued).

It might be more complicated to cancel a processing task but we are also exploring the subject.

Thanks!

[gmourier]

Hey 👋

Design Iteration 1

Small update: We are working on releasing it for v0.30.

Here is what we have imagined so far:

Introduce a POST /tasks/cancel endpoint where parameters could be used as a filter to match tasks to cancel

Usage examples

• POST /tasks/cancel?indexUid=products&type=documentAdditionOrUpdate
  • Send a cancel task for all tasks (processing one and enqueued) for the products index that are documentAdditionOrUpdate
• POST /tasks/cancel?status=processing
  • Send a cancel task for the processing one

Filters make it possible to match a group of tasks to cancel very extendedly.

For example, it is possible to cancel tasks for multiple indexes or types simultaneously!

POST /tasks/cancel?indexUid=products,comments&type=documentAdditionOrUpdate,settingsUpdate&status=enqueued

We also plan to indicate on a canceled task which task uid canceled it. Allowing to know more precisely why a task has been canceled if needed in the history.

Design questions

We would like to have your opinion on some design questions that remain.

Accepts uncancellable tasks

Finished tasks, those with the status succeeded, failed, and canceled cannot be canceled.

If you send POST /tasks/cancel?status=enqueued,succeeded what would you like to happen?

• Would you prefer a synchronous error telling you that the payload is invalid?
• Would you prefer Meilisearch to accept the payload and indicate in the details of the cancellation task the number of matched tasks and the actual number of tasks that have been canceled after that cancellation task has been processed?

Accuracy

In addition to the ability to select tasks to cancel via filters. We wonder about the usefulness of being able to send precise uids of tasks to cancel.

We could consider adding a taskUids parameter in the method body to accept an array of tasks to cancel.

Is this something that you think is critical?

Thanks!

[davelarkan]

Exciting news!

I think the ability to filter for tasks to be cancelled is really flexible and would work for most use cases.

The UX we were planning for Meilisearch Cloud would have allowed users to:

1. see a list of all their tasks
2. select one or more
3. click a button to cancel the selected tasks

While this behaviour is not essential for a first iteration, this wouldn't be possible without the ability to pass in a list of taskUids.

I'm also thinking of the use-case where a user adds several documentAdditionOrUpdate tasks and realises one was added in error (it contains the wrong documents for example). Without the ability to pass a taskUid(s) it seems like they would have to cancel all the enqueued tasks in the index and then send the correct ones again.

[gmourier]

Thanks, @davelarkan; that is super helpful; you just confirmed the need for that idea. @Kerollmops, @irevoire

I'm also thinking of the use-case where a user adds several documentAdditionOrUpdate tasks and realises one was added in error (it contains the wrong documents for example). Without the ability to pass a taskUid(s) it seems like they would have to cancel all the enqueued tasks in the index and then send the correct ones again.

We could also propose to filter with dates but I think it can quickly become annoying if we have to go down to the microsecond level to select a precise task.

[gmourier]

Quick update: Following @davelarkan's feedback, we did a second design iteration.

Signal to: @Kerollmops, @irevoire

Design Iteration 2

Specifying tasks to cancel by their uids

We would like to give the possibility to specify the tasks to be canceled by their uids using a uid url parameter.

• This parameter serves as a filter and can be used with other filters such as indexUid, type, and status to select tasks.
  • If uid is specified with the others parameters, the behavior will be to cancel only cancellable tasks that match said filters.
  • Adding the uid url parameter also allows it to be added on GET /tasks (it could be useful only to retrieve a specific set of tasks for further details)

We have considered only accepting either (indexUid, task and type) XOR (uid), but we feel that it adds complexity to return an error in the case both methods are used at the same time, forcing users to choose a way of doing it by digging the documentation seems to be a bad choice.

With the proposed solution, users can combine everything for more granular precision or choose one of the two methods they want.

Could it be problematic to list a lot of task uid in the uid parameter?

In a way, yes, because the URLs can be limited in size.

However, we can hardly imagine someone listing hundreds of IDs to cancel tasks when for this purpose, using the other filters allows to do it quickly and succinctly.

Why not specify the uids in the POST body?

It feels clumsy to have parameters in the url and body simultaneously. For consistency purposes, we should have chosen either one or the other, and we want to keep the ability to express filters from the URL as in the GET method.

What happens if I specify a task uid that does not exist when canceling a task?

This task will not be cancellable; It’s impossible to cancel tasks above or equal to the task cancellation request time. No error will be thrown.

Task Details

We imagine being able to give historical information to know what happened in more detail if needed.

Note: The new names introduced are not definitive.

taskCancelation type details

Note: It is possible to track the progress of a cancellation task. These tasks are prioritized so that the execution time will be very short. This means it is possible to cancel a cancellation task in a very small window of time.

[
  {
    "uid": 98,
    "indexUid": null, //a task cancellation can match multiple indexes; It is ALWAYS considered a global task
    "type": "taskCancelation",
    "status": "succeeded",
    "details": {
        "matchedTasks": 3 //Number of tasks matched by the URL parameters
        "canceledTasks": 1 //Number of tasks that have been canceled
        "originalQuery": "?uid=1,2,3&indexUid=movies,products", //An extract of the URL parameters to know what was asked when the task cancelation was emited
    },
    "enqueuedAt": "2022-08-01T23:23:00"
    "startedAt": "2022-08-01T23:23:00"
    "duration": "PT1S"
    "finishedAt": "2022-08-01T23:24:00"
  }
]

Canceled task status details

It is also possible to indicate which task initiated the cancellation of a task using the canceledBy field.

Note: Its location is not definitive; we are also considering whether to introduce it within the details object. It could also become a filter in the url to find in one go all the tasks that have been canceled by a cancellation task.

[
  {
    "taskUid": 20,
    "indexUid": "products",
    "type": "documentAdditionOrUpdate",
    "status": "canceled",
    "canceledBy": 98, //Relate to the taskCancellation
    "details": {
         "receivedDocuments": 1000000
         "indexedDocuments": 0
    },
    "enqueuedAt": "2022-08-01T22:00:00"
    "startedAt": null //could be null if never dequeued,
    "duration": null
    "finishedAt": "2022-08-01T23:24:00"
  },
]

API Key rights management

What happens if the API Key that emits the requests do not have rights on indexes containing tasks that can be canceled?

If I try to cancel tasks using ?indexUid=movies while using an API Key that do not contains movies as an accessible index. e.g indexes: ["product"].

The tasks belonging to movies are not canceled and not displayed in the matchedTask field in the task cancelation details; these are considered hidden.

What right does the API Key needs to have to emit cancelation tasks?

A new right tasks.cancel will be introduced. It will be available by default on the Default Admin API Key.

Note: We recommend viewing and canceling tasks with an API Key that has access to all indexes and has tasks.get and tasks.cancel rights. Without it, it might be possible to cancel tasks without being able to see their details, which makes little sense.

Answering previous design questions

In the first iteration, we asked:

"Finished tasks, those with the status succeeded, failed, and canceled cannot be canceled.

If you send POST /tasks/cancel?status=enqueued,succeeded what would you like to happen?

• (A) Would you prefer a synchronous error telling you that the payload is invalid?
• (B) Would you prefer Meilisearch to accept the payload and indicate in the details of the cancellation task the number of matched tasks and the actual number of tasks that have been canceled after that cancellation task has been processed?"

Option B has been decided for now.

---

Any feedback is appreciated; we can make a third iteration if needed!

[bidoubiwa]

B is better for me! More consistent with tasks failures

[curquiza]

With B, does it mean the task cancellation is also a task?

[gmourier]

More consistent with tasks failures

Adding a precision: There will be no failure; in the worst-case scenario, 0 tasks will be canceled because of the given payload matching 0 cancelable tasks.

With B, does it mean the task cancellation is also a task?

With A or B, there will be a new task type being taskCancelation (Naming will be debated in a last iteration 🤓)

[curquiza]

Ok so, what do you mean by "the payload is invalid"?

[gmourier]

(A) involve an invalid payload because this solution accepts only using (indexUid, type and status) XOR (uid); Using both throws an error.

With (B) the payload can't be invalid because (indexUid, type, status and uid) can all be used together to match tasks to cancel.

[gmourier]

During our second design iteration focused on delivering a way for tasks to be flushed (deleted from the disk), we imagined introducing filters on the date fields that could also be used to select tasks for cancellation. ✨

[gmourier]

Hey everyone 👋

v0.30 introduced the possibility to cancel any enqueued or processing tasks. 🪄

📚 More information here.

[FutureFutureTo]

Thank you!

…

On Tue, Dec 6, 2022 at 5:25 AM Guillaume Mourier ***@***.***> wrote: Hey everyone 👋 v0.30 <https://github.com/meilisearch/meilisearch/releases/tag/v0.30.0> introduced the possibility to cancel any enqueued or processing tasks. 📚 More information here <https://docs.meilisearch.com/reference/api/tasks.html#cancel-tasks>. — Reply to this email directly, view it on GitHub <#391 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADDKXZXIHTIELHP4WOQRNODWL4IBPANCNFSM5N5XCJKQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[gmourier]

Hello everyone 👋

I'm locking that discussion.

Please open a new discussion if you have feedback to share or if it relates to a new feature request.

Thank you 🙇‍♂️