HDDS-11511. Introduce metrics in deletion services of OM

[Tejaskriya]

What changes were proposed in this pull request?

The following metrics are added in OM's deletingServices:
KeyDeleting
No of keys processed for delete since last restart
No of keys sent to scm for delete and response(success/fail) since last restart

DirDeleting
No of dirs/empty dirs processed for delete since last restart
No of subfiles processed for delete since last restart

Dirpurge
No of dirs purged since last restart
No of subfiles purged since last restart

Keypurge
No of keys purged since last restart

What is the link to the Apache JIRA

https://issues.apache.org/jira/browse/HDDS-11511

How was this patch tested?

Tested manually in docker at the jmx endpoint http://localhost:9874/jmx?qry=Hadoop:service=OzoneManager,name=DeletingServiceMetrics :

{
  "beans" : [ {
    "name" : "Hadoop:service=OzoneManager,name=DeletingServiceMetrics",
    "modelerType" : "DeletingServiceMetrics",
    "tag.Context" : "ozone",
    "tag.Hostname" : "7c4d4db221d4",
    "NumDirsPurged" : 10,
    "NumDirsSentForPurge" : 10,
    "NumKeysProcessed" : 0,
    "NumKeysPurged" : 0,
    "NumKeysSentForPurge" : 0,
    "NumSubDirsMovedToDeletedDirTable" : 9,
    "NumSubDirsSentForPurge" : 0,
    "NumSubFilesMovedToDeletedTable" : 3,
    "NumSubFilesSentForPurge" : 3
  } ]
}

[Tejaskriya]

@ashishkumar50 thank you for the reviews, I have addressed the comments, could you please review the update?

[Tejaskriya]

@ashishkumar50 I have addressed all the comments, thank you for the review.
@errose28 could you please review the patch?

[sumitagrawl]

Plz check for review comments

[sumitagrawl]

we do not need capture runCount and startTime

[errose28]

I don't see anything wrong with capturing run count. It has been useful in the past to detect when the service has completed a run after a restart, and can be divided over past time intervals to get average time taken per run.

[sumitagrawl]

@errose28 I am not getting what we are going to achieve with runCount as metrics. If there are 10 threads running, it will be increase by 10 times as default.
We can use duration as metrics for performance, but not the startTime. This will give better metrics. So startTime may not be useful, this can be easily identified as Timer task.

For task running, based on delete count increase, and other data can show the progress. So Above metric is not useful.

[Tejaskriya]

@sumitagrawl @ashishkumar50 @errose28 I have updated the PR to have only restart metrics and removed the iteration specific ones.
If we determine that iteration specific metrics are needed uniformly across all the services, I can raise a different PR for adding it in all roles OM, SCM and DN.
The aim of this PR is focused to add the missing metrics in OM only. Could you please review the updated PR?

[sumitagrawl]

validate if {deletedDirsCount, movedDirsCount, movedFilesCount} as variable, PurgeDir metrics {NumSubDirectoriesMoved, NumSubFilesMoved, NumDirPurged} and Metrics here {dirDel, dirMove, fileMove} are same value, else this will create confusion between metrics, and also with logs.

[Tejaskriya]

Thanks for the review! I have verified the numbers using integration tests and updated the PR with some checks in the tests.

[Tejaskriya]

@sumitagrawl @ashishkumar50 When a delete is performed with skipTrash, the logs and metrics show the exact number of deletes as the user perceives. But in the case we don't skipTrash, due to a checkpoint directory created which is also deleted, the count of deleted directories is +1 -- which is reflected in both the pre-existing logs and metrics added in this PR.

All comments have been addressed. Can you please review the updated PR?

[sumitagrawl]

LGTM