[instrumentation-http] add hook for custom server and client metrics attributes

[pichlermarc]

Description

Warning

The two issues linked below (#4095, #4096) are REQUIRED to be finished before implementation of this feature can start. Any PR opened to add this hook before these are finished will be rejected.

Feature request: #5051

Goal of this issue is implementing two additional hooks for adding metric attributes for both client and server metrics in the @opentelemetry/instrumentation-http package. Adding this feature was requested multiple times to increase the usefulness of http.request.server.duration and http.request.client.duration metrics. For both of these metrics, it can make sense to attach additional attribute (such as http.route) so that users can split the metric later on

While discussing #5051 we've identified that we currently have a few blockers before we can implement this feature in a safe manner. With the current state of the SDK, misuse of such a hook can cause the end-user's application to crash.

Therefore, the following features/bugfixes need to be implemented before this feature is unblocked:

• [sdk-metrics] Implement experimenal cardinality limit specification #4095
• [sdk-metrics] Do not report metric data for previosuly observed attribute sets that were not observed during the current cycle  #4096

This issue is considered done when

• new hooks have been added to the options of HttpInstrumentation which receive the RequestOptions and returns these attributes to the metric
  • orient yourself on the existing hooks in instrumentation HTTP
  • consider implementing an interface that follows the below example (Attachment 1). This is just a suggestion, apply your own judgement in case something does not exactly work the way described below or if there's other ways that would make it more memory/compute efficient. The interface should consider that there may be additional metrics that will be added to this instrumentation in the future.
  • the added interface should have a JSDoc that states that it is important to keep cardinality for added attributes low, and should contain advice how to accomplish this
  • if duplicate emission of semantic conventions is still possible when implementing this feature, the new hook should only apply when the "stable" semantic conventions are emitted.
• the new hook functionality was tested in-depth
• the new hook functionality was documented in README.md
  • documentation should mention that it is important to keep cardinality of added metric attributes low and should contain advice on how to accomplish this

Additional context

• see [instrumentation-http] custom client/server metric attributes #5051 (and linked issues)

Attachment 1 (possible interface, apply own judgement when implementing)

new HttpInstrumentation({
  incomingRequestMetricAttributeHook: (request) => {
      return {
           // name of the metric to apply the attributes to
           'http.server.request.duration': {
             // it is up to the user to return a reasonable amount of key-value pairs here, this needs to be properly documented.
             'http.route': determineRoute(request.url), // functionality to determine route is up to the user, it's up to the user to ensure that data is not high cardinality
             'my.custom.attribute': 'foo123, // any custom attributes are allowed
           },
           // type leaves place to add more metrics in a non-breaking way later if needed
       };
   },
   // same behavior for outgoing hook
});