-
Notifications
You must be signed in to change notification settings - Fork 507
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Browse files
Browse the repository at this point in the history
* adding pattern_capture token filter docs #8240 Signed-off-by: Anton Rubin <anton.rubin@eliatra.com> * updating parameter table Signed-off-by: Anton Rubin <anton.rubin@eliatra.com> * Doc review Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> * Apply suggestions from code review Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> --------- Signed-off-by: Anton Rubin <anton.rubin@eliatra.com> Signed-off-by: Fanit Kolchina <kolchfa@amazon.com> Signed-off-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Fanit Kolchina <kolchfa@amazon.com> Co-authored-by: kolchfa-aws <105444904+kolchfa-aws@users.noreply.github.com> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit b1073cf) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
- Loading branch information
4 people
committed
Nov 25, 2024
1 parent
0e5b6aa
commit 6a86d94
Showing
2 changed files
with
98 additions
and
1 deletion.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| --- | ||
| layout: default | ||
| title: Pattern capture | ||
| parent: Token filters | ||
| nav_order: 310 | ||
| --- | ||
|
|
||
| # Pattern capture token filter | ||
|
|
||
| The `pattern_capture` token filter is a powerful filter that uses regular expressions to capture and extract parts of text according to specific patterns. This filter can be useful when you want to extract particular parts of tokens, such as email domains, hashtags, or numbers, and reuse them for further analysis or indexing. | ||
|
|
||
| ## Parameters | ||
|
|
||
| The `pattern_capture` token filter can be configured with the following parameters. | ||
|
|
||
| Parameter | Required/Optional | Data type | Description | ||
| :--- | :--- | :--- | :--- | ||
| `patterns` | Required | Array of strings | An array of regular expressions used to capture parts of text. | ||
| `preserve_original` | Required | Boolean| Whether to keep the original token in the output. Default is `true`. | ||
|
|
||
|
|
||
| ## Example | ||
|
|
||
| The following example request creates a new index named `email_index` and configures an analyzer with a `pattern_capture` filter to extract the local part and domain name from an email address: | ||
|
|
||
| ```json | ||
| PUT /email_index | ||
| { | ||
| "settings": { | ||
| "analysis": { | ||
| "filter": { | ||
| "email_pattern_capture": { | ||
| "type": "pattern_capture", | ||
| "preserve_original": true, | ||
| "patterns": [ | ||
| "^([^@]+)", | ||
| "@(.+)$" | ||
| ] | ||
| } | ||
| }, | ||
| "analyzer": { | ||
| "email_analyzer": { | ||
| "tokenizer": "uax_url_email", | ||
| "filter": [ | ||
| "email_pattern_capture", | ||
| "lowercase" | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| ## Generated tokens | ||
|
|
||
| Use the following request to examine the tokens generated using the analyzer: | ||
|
|
||
| ```json | ||
| POST /email_index/_analyze | ||
| { | ||
| "text": "john.doe@example.com", | ||
| "analyzer": "email_analyzer" | ||
| } | ||
| ``` | ||
| {% include copy-curl.html %} | ||
|
|
||
| The response contains the generated tokens: | ||
|
|
||
| ```json | ||
| { | ||
| "tokens": [ | ||
| { | ||
| "token": "john.doe@example.com", | ||
| "start_offset": 0, | ||
| "end_offset": 20, | ||
| "type": "<EMAIL>", | ||
| "position": 0 | ||
| }, | ||
| { | ||
| "token": "john.doe", | ||
| "start_offset": 0, | ||
| "end_offset": 20, | ||
| "type": "<EMAIL>", | ||
| "position": 0 | ||
| }, | ||
| { | ||
| "token": "example.com", | ||
| "start_offset": 0, | ||
| "end_offset": 20, | ||
| "type": "<EMAIL>", | ||
| "position": 0 | ||
| } | ||
| ] | ||
| } | ||
| ``` |