Skip to content

Commit

Permalink
Add migration phases pages (#8828)
Browse files Browse the repository at this point in the history
* Add first three migration phases pages

Signed-off-by: Archer <naarcha@amazon.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add backfill page.

Signed-off-by: Archer <naarcha@amazon.com>

* Update backfill.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add replayer page.

Signed-off-by: Archer <naarcha@amazon.com>

* Fix grammar.

Signed-off-by: Archer <naarcha@amazon.com>

* Add final migration phases page.

Signed-off-by: Archer <naarcha@amazon.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _migrations/migration-phases/verifying-tools-for-migration.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _migrations/migration-phases/backfill.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add migration phase links

Signed-off-by: Archer <naarcha@amazon.com>

* Edit migration console section

Signed-off-by: Archer <naarcha@amazon.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update migrating-metadata.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Apply suggestions from code review

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Rename traffic replacer.

Signed-off-by: Archer <naarcha@amazon.com>

* Update _migrations/migration-phases/verifying-tools-for-migration.md

Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add sentence about live traffic capture

Signed-off-by: Archer <naarcha@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _migrations/migration-phases/backfill.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _migrations/migration-phases/verifying-tools-for-migration.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Update _migrations/migration-phases/verifying-tools-for-migration.md

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

* Add editorial review.

Signed-off-by: Archer <naarcha@amazon.com>

* Additional editorial comments.

Signed-off-by: Archer <naarcha@amazon.com>

* Editorial for infra and traffic

Signed-off-by: Archer <naarcha@amazon.com>

* Editorial comments for using traffic replayer.

Signed-off-by: Archer <naarcha@amazon.com>

* Add final editorial comments.

Signed-off-by: Archer <naarcha@amazon.com>

* Apply suggestions from code review

Co-authored-by: Nathan Bower <nbower@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>

---------

Signed-off-by: Archer <naarcha@amazon.com>
Signed-off-by: Naarcha-AWS <97990722+Naarcha-AWS@users.noreply.github.com>
Co-authored-by: Nathan Bower <nbower@amazon.com>
  • Loading branch information
Naarcha-AWS and natebower authored Dec 3, 2024
1 parent e31a7e9 commit 1fcf278
Show file tree
Hide file tree
Showing 32 changed files with 923 additions and 957 deletions.
7 changes: 3 additions & 4 deletions _migrations/getting-started-data-migration.md
Original file line number Diff line number Diff line change
Expand Up @@ -181,7 +181,7 @@ To deploy Migration Assistant, use the following steps:
These commands deploy the following stacks:

* Migration Assistant network stack
* Reindex From Snapshot stack
* `Reindex-from-snapshot` stack
* Migration console stack

---
Expand Down Expand Up @@ -253,7 +253,7 @@ Run the following command to migrate metadata:
console metadata migrate [...]
```

For more information, see [Metadata migration].
For more information, see [Migrating metadata]({{site.url}}{{site.baseurl}}/migrations/migration-phases/migrating-metadata/).

---

Expand Down Expand Up @@ -285,7 +285,7 @@ You can now use RFS to migrate documents from your original cluster:
console backfill stop
```

For more information, see [Backfill execution].
For more information, see [Backfill]({{site.url}}{{site.baseurl}}/migrations/migration-phases/backfill/).

---

Expand Down Expand Up @@ -328,4 +328,3 @@ fields @message

If any failed documents are identified, you can index the failed documents directly as opposed to using RFS.

For more information, see [Backfill migration].
29 changes: 11 additions & 18 deletions _migrations/migration-console/accessing-the-migration-console.md
Original file line number Diff line number Diff line change
@@ -1,12 +1,15 @@
---
layout: default
title: Accessing the migration console
nav_order: 35
parent: Migration console
---

# Accessing the migration console

The Bootstrap box deployed through Migration Assistant contains a script that simplifies access to the migration console through that instance.

The Migrations Assistant deployment includes an ECS task that hosts tools to run different phases of the migration and check the progress or results of the migration.

## SSH into the Migration Console
Following the AWS Solutions deployment, the bootstrap box contains a script that simplifies access to the migration console through that instance.

To access the Migration Console, use the following commands:
To access the migration console, use the following commands:

```shell
export STAGE=dev
Expand All @@ -16,13 +19,7 @@ export AWS_REGION=us-west-2

When opening the console a message will appear above the command prompt, `Welcome to the Migration Assistant Console`.

<details>

<summary>
<b>SSH from any machine into Migration Console</b>
</summary>

On a machine with the [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) ↗ and the [AWS Session Manager Plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html) ↗, you can directly connect to the migration console. Ensure you've run `aws configure` with credentials that have access to the environment.
On a machine with the [AWS Command Line Interface (AWS CLI)](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html) and the [AWS Session Manager plugin](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html), you can directly connect to the migration console. Ensure that you've run `aws configure` with credentials that have access to the environment.

Use the following commands:

Expand All @@ -32,10 +29,6 @@ export SERVICE_NAME=migration-console
export TASK_ARN=$(aws ecs list-tasks --cluster migration-${STAGE}-ecs-cluster --family "migration-${STAGE}-${SERVICE_NAME}" | jq --raw-output '.taskArns[0]')
aws ecs execute-command --cluster "migration-${STAGE}-ecs-cluster" --task "${TASK_ARN}" --container "${SERVICE_NAME}" --interactive --command "/bin/bash"
```
</details>

## Troubleshooting

### Deployment Stage

Typically, `STAGE` is `dev`, but this may vary based on what the user specified during deployment.
Typically, `STAGE` is equivalent to a standard `dev` environment, but this may vary based on what the user specified during deployment.
7 changes: 6 additions & 1 deletion _migrations/migration-console/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,9 @@ layout: default
title: Migration console
nav_order: 30
has_children: true
---
---

The Migrations Assistant deployment includes an Amazon Elastic Container Service (Amazon ECS) task that hosts tools that run different phases of the migration and check the progress or results of the migration. This ECS task is called the **migration console**. The migration console is a command line interface used to interact with the deployed components of the solution.

This section provides information about how to access the migration console and what commands are supported.

Original file line number Diff line number Diff line change
@@ -1,83 +1,106 @@
---
layout: default
title: Command reference
nav_order: 35
parent: Migration console
---


# Migration console command reference

The Migration Assistant Console is a command line interface to interact with the deployed components of the solution.
Migration console commands follow this syntax: `console [component] [action]`. The components include `clusters`, `backfill`, `snapshot`, `metadata`, and `replay`. The console is configured with a registry of the deployed services and the source and target cluster, generated from the `cdk.context.json` values.

The commands are in the form of `console [component] [action]`. The components include `clusters`, `backfill` (e.g the Reindex from Snapshot service), `snapshot`, `metadata`, `replay`, etc. The console is configured with a registry of the deployed services and the source and target cluster, generated from the `cdk.context.json` values.

## Commonly Used Commands
## Commonly used commands

The exact commands used will depend heavily on use-case and goals, but the following are a series of common commands with a quick description of what they do.

### Check connection

Reports whether both the source and target clusters can be reached and provides their versions.

```sh
console clusters connection-check
```
Reports whether both the source and target clusters can be reached and their versions.

### Run `cat-indices`

Runs the `cat-indices` API on the cluster.

```sh
console clusters cat-indices
```
Runs the `_cat/indices` command on each cluster and prints the results.

***
### Create a snapshot

Creates a snapshot of the source cluster and stores it in a preconfigured Amazon Simple Storage Service (Amazon S3) bucket.

```sh
console snapshot create
```
Initiates creating a snapshot on the source cluster, into a pre-configured S3 bucket.

## Check snapshot status

Runs a detailed check on the snapshot creation status, including estimated completion time:

```sh
console snapshot status --deep-check
```
Runs a detailed check on the status of the snapshot creation, including estimated completion time.

***
## Evaluate metadata

Performs a dry run of metadata migration, showing which indexes, templates, and other objects will be migrated to the target cluster.

```sh
console metadata evaluate
```
Perform a dry run of metadata migration, showing which indices, templates, and other objects will be migrated to the target cluster.

## Migrate metadata

Migrates the metadata from the source cluster to the target cluster.

```sh
console metadata migrate
```
Perform an actual metadata migration.

***
## Start a backfill

```sh
console backfill start
```
If the Reindex From Snapshot service is enabled, start an instance of the service to begin moving documents to the target cluster.
If `Reindex-From-Snapshot` (RFS) is enabled, this command starts an instance of the service to begin moving documents to the target cluster:

There are similar `scale UNITS` and `stop` commands to change the number of active instances for RFS.


```sh
console backfill status --deep-check
console backfill start
```
See the current status of the backfill migration, with the number of instances operating and the progress of the shards.

***
## Check backfill status

Gets the current status of the backfill migration, including the number of operating instances and the progress of the shards.


## Start Traffic Replayer

If Traffic Replayer is enabled, this command starts an instance of Traffic Replayer to begin replaying traffic against the target cluster.
The `stop` command stops all active instances.

```sh
console replay start
```
If the Traffic Replayer service is enabled, start an instance of the service to begin replaying traffic against the target cluster.
The `stop` command stops all active instances.

***
## Read logs

Reads any logs that exist when running Traffic Replayer. Use tab completion on the path to fill in the available `NODE_IDs` and, if applicable, log file names. The tuple logs roll over at a certain size threshold, so there may be many files named with timestamps. The `jq` command pretty-prints each line of the tuple output before writing it to file.

```sh
console tuples show --in /shared-logs-output/traffic-replayer-default/[NODE_ID]/tuples/console.log | jq > readable_tuples.json
```
Use tab completion on the path to fill in the available node ids and, if applicable, log file names. The tuples logs roll over at a certain size threshold, so there may be many files named with timestamps. The `jq` command pretty-prints each line of the tuple output before writing it to file.


## Command Reference
All commands and options can be explored within the tool itself by using the `--help` option, either for the entire `console` application or for individual components (e.g. `console backfill --help`). The console also has command autocomplete set up to assist with usage.
## Help command

```
All commands and options can be explored within the tool itself by using the `--help` option, either for the entire `console` application or for individual components (for example, `console backfill --help`). For example:

```bash
$ console --help
Usage: console [OPTIONS] COMMAND [ARGS]...

Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
---
layout: default
title: Assessing your cluster for migration
nav_order: 60
has_children: true
parent: Migration phases
---

# Assessing your cluster for migration

The goal of Migration Assistant is to streamline the process of migrating from one location or version of Elasticsearch/OpenSearch to another. However, completing a migration sometimes requires resolving client compatibility issues before they can communicate directly with the target cluster.

## Understanding breaking changes

Before performing any upgrade or migration, you should review any breaking changes documentation. Even if the cluster is migrated, there may be changes required in order for clients to connect to the new cluster.

## Upgrade and breaking changes guides

For migration paths between Elasticsearch 6.8 and OpenSearch 2.x, you should be familiar with the following documentation, depending on your specific use case:

* [Upgrading Amazon OpenSearch Service domains](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/version-migration.html).

* [Amazon OpenSearch Service rename - Summary of changes](https://docs.aws.amazon.com/opensearch-service/latest/developerguide/rename.html).

* [OpenSearch breaking changes](https://opensearch.org/docs/latest/breaking-changes/).

The next step is to set up a proper test bed to verify that your applications will work as expected on the target version.

## Impact of data transformations

Any time you apply a transformation to your data, such as changing index names, modifying field names or field mappings, or splitting indexes with type mappings, these changes may need to be reflected in your client configurations. For example, if your clients are reliant on specific index or field names, you must ensure that their queries are updated accordingly.



We recommend running production-like queries against the target cluster before switching to actual production traffic. This helps verify that the client can:

- Communicate with the target cluster.
- Locate the necessary indexes and fields.
- Retrieve the expected results.

For complex migrations involving multiple transformations or breaking changes, we highly recommend performing a trial migration with representative, non-production data (for example, in a staging environment) to fully test client compatibility with the target cluster.



7 changes: 0 additions & 7 deletions _migrations/migration-phases/assessment/index.md

This file was deleted.

41 changes: 0 additions & 41 deletions _migrations/migration-phases/assessment/required-client-changes.md

This file was deleted.

This file was deleted.

Loading

0 comments on commit 1fcf278

Please sign in to comment.