Merge pull request #1900 from dfinity/dev-journey-2

Developer Journey Level 2: Space Explorer

docs/tutorials/developer-journey/index.md

@@ -28,12 +28,12 @@ The developer journey is designed to give both new and existing developers a cle
 
 ## Level 2: Space explorer
 
-- 2.1: Storage and persistence.
-- 2.2: Using third-party canisters.
-- 2.3: Advanced canister calls.
-- 2.4: Introduction to Candid.
-- 2.5: Unit testing and integration.
-- 2.6: Motoko: level 2.
+- [2.1: Canister upgrades, storage, and persistence.](level-2/2.1-storage-persistence.md)
+- [2.2: Advanced canister calls.](level-2/2.2-advanced-canister-calls.md)
+- [2.3: Using third-party canisters.](level-2/2.3-third-party-canisters.md)
+- [2.4: Introduction to Candid.](level-2/2.4-intro-candid.md)
+- [2.5: Unit, integration, and end2end testing](level-2/2.5-unit-testing.md)
+- [2.6: Motoko: level 2.](level-2/2.6-motoko-lvl2.md)
 
 ## Level 3: Space engineer 
 

docs/tutorials/developer-journey/level-1/1.1-live-demo.md

@@ -58,10 +58,12 @@ For a review on the project's structure and our `main.mo` source code file, you
 Before we can deploy the canister, we need to start the local execution environment. To do this, run the command:
 
 ```sh
-dfx start --background
+dfx start --clean --background 
 ```
 
-This command starts the local execution environment in the background. Without the `--background` flag, the output of the execution environment will take over the terminal window and you would need to open a second terminal window to continue running commands. 
+This command starts the local execution environment in the background. The `--clean` flag removes any orphan background processes or canister IDs that might cause conflicts in your environment. It is good practice to use this flag when starting `dfx`, since it can help prevent any issues if you forget to use `dfx stop` when moving between projects, or if you have processes running in another terminal that you forgot about. 
+
+Without the `--background` flag, the output of the execution environment will take over the terminal window and you would need to open a second terminal window to continue running commands. 
 
 Then, to deploy this canister to the playground, run the command:
 
@@ -144,4 +146,4 @@ You can play around with this user interface by inputting different text, such a
 
 Now that we've explored a live canister deployed on the Motoko playground, we'll move onto developing our own dapp rather than using the default template files. But first, let's go over the basics of Motoko in Motoko level 1. 
 
-- [1.2: Motoko level 1](1.2-motoko-lvl1.md).
+- [1.2: Motoko level 1](1.2-motoko-lvl1.md).

docs/tutorials/developer-journey/level-1/1.3-first-dapp.md

@@ -14,9 +14,10 @@ Before you start, verify that you have set up your developer environment accordi
 
 ## Creating a new project
 
-First, we need to create a new `dfx` project. Open a terminal window, navigate into your working directory (`developer_journey`), then use the command:
+First, we need to create a new `dfx` project. Open a terminal window, navigate into your working directory (`developer_journey`), then use the commands:
 
 ```
+dfx start --clean --background
 dfx new poll
 ```
 

docs/tutorials/developer-journey/level-1/1.4-using-cycles.md

@@ -54,7 +54,7 @@ dfx identity new DevJourney
 This command will return a seed phrase, which will be required to recover your identity if you ever need to. This seed phrase should be backed up, so that any cycles associated with your identity are not lost.
 :::
 
-Set this identity as the one to be used by `dfx` with the command:
+Set this identity as the one to be used by `dfx` in the current terminal session with the command:
 
 ```sh
 dfx identity use DevJourney
@@ -98,21 +98,17 @@ Once inside the Discord server, navigate into the `#cycles-faucet` channel.
 
 ![Cycles-faucet](../_attachments/cycles-faucet.png)
 
-In this channel, send a message. This message can be something along the lines of:
+In this channel, execute the following slash command:
 
-> Hello, I'd like to request a cycles coupon. Thank you.
-
-After you send this message, a member of the DFINITY team will reach out to you through a Discord direct message. 
+```
+/request
+```
 
 :::caution
-Please ensure that your Discord settings are set to allow direct messages from other users.
+Please ensure that your Discord settings are set to allow direct messages from other users. If you do not have this setting enabled, you will not receive a direct message from the faucet bot.
 :::
 
-In the direct message from the DFINITY team member, there will be a survey. You must complete this survey. 
-
-![Survey](../_attachments/faucet_step_2.png)
-
-Once completed, reply to the direct message to inform the team member that you've completed the survey. Then, they will send you the cycles coupon code.
+Once completed, our team will review your submission. If accepted, the faucet bot will send you a private message with a coupon code.
 
 Head back to the <https://faucet.dfinity.org> webpage. 
 

docs/tutorials/developer-journey/level-1/1.6-managing-canisters.md

@@ -349,6 +349,6 @@ dfx canister --network ic delete --all
 
 ## Next steps
 
-In the next step of our developer journey, we'll explore storage and persistence.
+In the next step of our developer journey, we'll explore upgrading canisters, storage, and persistence.
 
-- 2.1: Storage and persistence.
+- [2.1: Storage and persistence](../level-2/2.1-storage-persistence.md).

docs/tutorials/developer-journey/level-1/index.md

@@ -19,7 +19,7 @@
     - Printing values.
     - Passing text arguments.
 
-- [1.3: Developing your first dapp](1.3-first-dapp.md): 
+- [1.3: Developing your first dapp](1.3-first-dapp.md): Now we're ready to develop our first dapp on the Internet Computer! This module covers: 
     - Creating a new project.
     - Reviewing the project's file structure.
     - Writing the backend canister code.
@@ -37,16 +37,16 @@
     - Adding pre-developed frontend code.
     - Re-deploying the dapp.
 
-- [1.4: Acquiring and using cycles](1.4-using-cycles.md): 
+- [1.4: Acquiring and using cycles](1.4-using-cycles.md): To deploy dapps on the IC mainnet, we'll need to acquire cycles. This module covers: 
     - Overview of cycles.
     - Creating a developer identity.
     - Acquiring cycles using a cycles coupon.
     - Converting ICP tokens to cycles.
 
-- [1.5: Deploying canisters](1.5-deploying-canisters.md): 
+- [1.5: Deploying canisters](1.5-deploying-canisters.md) Once we have acquired some cycles, we can deploy our dapp to the mainnet. This module covers: 
     - Deploying to the mainnet.
 
-- [1.6: Managing canisters](1.6-managing-canisters.md): 
+- [1.6: Managing canisters](1.6-managing-canisters.md): Once our dapp is deployed on the mainnet, we need to learn how we can manage the canister. This module covers: 
     - Obtaining a canister's ID.
     - Obtaining canister information.
     - Adding an identity as a controller of a canister.

docs/tutorials/developer-journey/level-2/2.1-storage-persistence.md

@@ -0,0 +1,191 @@
+# 2.1: Canister upgrades, storage, and persistence 
+
+## Overview
+
+The Internet Computer handles persistent data storage using a feature known as **stable memory**. **Stable memory** is a unique feature of the Internet Computer that defines a data store separate from the canister's regular Wasm memory data store, which is known as **heap storage**. A canister's heap storage is not persistent storage and does not persist across canister upgrades; canister data and state stored in heap storage is removed when a canister is upgraded or reinstalled. For immutable canisters that use less than the heap storage limit of 4GiB, heap storage can be used. For larger canisters, and especially those that intend to be upgraded and changed over time, stable memory is important since it persists across canister upgrades, has a much larger storage capacity than heap storage, and is very beneficial for vertically scaling a dapp. 
+
+To use stable memory requires anticipating and indicating which canister data you want to be retained after a canister upgrade. For some canisters, this data might be all of the canister's data, while for others it may be only certain parts or none. By default, stable memory starts empty, and can hold up to 96GiB as long as the subnet the canister is running on has the available space. If a canister uses more stable memory than 96GiB, the canister will trap and become unrecoverable. 
+
+## Memory types and terms 
+
+There are several terms associated with memory and storage on the IC. To avoid confusion between them, we'll define them here. 
+
+- **Stable memory:** Stable memory refers to the Internet Computer's long-term data storage feature. Stable memory is not language specific, and can be utilized by canisters written in Motoko, Rust, or any other language. Stable memory can hold up to 96GiB if the subnet can accommodate it. 
+
+- **Heap storage:** Heap storage refers to the regular Wasm memory data store for each canister. This storage is temporary and is not persisted across canister upgrades. Data stored in heap storage is removed when the canister is upgraded or reinstalled. Heap storage is limited to 4GiB. 
+
+- **Stable storage:** Stable storage is a Motoko-specific term that refers to the Motoko stable storage feature. Stable storage uses the IC's stable memory to persist data across canister upgrades. 
+
+- **Stable variables:** Stable variables are a Motoko-specific feature that refers to variables defined in Motoko that use the `stable` modifier which indicates that the value of the variable should be persisted across canister upgrades. 
+
+To further understand stable memory and how to use it, let's learn about upgrading canisters. 
+
+## Upgrading canisters
+
+Once a canister has been deployed and is running, there may need to be changes made to the canister's code to fix bugs or introduce new features. To make these changes, the canister must be upgraded. 
+
+Upgrading a canister is a key feature of the IC, since it allows canister smart contracts to persist using Wasm memory that utilizes the IC's stable memory feature. When a canister is upgraded, the existing state of the canister is preserved while there are changes being made to the canister's code. 
+
+### Motoko stable memory workflow
+
+For canisters written in Motoko, stable memory can be utilized through the Motoko stable storage and stable variable features. These features are most notably used in the canister upgrade process. The upgrade process for Motoko canisters is as follows:
+
+- First, the canister must be stopped so it does not accept any new incoming requests. 
+- A `pre_upgrade` hook is executed if one is defined. This hook can be used for functions such as creating a backup of data. Note: Having a `pre_upgrade` hook is not recommended, since the `pre_upgrade` hook is run in the current Wasm. If there are any bugs or errors, the canister will trap.
+- Then, the system discards the canister's heap memory and initializes a new version of the canister's Wasm module. Stable memory is preserved and made available to the new Wasm module.
+- Next, the new canister code is installed using the `--mode upgrade` flag.
+- Then, the canister is started, now running the newly upgraded code. 
+- Any stable variables are re-loaded as part of the `post_upgrade` hook. After the stable variables are re-loaded, the stable memory bytes that stored those variables are overwritten with zeroes to minimize stable memory costs for the canister. 
+
+### Rust stable memory workflow
+
+For canisters written in Rust, stable memory can be utilized through two Rust crates: [ic-stable-memory](https://github.com/seniorjoinu/ic-stable-memory) and [ic-stable-structures](https://github.com/dfinity/stable-structures). These features as most notably used in the canister upgrade process. The upgrade process for Rust canisters is as follows:
+
+- First, the canister must be stopped so it does not accept any new incoming requests. 
+- A `pre_upgrade` hook is executed if one is defined. This hook can be used for functions such as creating a backup of data. 
+- Then, the system discards the canister's heap memory and initializes a new version of the canister's Wasm module. Stable memory is preserved and made available to the new Wasm module.
+- Next, the new canister code is installed using the `--mode upgrade` flag.
+- Then, the canister is started, now running the newly upgraded code. 
+- A `post_upgrade` hook is called on the newly created instance if one is defined it. The `init` function is not executed.
+
+## Stable storage and stable variables 
+
+Motoko supports preserving a canister's state using the IC's stable memory through a Motoko-specific feature known as **stable storage**, which is designed to accommodate changes to both the application data and the Motoko compiler. Stable storage utilizes the IC's **stable memory** feature that we discussed previously. 
+
+:::info
+Upgrading canisters written in Rust and other languages use a different workflow which incorporates serialization of the canister's data. For more information on Rust upgrades, see the documentation [here](../../../developer-docs/backend/rust/7-upgrading.md).
+:::
+
+A **stable variable** is a variable defined within an actor which uses the `stable` keyword as a modifier in the variable's declaration. This indicates that the data stored in the variable should be stored using **stable storage**. If this `stable` keyword is not used, the variable is defined as `flexible` by default, which means it's data does not persist across canister upgrades.
+
+## Interactive example
+Let's take a look at defining and using a stable variable that uses Motoko's stable storage feature. 
+
+### Prerequisites
+
+Before you start, verify that you have set up your developer environment according to the instructions in [0.3: Developer environment setup](../level-0/03-dev-env.md).
+
+### Creating a new project
+
+To get started, create a new project in your working directory. Open a terminal window, navigate into your working directory (`developer_journey`), then use the commands:
+
+```
+dfx start --clean --background
+dfx new counter
+cd counter
+```
+
+Then, open the file `src/counter_backend/main.mo`. Delete all of the content within this file. 
+
+### Defining a stable variable
+
+The following code example defines an actor called `Counter`, which includes a stable variable that preserves the counter's value across upgrades. Paste this code into the `src/counter_backend/main.mo` file:
+
+```motoko
+actor Counter {
+
+  stable var value = 0;
+
+  public func inc() : async Nat {
+    value += 1;
+    return value;
+  };
+}
+```
+
+:::info
+The `stable` or `flexible` variable modifications can only be used on `let` and `var` declarations that are within actor fields. These modifiers cannot be used elsewhere in the program.
+:::
+
+You can take a deeper dive into stable variables in the [documentation here](https://internetcomputer.org/docs/current/motoko/main/upgrades#declaring-stable-variables). 
+
+Save this file. 
+
+### Deploying your counter dapp
+
+To upgrade our canister, first we need to deploy our initial version of the canister. Deploy the canister with the command:
+
+```
+dfx deploy counter_backend
+```
+
+:::info
+For this tutorial, we're using the local replica environment to deploy our canisters. You can deploy yours on the mainnet with the flag `--network ic`. Remember that deploying to the mainnet will cost cycles.
+:::
+
+You can interact with the counter dapp through the Candid UI URL returned in the output of the `dfx deploy` command, such as:
+
+```
+Deployed canisters.
+URLs:
+  Backend canister via Candid interface:
+    counter_backend: http://127.0.0.1:8080/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=bkyz2-fmaaa-aaaaa-qaaaq-cai
+```
+
+Open the URL in a web browser, then interact with the counter by clicking the 'Call' button 3 times. The counter value will now return '3':
+
+![Counter Candid UI 1](../_attachments/counter_ui1.png)
+
+Since this value is being stored in our stable variable, this value of '3' will persist through our canister upgrade. This is a great example of **persistent storage** using **stable variables**. 
+
+### Stable variables in action
+
+Now, let's make some changes to upgrade the canister. First, stop the canister with the command:
+
+```
+dfx canister stop counter_backend
+```
+
+Then, let's alter the code of our canister. To keep things simple, we're going to change the counter increment value from '1' to '3'. Our altered code looks like this:
+
+```motoko
+actor Counter {
+
+  stable var value = 0;
+
+  public func inc() : async Nat {
+    value += 3;
+    return value;
+  };
+}
+```
+
+Save the file. Now, to upgrade the canister, use the command:
+
+```
+dfx canister install counter_backend --mode upgrade
+```
+
+Then, to confirm the canister's code has been upgraded, start and deploy the canister again with the command:
+
+```
+dfx canister start counter_backend
+dfx deploy counter_backend
+```
+
+This time, the output should include information about the canister's upgrade, such as:
+
+```
+Deploying: counter_backend
+All canisters have already been created.
+Building canisters...
+Installing canisters...
+Upgrading code for canister counter_backend, with canister ID bkyz2-fmaaa-aaaaa-qaaaq-cai
+Deployed canisters.
+URLs:
+  Backend canister via Candid interface:
+    counter_backend: http://127.0.0.1:8080/?canisterId=bd3sg-teaaa-aaaaa-qaaba-cai&id=bkyz2-fmaaa-aaaaa-qaaaq-cai
+```
+
+Now, navigate back to the Candid UI URL provided in the output and click the 'Call' button twice. This time, the counter should increment by 3, resulting in the value '9', since our stable variable held our previous counter value of '3' that was saved prior to the upgrade. 
+
+![Counter Candid UI 2](../_attachments/counter_ui2.png)
+
+
+## Next steps
+
+Next, we'll look at advanced canister calls, such as inter-canister calls and canister query methods.
+
+- [2.2: Advanced canister calls](2.2-advanced-canister-calls.md).
+
+