diff --git a/.github/CONTRIBUTING.md b/.github/CONTRIBUTING.md new file mode 100644 index 00000000..b518511a --- /dev/null +++ b/.github/CONTRIBUTING.md @@ -0,0 +1,19 @@ +# Contributing + +Thank you for considering contributing to this product! We welcome any contributions, whether it's bug fixes, new features, or improvements to the existing codebase. + +### Your First Pull Request + +Working on your first Pull Request? You can learn how from this free video series: + +[How to Contribute to an Open Source Project on GitHub](https://egghead.io/courses/how-to-contribute-to-an-open-source-project-on-github) + +To help you get familiar with our contribution process, we have a list of [good first issues](../../issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22) that contain bugs that have a relatively limited scope. This is a great place to get started. + +If you decide to fix an issue, please be sure to check the comment thread in case somebody is already working on a fix. If nobody is working on it at the moment, please leave a comment stating that you intend to work on it so other people don’t accidentally duplicate your effort. + +If somebody claims an issue but doesn’t follow up for more than two weeks, it’s fine to take it over but you should still leave a comment. **Issues won't be assigned to anyone outside the core team**. + +### Contribution Prerequisites + +... 🚧 Work in progress 🚧 ... diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md new file mode 100644 index 00000000..9f270782 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/bug_report.md @@ -0,0 +1,42 @@ +--- +name: Bug report +about: Create a report to help us improve +title: '' +labels: 'Bug' + +--- + +**Describe the bug** +A clear and concise description of what the bug is. + +Transaction ID +Address +Block# +Time stamp + +**To Reproduce** +Steps to reproduce the behavior: +1. Go to '...' +2. Click on '....' +3. Scroll down to '....' +4. See error + +**Expected behavior** +A clear and concise description of what you expected to happen. + +**Screenshots** +If applicable, add screenshots or consol.log to help explain your problem. + +**Desktop (please complete the following information):** + - OS: [e.g. iOS] + - Browser [e.g. chrome, safari] + - Version [e.g. 22] + +**Smartphone (please complete the following information):** + - Device: [e.g. iPhone6] + - OS: [e.g. iOS8.1] + - Browser [e.g. stock browser, safari] + - Version [e.g. 22] + +**Additional context** +Add any other context about the problem here: diff --git a/.github/ISSUE_TEMPLATE/config.yml b/.github/ISSUE_TEMPLATE/config.yml new file mode 100644 index 00000000..9e27ed98 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/config.yml @@ -0,0 +1,4 @@ +contact_links: + - name: Ask the community + url: https://discord.gg/KrqnVg8D + about: Ask and discuss questions with other Stacks community developers. \ No newline at end of file diff --git a/.github/ISSUE_TEMPLATE/feature_request.md b/.github/ISSUE_TEMPLATE/feature_request.md new file mode 100644 index 00000000..1e8733f9 --- /dev/null +++ b/.github/ISSUE_TEMPLATE/feature_request.md @@ -0,0 +1,23 @@ +--- +name: Feature request +about: Suggest an idea for this project +title: '' +labels: 'enhancement' + +--- + +**Please review the existing enhancement issues at the link below before creating a new one to ensure you do not create a duplicate request.** +If you see an [existing enhancement issues](https://github.com/blockstack/explorer/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement) please comment on the issue or upvote the issue with a :thumbsup: + + +**Is your feature request related to a problem? Please describe.** +A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] + +**Describe the solution you'd like** +A clear and concise description of what you want to happen. + +**Describe alternatives you've considered** +A clear and concise description of any alternative solutions or features you've considered. + +**Additional context** +Add any other context or screenshots about the feature request here. diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md new file mode 100644 index 00000000..c4dc4d43 --- /dev/null +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -0,0 +1,19 @@ +### Description + + + +#### Breaking change? + + + +### Example + + + +--- + +### Checklist + +- [ ] All tests pass +- [ ] Tests added in this PR (if applicable) + diff --git a/README.md b/README.md index 34e0101e..47956dd6 100644 --- a/README.md +++ b/README.md @@ -22,6 +22,9 @@ Stacks blockchain and exposes it via JSON REST API endpoints. * [Job Queue](#job-queue) * [Process Smart Contract Job](#process-smart-contract-job) * [Process Token Job](#process-token-job) +* [Bugs and Feature Requests](#bugs-and-feature-requests) +* [Contribute](#contribute) +* [Community](#community) ## Features @@ -218,3 +221,34 @@ to save it into the local database for API endpoints to return. If a `429` (Too Many Requests) status code is returned by a hostname used to fetch metadata, the service will cease all further requests to it until a reasonable amount of time has passed or until the time specified by the host in a `Retry-After` response header. + +## Bugs and feature requests + +If you encounter a bug or have a feature request, we encourage you to follow the steps below: + + 1. **Search for existing issues:** Before submitting a new issue, please search [existing and closed issues](../../issues) to check if a similar problem or feature request has already been reported. + 1. **Open a new issue:** If it hasn't been addressed, please [open a new issue](../../issues/new/choose). Choose the appropriate issue template and provide as much detail as possible, including steps to reproduce the bug or a clear description of the requested feature. + 1. **Evaluation SLA:** Our team reads and evaluates all the issues and pull requests. We are avaliable Monday to Friday and we make a best effort to respond within 7 business days. + +Please **do not** use the issue tracker for personal support requests or to ask for the status of a transaction. You'll find help at the [#support Discord channel](https://discord.gg/SK3DxdsP). + + +## Contribute + +Development of this product happens in the open on GitHub, and we are grateful to the community for contributing bugfixes and improvements. Read below to learn how you can take part in improving the product. + +### Code of Conduct +Please read our [Code of conduct](../../../.github/blob/main/CODE_OF_CONDUCT.md) since we expect project participants to adhere to it. + +### Contributing Guide +Read our [contributing guide](.github/CONTRIBUTING.md) to learn about our development process, how to propose bugfixes and improvements, and how to build and test your changes. + +## Community + +Join our community and stay connected with the latest updates and discussions: + +- [Join our Discord community chat](https://discord.gg/ZQR6cyZC) to engage with other users, ask questions, and participate in discussions. + +- [Visit hiro.so](https://www.hiro.so/) for updates and subcribing to the mailing list. + +- Follow [Hiro on Twitter.](https://twitter.com/hirosystems) diff --git a/client/typescript/api.ts b/client/typescript/api.ts index d979bc90..fe567162 100644 --- a/client/typescript/api.ts +++ b/client/typescript/api.ts @@ -4,7 +4,7 @@ * Token Metadata API * Service that indexes metadata for every SIP-009, SIP-010, and SIP-013 Token in the Stacks blockchain and exposes it via REST API endpoints. * - * OpenAPI spec version: v0.3.0 + * OpenAPI spec version: v0.4.0 * * * NOTE: This file is auto generated by the swagger code generator program. @@ -268,6 +268,12 @@ export interface FtBasicMetadataResponse { * @memberof FtBasicMetadataResponse */ sender_address: string; + /** + * + * @type {string} + * @memberof FtBasicMetadataResponse + */ + contract_principal: string; } /** * @@ -478,14 +484,18 @@ export interface NotFoundErrorResponse { * @export * @interface Order */ -export interface Order { +export enum Order { + asc = 'asc', + desc = 'desc', } /** * * @export * @interface OrderBy */ -export interface OrderBy { +export enum OrderBy { + name = 'name', + symbol = 'symbol', } /** * diff --git a/client/typescript/api_test.spec.ts b/client/typescript/api_test.spec.ts index 2988eb1a..0e29320a 100644 --- a/client/typescript/api_test.spec.ts +++ b/client/typescript/api_test.spec.ts @@ -2,7 +2,7 @@ * Token Metadata API * Service that indexes metadata for every SIP-009, SIP-010, and SIP-013 Token in the Stacks blockchain and exposes it via REST API endpoints. * - * OpenAPI spec version: v0.3.0 + * OpenAPI spec version: v0.4.0 * * * NOTE: This file is auto generated by the swagger code generator program. diff --git a/client/typescript/configuration.ts b/client/typescript/configuration.ts index 65d047e3..9e7a107e 100644 --- a/client/typescript/configuration.ts +++ b/client/typescript/configuration.ts @@ -3,7 +3,7 @@ * Token Metadata API * Service that indexes metadata for every SIP-009, SIP-010, and SIP-013 Token in the Stacks blockchain and exposes it via REST API endpoints. * - * OpenAPI spec version: v0.3.0 + * OpenAPI spec version: v0.4.0 * * * NOTE: This file is auto generated by the swagger code generator program. diff --git a/client/typescript/index.ts b/client/typescript/index.ts index 8456b0c9..505cd860 100644 --- a/client/typescript/index.ts +++ b/client/typescript/index.ts @@ -3,7 +3,7 @@ * Token Metadata API * Service that indexes metadata for every SIP-009, SIP-010, and SIP-013 Token in the Stacks blockchain and exposes it via REST API endpoints. * - * OpenAPI spec version: v0.3.0 + * OpenAPI spec version: v0.4.0 * * * NOTE: This file is auto generated by the swagger code generator program. diff --git a/client/typescript/package-lock.json b/client/typescript/package-lock.json index 5eaabd2b..31f5e962 100644 --- a/client/typescript/package-lock.json +++ b/client/typescript/package-lock.json @@ -1,12 +1,12 @@ { "name": "@hirosystems/token-metadata-api-client", - "version": "1.1.0", + "version": "1.2.0", "lockfileVersion": 2, "requires": true, "packages": { "": { "name": "@hirosystems/token-metadata-api-client", - "version": "1.1.0", + "version": "1.2.0", "license": "GPL-3.0", "dependencies": { "isomorphic-fetch": "^3.0.0" diff --git a/client/typescript/package.json b/client/typescript/package.json index 8ec2340c..6cd4513b 100644 --- a/client/typescript/package.json +++ b/client/typescript/package.json @@ -1,6 +1,6 @@ { "name": "@hirosystems/token-metadata-api-client", - "version": "1.1.0", + "version": "1.2.0", "description": "Client for @hirosystems/token-metadata-api", "author": "Hiro Systems PBC (https://hiro.so)", "keywords": [ diff --git a/docs/feature-guides/metadata-api.md b/docs/feature-guides/metadata-api.md new file mode 100644 index 00000000..04d2dcd3 --- /dev/null +++ b/docs/feature-guides/metadata-api.md @@ -0,0 +1,24 @@ +--- +title: API-Metadata features +--- + +## Features + +Following are the features of Token-Metadata API: + +* Complete [SIP-016](https://github.com/stacksgov/sips/blob/main/sips/sip-016/sip-016-token-metadata.md) metadata ingestion for + * [SIP-009](https://github.com/stacksgov/sips/blob/main/sips/sip-009/sip-009-nft-standard.md) + Non-Fungible Tokens + * [SIP-010](https://github.com/stacksgov/sips/blob/main/sips/sip-010/sip-010-fungible-token-standard.md) + Fungible Tokens + * [SIP-013](https://github.com/stacksgov/sips/blob/main/sips/sip-013/sip-013-semi-fungible-token-standard.md) + Semi-Fungible Tokens +* Automatic metadata refreshes via [SIP-019](https://github.com/stacksgov/sips/pull/72) + notifications +* Metadata localization support +* Metadata fetching via `http:`, `https:`, `data:` URIs. Also supported via customizable gateways: + * IPFS + * Arweave +* Easy to use REST JSON endpoints with ETag caching.= +* Prometheus metrics for job queue status, contract and token counts, API performance, etc +* Image cache/CDN support diff --git a/docs/feature-guides/token-metadata-api.md b/docs/feature-guides/token-metadata-api.md deleted file mode 100644 index ebb97754..00000000 --- a/docs/feature-guides/token-metadata-api.md +++ /dev/null @@ -1,156 +0,0 @@ ---- - -title: Token Metadata API - ---- - -# Token Metadata API - -A microservice that indexes metadata for all Fungible, Non-Fungible, and Semi-Fungible Tokens in the Stacks blockchain and exposes it via JSON REST API endpoints. - -## Service architecture - -This section gives you an overview of external and internal architectural diagrams. - -### External architecture - - -The external architectural diagram shows how the Token metadata API is connected to three different systems, Stacks node, Stacks blockchain API database, and Postgres database. - -![Architecture](../../architecture.png) - -1. Token metadata API interacts with Stacks Blockchain API database( referred to as Local Metadata DB in the diagram) to import all historical smart contracts when booting up and to listen for new contracts that may be deployed. Read-only access is recommended as this service will never need to write anything to this database(DB). -2. A Stacks node to respond to all read-only contract calls required when fetching token metadata (calls to get token count, token metadata URIs, etc.). -3. A local Postgres DB to store all processed metadata info. - -The service needs to fetch external metadata files (JSONs, images) from the internet, so it must have access to external networks. - -### Internal architecture - - -The following is the internal architectural diagram of the Token metadata API. - -![Flowchart](../../flowchart.png) - -#### Blockchain importer - - -The [`BlockchainImporter`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/blockchain-api/blockchain-importer.ts) is a component in the Token metadata API that takes token contracts from the API database. This component is only used on service boot. - -It connects to the Stacks Blockchain API database and scans the entire `smart_contracts` table looking for any contract that conforms to [SIP-009](https://github.com/stacksgov/sips/blob/main/sips/sip-009/sip-009-nft-standard.md), SIP-010 or SIP-013. When it finds a token contract, it creates a [`ProcessSmartContractJob`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/queue/job/process-smart-contract-job.ts) and adds it to the [Job queue](#job-queue), ßso its tokens can be read and processed thereafter. - -This process runs only once. If the Token metadata API is ever restarted, though, this component re-scans the API `smart_contracts` table from the last processed block height. So, it can pick up any newer contracts it might have missed while the service was unavailable. - -#### Smart contract monitor - -The [`BlockchainSmartContractMonitor`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/blockchain-api/blockchain-smart-contract-monitor.ts) component constantly listens to the following Stacks Blockchain API events: - -* **Smart contract log events** - - If a contract `print` event conforms to SIP-019, it finds the affected tokens and marks them for metadata refresh. - -* **Smart contract deployments** - - If the new contract is a token contract, it saves the new token contract and adds the contract to the job queue for token processing. - -This process is kept alive throughout the entire service lifetime. - -#### Job queue - - -The role of the [`JobQueue`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/queue/job-queue.ts) is to perform all the smart contract and token processing in the service. - -It is a priority queue that organizes all necessary work for contract ingestion and token metadata processing. Every job this queue processes corresponds to one row in the `jobs` DB table, which marks its processing status and related objects to be worked on (smart contract or token). - -This object essentially runs an infinite loop that follows these steps: -1. Upon `start()`, it fetches a set number of job rows that are `'pending'` and loads their corresponding `Job` objects into memory for processing, marking those rows now as `'queued'`. -2. It executes each loaded job to completion concurrently. Depending on success or failure, the job row is marked as either `'done'` or `'failed'`. -3. Once all loaded jobs are done (and the queue is now empty), it goes back to step 1. - -There are two environment variables that can help you tune how the queue performs: -* `ENV.JOB_QUEUE_SIZE_LIMIT`: The in-memory size of the queue, i.e., the number of pending jobs that are loaded from the database while they wait for execution (see step 1 above). -* `ENV.JOB_QUEUE_CONCURRENCY_LIMIT`: The number of jobs that will be run simultaneously. - -This queue runs continuously and can handle an unlimited number of jobs. - -##### Process smart contract job - -This job makes a contract call to the Stacks node to determine the total number of tokens declared by the given contract. Once determined, it creates and enqueues all of these tokens for metadata ingestion. - -##### Process token job - -This job fetches the metadata JSON object for a single token and other relevant properties depending on the token type (symbol, decimals, etc.). Once fetched, it parses and ingests this data to save it into the local database for API endpoints to return. - -## Features - -* Complete - [SIP-016](https://github.com/stacksgov/sips/blob/main/sips/sip-016/sip-016-token-metadata.md) - metadata ingestion for - * [SIP-009](https://github.com/stacksgov/sips/blob/main/sips/sip-009/sip-009-nft-standard.md) - Non-Fungible Tokens - * [SIP-010](https://github.com/stacksgov/sips/blob/main/sips/sip-010/sip-010-fungible-token-standard.md) - Fungible Tokens - * [SIP-013](https://github.com/stacksgov/sips/blob/main/sips/sip-013/sip-013-semi-fungible-token-standard.md) - Semi-Fungible Tokens -* Automatic metadata refreshes via [SIP-019](https://github.com/stacksgov/sips/pull/72) - notifications. -* Metadata localization support. -* Metadata fetching via `http:`, `https:`, `data:` URIs. Also supported via customizable gateways: - * IPFS - * Arweave -* Easy to use REST JSON endpoints with ETag caching. -* Prometheus metrics for job queue status, contract and token counts, API performance, etc. -* Image cache/CDN support. - -## API reference - -See the [Token metadata API Reference](https://docs.hiro.so/metadata/) for more information. - -## Quick start - -### System requirements - -The Token metadata API is a microservice with hard dependencies on other Stacks blockchain components. Before you start, you'll need to have access to the following: - -1. A fully synchronized [Stacks node](https://github.com/stacks-network/stacks-blockchain) -1. A fully synchronized instance of the [Stacks Blockchain API](https://github.com/hirosystems/stacks-blockchain-api) running in `default` or `write-only` mode, with its Postgres database exposed for new connections. A read-only DB replica is also acceptable. -1. A local writeable Postgres database for token metadata storage - -### Run service - -This section helps you to initiate the service by following the steps below. - -1. Clone the repository by using the following command: - -`git clone https://github.com/hirosystems/token-metadata-api.git` - -1. Create a `.env` file and specify the appropriate values to configure access to the Stacks API database, the Token metadata API local database, and the Stacks node RPC interface. See [`env.ts`](https://github.com/hirosystems/token-metadata-api/tree/master/src/env.ts) for all available configuration options. - -2. Build the app (NodeJS v18+ is required) - -``` -npm install -npm run build -``` - -1. Start the service - -``` -npm run start -``` - -### Stop service - -When shutting down, you should always prefer to send the `SIGINT` signal instead of `SIGKILL` so the service has time to finish any pending background work and all dependencies are gracefully disconnected. - -### Using image cache service - -The Token metadata API allows you to specify the path to a custom script that can pre-process every image URL detected by the service before it's inserted into the DB. This will enable you to serve CDN image URLs in your metadata responses instead of raw URLs, providing key advantages such as: - -* Improves image load speed -* Increases reliability in case the original image becomes unavailable -* Protects original image hosts from [DDoS attacks](https://wikipedia.org/wiki/Denial-of-service_attack) -* Increases user privacy - -An example IMGIX processor script is included in [`config/image-cache.js`](https://github.com/hirosystems/token-metadata-api/blob/master/config/image-cache.js). -You can customize the script path by altering the `METADATA_IMAGE_CACHE_PROCESSOR` environment variable. diff --git a/docs/getting-started.md b/docs/getting-started.md new file mode 100644 index 00000000..25185a24 --- /dev/null +++ b/docs/getting-started.md @@ -0,0 +1,32 @@ +--- +title: Getting started +--- + +The Token metadata API is a microservice with hard dependencies on other Stacks blockchain components. Before you start, you'll need to have access to the following: + +1. A fully synchronized [Stacks node](https://github.com/stacks-network/stacks-blockchain) +2. A fully synchronized instance of the [Stacks Blockchain API](https://github.com/hirosystems/stacks-blockchain-api) running in `default` or `write-only` mode, with its Postgres database exposed for new connections. A read-only DB replica is also acceptable +3. A local writeable Postgres database for token metadata storage + +## Run service + +This section helps you to initiate the service by following the steps below. + +1. Clone the repository by using the following command: + + `git clone https://github.com/hirosystems/token-metadata-api.git` + +1. Create a `.env` file and specify the appropriate values to configure access to the Stacks API database, the Token metadata API local database, and the Stacks node RPC interface. See [`env.ts`](https://github.com/hirosystems/token-metadata-api/tree/master/src/env.ts) for all available configuration options. + +2. Build the app (NodeJS v18+ is required) + + ``` + npm install + npm run build + ``` + +3. Start the service + + ``` + npm run start + ``` diff --git a/docs/how-to/how-to-stop-service.md b/docs/how-to/how-to-stop-service.md new file mode 100644 index 00000000..8e8e6aa5 --- /dev/null +++ b/docs/how-to/how-to-stop-service.md @@ -0,0 +1,7 @@ +--- +title: How to stop the service +--- + +# How to stop the Metadata service + +When shutting down, you should always prefer to send the `SIGINT` signal instead of `SIGKILL.` Hence, the service has time to finish any pending background work, and all dependencies are gracefully disconnected. \ No newline at end of file diff --git a/docs/how-to/how-to-use-image-cache-service.md b/docs/how-to/how-to-use-image-cache-service.md new file mode 100644 index 00000000..9fbe722c --- /dev/null +++ b/docs/how-to/how-to-use-image-cache-service.md @@ -0,0 +1,15 @@ +--- +title: How to use Image Cache Service +--- + +# How to use Image Cache Service + +The Token metadata API allows you to specify the path to a custom script that can pre-process every image URL detected by the service before it's inserted into the DB. This will enable you to serve CDN image URLs in your metadata responses instead of raw URLs, providing key advantages such as: + +* Improves image load speed +* Increases reliability in case the original image becomes unavailable +* Protects original image hosts from [DDoS attacks](https://wikipedia.org/wiki/Denial-of-service_attack) +* Increases user privacy + +An example IMGIX processor script is included in [`config/image-cache.js`](https://github.com/hirosystems/token-metadata-api/blob/master/config/image-cache.js). +You can customize the script path by altering the `METADATA_IMAGE_CACHE_PROCESSOR` environment variable. \ No newline at end of file diff --git a/docs/overview.md b/docs/overview.md new file mode 100644 index 00000000..409c5192 --- /dev/null +++ b/docs/overview.md @@ -0,0 +1,83 @@ +--- + +title: Overview + +--- + +# API - Token Metadata Overview + +[Open API Specification](https://github.com/hirosystems/token-metadata-api) +[Source code repository](https://github.com/hirosystems/token-metadata-api) + +Token metadata is a microservice that indexes metadata for all Fungible, Non-Fungible, and Semi-Fungible Tokens in the Stacks blockchain and exposes it via JSON REST API endpoints. + +## Service architecture + +This section gives you an overview of external and internal architectural diagrams. + +### External architecture + +The external architectural diagram shows how the Token metadata API is connected to three systems: Stacks node, Stacks blockchain API database, and Postgres database. + +![Architecture](../architecture.png) + +1. Token metadata API interacts with the Stacks Blockchain API database( referred to as Local Metadata DB in the diagram) to import all historical smart contracts when booting up and to listen for new contracts that may be deployed. Read-only access is recommended as this service will never need to write anything to this database(DB). +2. A Stacks node to respond to all read-only contract calls required when fetching token metadata (calls to get token count, token metadata URIs, etc.). +3. A local Postgres DB to store all processed metadata info. + +The service needs to fetch external metadata files (JSONs, images) from the internet, so it must have access to external networks. + +### Internal architecture + +The following is the internal architectural diagram of the Token metadata API. + +![Flowchart](../flowchart.png) + +#### Blockchain importer + +The [`BlockchainImporter`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/blockchain-api/blockchain-importer.ts) is a component in the Token metadata API that takes token contracts from the API database. This component is only used on service boot. + +It connects to the Stacks Blockchain API database and scans the entire `smart_contracts` table looking for any contract that conforms to [SIP-009](https://github.com/stacksgov/sips/blob/main/sips/sip-009/sip-009-nft-standard.md), SIP-010 or SIP-013. When it finds a token contract, it creates a [`ProcessSmartContractJob`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/queue/job/process-smart-contract-job.ts) and adds it to the [Job queue](#job-queue), ßso its tokens can be read and processed thereafter. + +This process runs only once. If the Token metadata API is ever restarted, though, this component re-scans the API `smart_contracts` table from the last processed block height. It can pick up any newer contracts it might have missed while the service was unavailable. + +#### Smart contract monitor + +The [`BlockchainSmartContractMonitor`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/blockchain-api/blockchain-smart-contract-monitor.ts) component constantly listens to the following Stacks Blockchain API events: + +* **Smart contract log events** + + If a contract `print` event conforms to SIP-019, it finds the affected tokens and marks them for metadata refresh. + +* **Smart contract deployments** + + If the new contract is a token contract, it saves the new token contract and adds the contract to the job queue for token processing. + +This process is kept alive throughout the entire service lifetime. + +#### Job queue + +The role of the [`JobQueue`](https://github.com/hirosystems/token-metadata-api/tree/master/src/token-processor/queue/job-queue.ts) is to perform all the smart contract and token processing in the service. + +It is a priority queue that organizes all necessary work for contract ingestion and token metadata processing. Every job this queue processes corresponds to one row in the `jobs` DB table, which marks its processing status and related objects to be worked on (smart contract or token). + +This object essentially runs an infinite loop that follows these steps: + +1. Upon `start(),` it fetches a set number of job rows that are `'pending'` and loads their corresponding `Job` objects into memory for processing, marking those rows now as `'queued'`. +2. It executes each loaded job to completion concurrently. The job row is marked as either `'done'` or `'failed'` depending on success or failure. +3. Once all loaded jobs are done (and the queue is now empty), it goes back to step 1. + +There are two environment variables that can help you tune how the queue performs: + +* `ENV.JOB_QUEUE_SIZE_LIMIT`: The in-memory size of the queue, i.e., the number of pending jobs that are loaded from the database while they wait for execution (see step 1 above). +* `ENV.JOB_QUEUE_CONCURRENCY_LIMIT`: The number of jobs that will be run simultaneously. + +This queue runs continuously and can handle an unlimited number of jobs. + +##### Process smart contract job + +This job makes a contract call to the Stacks node to determine the total number of tokens declared by the given contract. Once determined, it creates and enqueues all these tokens for metadata ingestion. + +##### Process token job + +This job fetches the metadata JSON object for a single token and other relevant properties depending on the token type (symbol, decimals, etc.). Once fetched, it parses and ingests this data to save it into the local database for API endpoints to return. diff --git a/src/api/routes/ft.ts b/src/api/routes/ft.ts index 136f2c94..40a60b80 100644 --- a/src/api/routes/ft.ts +++ b/src/api/routes/ft.ts @@ -79,6 +79,7 @@ const IndexRoutes: FastifyPluginCallback, Server, TypeBoxTy sender_address: t.principal?.split('.')[0], image_uri: t.cached_image, image_canonical_uri: t.image, + contract_principal: t.principal, })), }); } diff --git a/src/api/schemas.ts b/src/api/schemas.ts index 093333f6..c93ea400 100644 --- a/src/api/schemas.ts +++ b/src/api/schemas.ts @@ -288,6 +288,9 @@ export const FtBasicMetadataResponse = Type.Object( examples: ['0xef2ac1126e16f46843228b1dk4830e19eb7599129e4jf392cab9e65ae83a45c0'], }), sender_address: Type.String({ examples: ['ST399W7Z9WS0GMSNQGJGME5JAENKN56D65VGMGKGA'] }), + contract_principal: Type.String({ + examples: ['SP1H1733V5MZ3SZ9XRW9FKYGEZT0JDGEB8Y634C7R.miamicoin-token-v2'], + }), }, { title: 'Ft Basic Metadata Response' } ); diff --git a/src/pg/pg-store.ts b/src/pg/pg-store.ts index a5067f71..3e0d9e44 100644 --- a/src/pg/pg-store.ts +++ b/src/pg/pg-store.ts @@ -516,7 +516,7 @@ export class PgStore extends BasePgStore { OFFSET ${args.page.offset} `; return { - total: results[0].total ?? 0, + total: results[0]?.total ?? 0, results: results ?? [], }; }); diff --git a/tests/ft.test.ts b/tests/ft.test.ts index 7b5e5bf8..c74ade47 100644 --- a/tests/ft.test.ts +++ b/tests/ft.test.ts @@ -365,6 +365,7 @@ describe('FT routes', () => { token_uri: 'https://ipfs.io/abcd.json', total_supply: '200000', tx_id: '0xbdc41843d5e0cd4a70611f6badeb5c87b07b12309e77c4fbaf2334c7b4cee89b', + contract_principal: 'SP22PCWZ9EJMHV4PHVS0C8H3B3E4Q079ZHY6CXDS1.meme-token', }); expect(json.results[1]).toStrictEqual({ decimals: 6, @@ -377,6 +378,7 @@ describe('FT routes', () => { token_uri: 'https://cdn.citycoins.co/metadata/miamicoin.json', total_supply: '5586789829000000', tx_id: '0xa80a44790929467693ccb33a212cf50878a6ad572c4c5b8e7d9a5de794fbefa2', + contract_principal: 'SP1H1733V5MZ3SZ9XRW9FKYGEZT0JDGEB8Y634C7R.miamicoin-token-v2', }); expect(json.results[2]).toStrictEqual({ decimals: 6, @@ -389,6 +391,7 @@ describe('FT routes', () => { token_uri: 'https://app.stackswap.org/token/stsw.json', total_supply: '1000000000000000', tx_id: '0x3edffbd025ca2c29cfde8c583c0e0babacd4aa21075d10307d37c64ae78d579e', + contract_principal: 'SP1Z92MPDQEWZXW36VX71Q25HKF5K2EPCJ304F275.stsw-token-v4a', }); }); @@ -402,6 +405,14 @@ describe('FT routes', () => { const json = response.json(); expect(json.total).toBe(1); expect(json.results[0].symbol).toBe('MIA'); + + const response2 = await fastify.inject({ + method: 'GET', + url: '/metadata/ft?name=nothing', // No match + }); + expect(response2.statusCode).toBe(200); + const json2 = response2.json(); + expect(json2.total).toBe(0); }); test('filters by symbol', async () => { @@ -414,6 +425,14 @@ describe('FT routes', () => { const json = response.json(); expect(json.total).toBe(1); expect(json.results[0].symbol).toBe('MIA'); + + const response2 = await fastify.inject({ + method: 'GET', + url: '/metadata/ft?symbol=nothing', // No match + }); + expect(response2.statusCode).toBe(200); + const json2 = response2.json(); + expect(json2.total).toBe(0); }); test('filters by address', async () => { @@ -426,6 +445,14 @@ describe('FT routes', () => { const json = response.json(); expect(json.total).toBe(1); expect(json.results[0].symbol).toBe('MIA'); + + const response2 = await fastify.inject({ + method: 'GET', + url: '/metadata/ft?address=SP1GK6VGCQQGP1PXH5676BY0334CZC41EAA7K1EK3', // No match + }); + expect(response2.statusCode).toBe(200); + const json2 = response2.json(); + expect(json2.total).toBe(0); }); test('sorts by name', async () => {