-
Notifications
You must be signed in to change notification settings - Fork 20
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
### Features * [Coordinators] moving /server/bin to / * [GCP] Add realtime notifier tool * [sharding] Add metrics for set query * A delta based request generator that creates KV requests from delta files * A generic grpc client that sends a request and returns a response * Add ability to point to real coordinators through parameters * Add bazel configs for roma legacy vs sandboxed * Add bazel macro to generate UDF delta file from wasm binary and JS * Add delta based request generator to the simulation system * Add delta file notifier and loader to the request simulation system * Add Dependencies section to release notes * Add functions for serializing and deserializing shard mapping records * Add GCP features and fixes sections in release notes * add GCP platform and instance * Add GCP realtime notifier * Add GCP terraform config for metrics collector * Add logical sharding config schema and constants * Add metrics collector endpoint to the parameter * Add metrics collector to periodically print and publish metrics * Add NAT gateway for public internet access * Add OSSF Scorecard badge to top-level README * Add OSSF Scorecard GitHub Action * add service mesh to GCP platform * Add support to execute queries over sharded sets. * Add tool to convert C++ to JS with inline WASM * add unit tests to gcp parameter client * Add version to UdfConfig. * Adding GCP version of the message service * Build AMI for request simulation system * Coordinators: Add missing permission * Deploy otel with request simulation system to AWS * Enable metrics and tune the performance for request simulation system * GCP terraform and parameter client refactor * Generate synthetic requests at fixed rate * grpc client worker to send requests at configurable QPS * Integrating kv value server with the KeyFetcherManagerInterface * Register get values hook for string and binary output format * Run request simulation system in local * Skip delta and snapshot files that belong to other shards * Support writing shard mapping configs from csv * Upgrade build-system to v0.33.0 * Upgrade data-plane-shared-libraries to 2023-07-12 commit. * Upgrade data-plane-shared-libraries to 2023-07-21 commit. * Upgrade data-plane-shared-libraries to 2023-07-26 commit. * Upgrade data-plane-shared-libraries to 2023-08-16 commit. * use local parameter client for gcp platform's local instance ### Bug Fixes * Acquire read lock on the set before iterating over it. * Add action_env for asan bazel config * Add noexcept bazel config * Add seccomp-unconfined flag to build_and_test_all_in_docker * Check that fb strings are not nullptr. * Check the metrics collector connection during telemetry initialization * Do not pass metrics collector endpoint for local and aws instance * Don't move references in sample udf code. * Explicitly cast return values of set operations to r-value references * fix another asan error in the test * Fix data loading num threads in param client local test * Fix grpc client error * Fix the error messages printed in the unit test logs * Fix UDF function handler name * logMessage should set an output string. * make the response outlive grpc client call * Rearrange bazel config for clarity * Reduce the number of client workers to 2 in the unit tests to limit the number of threads created * remove GRPC 4mb payload limit * remove local variant of component tools. * Remove redundant docker security-opt * Set bazel workspace name * Write data record for set data in the delta test file generator ### Dependencies * **deps:** Upgrade build-system to 0.42.1 * **deps:** Upgrade build-system to v0.41.1 * **deps:** Upgrade emscripten to 3.1.44 ### GCP: Features * **GCP:** Add realtime thread pool manager ### Documentation * Add docs on getValuesBinary API * Realtime directory is not optional for local dev * Update docker run instructions to include security-opt flag * Update inline WASM docs with instructions on how to test it * Update screenshot of delta file Bug: N/A Change-Id: I82d6a3c2757f29c8c4ce878e04936639cedcd218 GitOrigin-RevId: 34dcc51f137f5349c8ad05c5738771f6a5fda548
- Loading branch information
Showing
392 changed files
with
17,057 additions
and
2,564 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| # Copyright 2023 Google LLC | ||
| # | ||
| # Licensed under the Apache License, Version 2.0 (the "License"); | ||
| # you may not use this file except in compliance with the License. | ||
| # You may obtain a copy of the License at | ||
| # | ||
| # http://www.apache.org/licenses/LICENSE-2.0 | ||
| # | ||
| # Unless required by applicable law or agreed to in writing, software | ||
| # distributed under the License is distributed on an "AS IS" BASIS, | ||
| # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
| # See the License for the specific language governing permissions and | ||
| # limitations under the License. | ||
|
|
||
| # Workflow for the OSSF Scorecards Action | ||
| # https://github.com/ossf/scorecard-action#installation | ||
|
|
||
| name: Scorecard supply-chain security | ||
| on: | ||
| # For Branch-Protection check. Only the default branch is supported. See | ||
| # https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection | ||
| branch_protection_rule: | ||
| # To guarantee Maintained check is occasionally updated. See | ||
| # https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained | ||
| schedule: | ||
| - cron: '35 10 * * 4' | ||
| push: | ||
| branches: | ||
| - main | ||
|
|
||
| # Declare default permissions as read only. | ||
| permissions: read-all | ||
|
|
||
| jobs: | ||
| analysis: | ||
| name: Scorecard analysis | ||
| runs-on: ubuntu-latest | ||
| permissions: | ||
| # Needed to upload the results to code-scanning dashboard. | ||
| security-events: write | ||
| # Needed to publish results and get a badge (see publish_results below). | ||
| id-token: write | ||
| # Uncomment the permissions below if installing in a private repository. | ||
| # contents: read | ||
| # actions: read | ||
|
|
||
| steps: | ||
| - name: Checkout code | ||
| uses: actions/checkout@93ea575cb5d8a053eaa0ac8fa3b40d7e05a33cc8 # v3.1.0 | ||
| with: | ||
| persist-credentials: false | ||
|
|
||
| - name: Run analysis | ||
| uses: ossf/scorecard-action@e38b1902ae4f44df626f11ba0734b14fb91f8f86 # v2.1.2 | ||
| with: | ||
| results_file: results.sarif | ||
| results_format: sarif | ||
| # (Optional) "write" PAT token. Uncomment the `repo_token` line below if: | ||
| # - you want to enable the Branch-Protection check on a *public* repository, or | ||
| # - you are installing Scorecard on a *private* repository | ||
| # To create the PAT, follow the steps in https://github.com/ossf/scorecard-action#authentication-with-pat. | ||
| # repo_token: ${{ secrets.SCORECARD_TOKEN }} | ||
|
|
||
| # Public repositories: | ||
| # - Publish results to OpenSSF REST API for easy access by consumers | ||
| # - Allows the repository to include the Scorecard badge. | ||
| # - See https://github.com/ossf/scorecard-action#publishing-results. | ||
| # For private repositories: | ||
| # - `publish_results` will always be set to `false`, regardless | ||
| # of the value entered here. | ||
| publish_results: true | ||
|
|
||
| # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF | ||
| # format to the repository Actions tab. | ||
| - name: Upload artifact | ||
| uses: actions/upload-artifact@3cea5372237819ed00197afe530f5a7ea3e805c8 # v3.1.0 | ||
| with: | ||
| name: SARIF file | ||
| path: results.sarif | ||
| retention-days: 5 | ||
|
|
||
| # Upload the results to GitHub's code scanning dashboard. | ||
| - name: Upload to code-scanning | ||
| uses: github/codeql-action/upload-sarif@17573ee1cc1b9d061760f3a006fc4aac4f944fd5 # v2.2.4 | ||
| with: | ||
| sarif_file: results.sarif |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -10,6 +10,7 @@ | |
| #Backup files | ||
| *~ | ||
|
|
||
| *.log | ||
| *.tar | ||
| *.eif | ||
| /dist/ | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.