diff --git a/README.md b/README.md index dd9fbd6..a85349f 100644 --- a/README.md +++ b/README.md @@ -1,130 +1,6 @@ -## Distribute Load Testing Using GKE +## Distributed Load Testing Using GKE and Locust -## Introduction - -Load testing is key to the development of any backend infrastructure because load tests demonstrate how well the system functions when faced with real-world demands. An important aspect of load testing is the proper simulation of user and device behavior to identify and understand any possible system bottlenecks, well in advance of deploying applications to production. - -However, dedicated test infrastructure can be expensive and difficult to maintain because it is not needed on a continuous basis. Moreover, dedicated test infrastructure is often a one-time capital expense with a fixed capacity, which makes it difficult to scale load testing beyond the initial investment and can limit experimentation. This can lead to slowdowns in productivity for development teams and lead to applications that are not properly tested before production deployments. - -## Before you begin - -Open Cloud Shell to execute the commands listed in this tutorial. - -Define environment variables for the project id, region and zone you want to use for this tutorial. - - $ PROJECT=$(gcloud config get-value project) - $ REGION=us-central1 - $ ZONE=${REGION}-b - $ CLUSTER=gke-load-test - $ TARGET=${PROJECT}.appspot.com - $ gcloud config set compute/region $REGION - $ gcloud config set compute/zone $ZONE - -**Note:** Following services should be enabled in your project: -Cloud Build -Kubernetes Engine -Google App Engine Admin API -Cloud Storage - - $ gcloud services enable \ - cloudbuild.googleapis.com \ - compute.googleapis.com \ - container.googleapis.com \ - containeranalysis.googleapis.com \ - containerregistry.googleapis.com - -## Load testing tasks - -To deploy the load testing tasks, you first deploy a load testing master and then deploy a group of load testing workers. With these load testing workers, you can create a substantial amount of traffic for testing purposes. - -**Note:** Keep in mind that generating excessive amounts of traffic to external systems can resemble a denial-of-service attack. Be sure to review the Google Cloud Platform Terms of Service and the Google Cloud Platform Acceptable Use Policy. - -## Load testing master - -The first component of the deployment is the Locust master, which is the entry point for executing the load testing tasks described above. The Locust master is deployed with a single replica because we need only one master. - -The configuration for the master deployment specifies several elements, including the ports that need to be exposed by the container (`8089` for web interface, `5557` and `5558` for communicating with workers). This information is later used to configure the Locust workers. The following snippet contains the configuration for the ports: - - ports: - - name: loc-master-web - containerPort: 8089 - protocol: TCP - - name: loc-master-p1 - containerPort: 5557 - protocol: TCP - - name: loc-master-p2 - containerPort: 5558 - protocol: TCP - -Next, we would deploy a Service to ensure that the exposed ports are accessible to other pods via `hostname:port` within the cluster, and referenceable via a descriptive port name. The use of a service allows the Locust workers to easily discover and reliably communicate with the master, even if the master fails and is replaced with a new pod by the deployment. The Locust master service also includes a directive to create an external forwarding rule at the cluster level (i.e. type of LoadBalancer), which provides the ability for external traffic to access the cluster resources. - -After you deploy the Locust master, you can access the web interface using the public IP address of the external forwarding rule. After you deploy the Locust workers, you can start the simulation and look at aggregate statistics through the Locust web interface. - -## Load testing workers - -The next component of the deployment includes the Locust workers, which execute the load testing tasks described above. The Locust workers are deployed by a single deployment that creates multiple pods. The pods are spread out across the Kubernetes cluster. Each pod uses environment variables to control important configuration information such as the hostname of the system under test and the hostname of the Locust master. - -After the Locust workers are deployed, you can return to the Locust master web interface and see that the number of slaves corresponds to the number of deployed workers. - -## Setup - -1. Create GKE cluster - - $ gcloud container clusters create $CLUSTER \ - --zone $ZONE \ - --scopes "https://www.googleapis.com/auth/cloud-platform" \ - --num-nodes "3" \ - --enable-autoscaling --min-nodes "3" \ - --max-nodes "10" \ - --addons HorizontalPodAutoscaling,HttpLoadBalancing - - $ gcloud container clusters get-credentials $CLUSTER \ - --zone $ZONE \ - --project $PROJECT - -2. Clone tutorial repo in a local directory on your cloud shell environment - - $ git clone - -3. Build docker image and store it in your project's container registry - - $ pushd gke-load-test - $ gcloud builds submit --tag gcr.io/$PROJECT/locust-tasks:latest docker-image/. - -4. Deploy sample application on GAE - - $ gcloud app deploy sample-webapp/app.yaml --project=$PROJECT - -5. Replace [TARGET_HOST] and [PROJECT_ID] in locust-master-controller.yaml and locust-worker-controller.yaml with the deployed endpoint and project-id respectively. - - $ sed -i -e "s/\[TARGET_HOST\]/$TARGET/g" kubernetes-config/locust-master-controller.yaml - $ sed -i -e "s/\[TARGET_HOST\]/$TARGET/g" kubernetes-config/locust-worker-controller.yaml - $ sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kubernetes-config/locust-master-controller.yaml - $ sed -i -e "s/\[PROJECT_ID\]/$PROJECT/g" kubernetes-config/locust-worker-controller.yaml - -6. Deploy Locust master and worker nodes: - - $ kubectl apply -f kubernetes-config/locust-master-controller.yaml - $ kubectl apply -f kubernetes-config/locust-master-service.yaml - $ kubectl apply -f kubernetes-config/locust-worker-controller.yaml - -7. Get the external ip of Locust master service - - $ EXTERNAL_IP=$(kubectl get svc locust-master -o yaml | grep ip | awk -F":" '{print $NF}') - -8. Starting load testing -The Locust master web interface enables you to execute the load testing tasks against the system under test, as shown in the following image. Access the url as http://$EXTERNAL_IP:8089. - -To begin, specify the total number of users to simulate and a rate at which each user should be spawned. Next, click Start swarming to begin the simulation. To stop the simulation, click **Stop** and the test will terminate. The complete results can be downloaded into a spreadsheet. - -9. [Optional] Scaling clients -Scaling up the number of simulated users will require an increase in the number of Locust worker pods. To increase the number of pods deployed by the deployment, Kubernetes offers the ability to resize deployments without redeploying them. For example, the following command scales the pool of Locust worker pods to 20: - - $ kubectl scale deployment/locust-worker --replicas=20 - -## Cleaning up - - $ gcloud container clusters delete $CLUSTER --zone $ZONE +This is the sample code for the [Distributed load testing using Google Kubernetes Engine](https://cloud.google.com/architecture/distributed-load-testing-using-gke) tutorial. ## License diff --git a/docker-image/Dockerfile b/docker-image/Dockerfile index 230756a..0f0ff2c 100644 --- a/docker-image/Dockerfile +++ b/docker-image/Dockerfile @@ -1,4 +1,4 @@ -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,8 +13,8 @@ # limitations under the License. -# Start with a base Python 3.7.2 image -FROM python:3.7.2 +# Start with a base image Python 3.9.12 Debian 11 (bullseye) slim +FROM python:3.9.12-slim-bullseye # Add the licenses for third party software and libraries ADD licenses /licenses diff --git a/docker-image/locust-tasks/requirements.txt b/docker-image/locust-tasks/requirements.txt index 14c17cf..1fa8603 100644 --- a/docker-image/locust-tasks/requirements.txt +++ b/docker-image/locust-tasks/requirements.txt @@ -1,18 +1,31 @@ -certifi==2019.3.9 -chardet==3.0.4 -Click==7.0 -Flask==1.0.2 -gevent==1.4.0 -greenlet==0.4.15 -idna==2.8 -itsdangerous==1.1.0 -Jinja2==2.10.1 -locustio==0.11.0 -MarkupSafe==1.1.1 -msgpack==0.6.1 +Brotli==1.0.9 +certifi==2021.10.8 +chardet==4.0.0 +charset-normalizer==2.0.12 +click==8.1.2 +ConfigArgParse==1.5.3 +Flask==2.1.1 +Flask-BasicAuth==0.2.0 +Flask-Cors==3.0.10 +gevent==21.12.0 +geventhttpclient==1.5.3 +greenlet==1.1.2 +idna==3.3 +importlib-metadata==4.11.3 +itsdangerous==2.1.2 +Jinja2==3.0.3 +locust==2.8.6 +MarkupSafe==2.1.1 +msgpack==1.0.3 msgpack-python==0.5.6 -pyzmq==18.0.1 -requests==2.21.0 -six==1.12.0 -urllib3==1.24.2 -Werkzeug==0.15.1 +psutil==5.9.0 +pyzmq==22.3.0 +requests==2.27.1 +roundrobin==0.0.2 +six==1.16.0 +typing_extensions==4.1.1 +urllib3==1.26.9 +Werkzeug==2.1.1 +zipp==3.8.0 +zope.event==4.5.0 +zope.interface==5.4.0 diff --git a/docker-image/locust-tasks/run.sh b/docker-image/locust-tasks/run.sh index ba5f684..4f909ed 100644 --- a/docker-image/locust-tasks/run.sh +++ b/docker-image/locust-tasks/run.sh @@ -1,6 +1,6 @@ #!/bin/bash -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -22,9 +22,9 @@ LOCUST_MODE=${LOCUST_MODE:-standalone} if [[ "$LOCUST_MODE" = "master" ]]; then LOCUS_OPTS="$LOCUS_OPTS --master" elif [[ "$LOCUST_MODE" = "worker" ]]; then - LOCUS_OPTS="$LOCUS_OPTS --slave --master-host=$LOCUST_MASTER" + LOCUS_OPTS="$LOCUS_OPTS --worker --master-host=$LOCUST_MASTER" fi echo "$LOCUST $LOCUS_OPTS" -$LOCUST $LOCUS_OPTS \ No newline at end of file +$LOCUST $LOCUS_OPTS diff --git a/docker-image/locust-tasks/tasks.py b/docker-image/locust-tasks/tasks.py index 6e6fc55..4e6d6ea 100644 --- a/docker-image/locust-tasks/tasks.py +++ b/docker-image/locust-tasks/tasks.py @@ -1,6 +1,6 @@ #!/usr/bin/env python -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -18,9 +18,11 @@ import uuid from datetime import datetime -from locust import HttpLocust, TaskSet, task +from locust import FastHttpUser, TaskSet, task +# [START locust_test_task] + class MetricsTaskSet(TaskSet): _deviceid = None @@ -38,5 +40,7 @@ def post_metrics(self): "/metrics", {"deviceid": self._deviceid, "timestamp": datetime.now()}) -class MetricsLocust(HttpLocust): - task_set = MetricsTaskSet \ No newline at end of file +class MetricsLocust(FastHttpUser): + tasks = {MetricsTaskSet} + +# [END locust_test_task] diff --git a/kubernetes-config/locust-master-controller.yaml b/kubernetes-config/locust-master-controller.yaml.tpl similarity index 86% rename from kubernetes-config/locust-master-controller.yaml rename to kubernetes-config/locust-master-controller.yaml.tpl index 9e9322c..e5069b5 100644 --- a/kubernetes-config/locust-master-controller.yaml +++ b/kubernetes-config/locust-master-controller.yaml.tpl @@ -1,4 +1,4 @@ -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -31,12 +31,12 @@ spec: spec: containers: - name: locust-master - image: gcr.io/[PROJECT_ID]/locust-tasks:latest + image: ${REGION}-docker.pkg.dev/${PROJECT}/${AR_REPO}/${LOCUST_IMAGE_NAME}:${LOCUST_IMAGE_TAG} env: - name: LOCUST_MODE value: master - name: TARGET_HOST - value: https://[TARGET_HOST] + value: https://${SAMPLE_APP_TARGET} ports: - name: loc-master-web containerPort: 8089 diff --git a/kubernetes-config/locust-master-service.yaml b/kubernetes-config/locust-master-service.yaml.tpl similarity index 78% rename from kubernetes-config/locust-master-service.yaml rename to kubernetes-config/locust-master-service.yaml.tpl index 3757115..6cca5fe 100644 --- a/kubernetes-config/locust-master-service.yaml +++ b/kubernetes-config/locust-master-service.yaml.tpl @@ -1,4 +1,4 @@ -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,10 +21,6 @@ metadata: app: locust-master spec: ports: - - port: 8089 - targetPort: loc-master-web - protocol: TCP - name: loc-master-web - port: 5557 targetPort: loc-master-p1 protocol: TCP @@ -35,4 +31,21 @@ spec: name: loc-master-p2 selector: app: locust-master +--- +kind: Service +apiVersion: v1 +metadata: + name: locust-master-web + annotations: + networking.gke.io/load-balancer-type: "Internal" + labels: + app: locust-master +spec: + ports: + - port: 8089 + targetPort: loc-master-web + protocol: TCP + name: loc-master-web + selector: + app: locust-master type: LoadBalancer diff --git a/kubernetes-config/locust-worker-controller.yaml b/kubernetes-config/locust-worker-controller.yaml.tpl similarity index 83% rename from kubernetes-config/locust-worker-controller.yaml rename to kubernetes-config/locust-worker-controller.yaml.tpl index a2d205f..6583b38 100644 --- a/kubernetes-config/locust-worker-controller.yaml +++ b/kubernetes-config/locust-worker-controller.yaml.tpl @@ -1,4 +1,4 @@ -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -30,11 +30,11 @@ spec: spec: containers: - name: locust-worker - image: gcr.io/[PROJECT_ID]/locust-tasks:latest + image: ${REGION}-docker.pkg.dev/${PROJECT}/${AR_REPO}/${LOCUST_IMAGE_NAME}:${LOCUST_IMAGE_TAG} env: - name: LOCUST_MODE value: worker - name: LOCUST_MASTER value: locust-master - name: TARGET_HOST - value: https://[TARGET_HOST] + value: https://${SAMPLE_APP_TARGET} diff --git a/sample-webapp/.gcloudignore b/sample-webapp/.gcloudignore new file mode 100644 index 0000000..a987f11 --- /dev/null +++ b/sample-webapp/.gcloudignore @@ -0,0 +1,19 @@ +# This file specifies files that are *not* uploaded to Google Cloud Platform +# using gcloud. It follows the same syntax as .gitignore, with the addition of +# "#!include" directives (which insert the entries of the given .gitignore-style +# file at that point). +# +# For more information, run: +# $ gcloud topic gcloudignore +# +.gcloudignore +# If you would like to upload your .git directory, .gitignore file or files +# from your .gitignore file, remove the corresponding line +# below: +.git +.gitignore + +# Python pycache: +__pycache__/ +# Ignored by the build system +/setup.cfg \ No newline at end of file diff --git a/sample-webapp/app.yaml b/sample-webapp/app.yaml index fd4770d..b56981a 100644 --- a/sample-webapp/app.yaml +++ b/sample-webapp/app.yaml @@ -1,4 +1,4 @@ -# Copyright 2015 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,7 +13,7 @@ # limitations under the License. -runtime: python37 +runtime: python39 instance_class: F2 diff --git a/sample-webapp/main.py b/sample-webapp/main.py index 06bc573..2040de7 100644 --- a/sample-webapp/main.py +++ b/sample-webapp/main.py @@ -1,6 +1,6 @@ #!/usr/bin/env python -# Copyright 2019 Google Inc. All rights reserved. +# Copyright 2022 Google Inc. All rights reserved. # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -24,6 +24,7 @@ def root(): return 'Welcome to the "Distributed Load Testing Using Kubernetes" sample web app\n' +# [START sample_app_endpoints] @app.route('/login', methods=['GET', 'POST']) def login(): deviceid = request.values.get('deviceid') @@ -33,8 +34,9 @@ def login(): def metrics(): deviceid = request.values.get('deviceid') timestamp = request.values.get('timestamp') - + return '/metrics - device: {}, timestamp: {}\n'.format(deviceid, timestamp) +# [END sample_app_endpoints] if __name__ == '__main__': diff --git a/scripts/start-proxy.sh b/scripts/start-proxy.sh new file mode 100755 index 0000000..8cc808f --- /dev/null +++ b/scripts/start-proxy.sh @@ -0,0 +1,32 @@ +# Copyright 2022 Google Inc. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -e + +gcloud compute instances create-with-container ${PROXY_VM} \ + --zone ${ZONE} \ + --container-image gcr.io/cloud-marketplace/google/nginx1:latest \ + --container-mount-host-path=host-path=/tmp/server.conf,mount-path=/etc/nginx/conf.d/default.conf \ + --metadata=startup-script="#! /bin/bash + cat < /tmp/server.conf + server { + listen 8089; + location / { + proxy_pass http://${INTERNAL_LB_IP}:8089; + } + } +EOF" + +echo "To open an SSH tunnel between your workstation and this proxy instance, use this command:" +echo " gcloud compute ssh --zone ${ZONE} ${PROXY_VM} -- -N -L 8089:localhost:8089"