Skip to content

Commit

Permalink
Add a doc paragraph for env vars configurations
Browse files Browse the repository at this point in the history
  • Loading branch information
@hussein-awala
hussein-awala committed Jan 26, 2024
1 parent 5380528 commit 13e9ecd
Showing 1 changed file with 40 additions and 0 deletions.
40 changes: 40 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -142,6 +142,46 @@ To install the helm chart, you can run:
helm install chart spark-on-k8s --values examples/helm/values.yaml
```

## Configuration

The Python client and the CLI can be configured with environment variables to avoid passing the same arguments every
time if you have a common configuration for all your apps. The environment variables are the same for both the client
and the CLI. Here is a list of the available environment variables:

| Environment Variable | Description | Default |
|------------------------------------------|-------------------------------------------------------|--------------|
| SPARK_ON_K8S_DOCKER_IMAGE | The docker image to use for the spark pods | |
| SPARK_ON_K8S_APP_PATH | The path to the app file | |
| SPARK_ON_K8S_NAMESPACE | The namespace to use | default |
| SPARK_ON_K8S_SERVICE_ACCOUNT | The service account to use | spark |
| SPARK_ON_K8S_SPARK_CONF | The spark configuration to use | {} |
| SPARK_ON_K8S_CLASS_NAME | The class name to use | |
| SPARK_ON_K8S_APP_ARGUMENTS | The arguments to pass to the app | [] |
| SPARK_ON_K8S_APP_WAITER | The waiter to use to wait for the app to finish | no_wait |
| SPARK_ON_K8S_IMAGE_PULL_POLICY | The image pull policy to use | IfNotPresent |
| SPARK_ON_K8S_UI_REVERSE_PROXY | Whether to use a reverse proxy to access the spark UI | false |
| SPARK_ON_K8S_DRIVER_CPU | The driver CPU | 1 |
| SPARK_ON_K8S_DRIVER_MEMORY | The driver memory | 1024 |
| SPARK_ON_K8S_DRIVER_MEMORY_OVERHEAD | The driver memory overhead | 512 |
| SPARK_ON_K8S_EXECUTOR_CPU | The executor CPU | 1 |
| SPARK_ON_K8S_EXECUTOR_MEMORY | The executor memory | 1024 |
| SPARK_ON_K8S_EXECUTOR_MEMORY_OVERHEAD | The executor memory overhead | 512 |
| SPARK_ON_K8S_EXECUTOR_MIN_INSTANCES | The minimum number of executor instances | |
| SPARK_ON_K8S_EXECUTOR_MAX_INSTANCES | The maximum number of executor instances | |
| SPARK_ON_K8S_EXECUTOR_INITIAL_INSTANCES | The initial number of executor instances | |
| SPARK_ON_K8S_CONFIG_FILE | The path to the config file | |
| SPARK_ON_K8S_CONTEXT | The context to use | |
| SPARK_ON_K8S_CLIENT_CONFIG | The sync Kubernetes client configuration to use | |
| SPARK_ON_K8S_ASYNC_CLIENT_CONFIG | The async Kubernetes client configuration to use | |
| SPARK_ON_K8S_IN_CLUSTER | Whether to use the in cluster Kubernetes config | false |
| SPARK_ON_K8S_API_DEFAULT_NAMESPACE | The default namespace to use for the API | default |
| SPARK_ON_K8S_API_HOST | The host to use for the API | 127.0.0.1 |
| SPARK_ON_K8S_API_PORT | The port to use for the API | 8000 |
| SPARK_ON_K8S_API_WORKERS | The number of workers to use for the API | 4 |
| SPARK_ON_K8S_API_LOG_LEVEL | The log level to use for the API | info |
| SPARK_ON_K8S_API_LIMIT_CONCURRENCY | The limit concurrency to use for the API | 1000 |


## Examples

Here are some examples of how to package and submit spark apps with this package.
Expand Down

0 comments on commit 13e9ecd

Please sign in to comment.