=======

Datafold Google module

This repository provisions resources on Google, preparing them for a deployment of the application on a GKE cluster.

About this module

Prerequisites

• A Google cloud account, preferably a new isolated one.
• Terraform >= 1.4.6
• A customer contract with Datafold
  • The application does not work without credentials supplied by sales
• Access to our public helm-charts repository

This deployment will create the following resources:

• Google VPC
• Google subnet
• Google GCS bucket for clickhouse backups
• Google external application load balancer
• Google HTTPS certificate, unless preregistered and provided
• Three persistent disk volumes for local data storage
• A GKE cluster
• Service accounts for the GKE cluster to perform actions outside of its cluster boundary:
  • Provisioning persistent disk volumes
  • Updating Network Endpoint Group to route traffic to pods directly

Negative scope

• This module will not provision DNS names in your zone.

How to use this module

• See the example for a potential setup, which has dependencies on our helm-charts
• Create secret files with our variables

Examples

• Implement the example in this repository
• Change the settings
• Run terraform init
• Run terraform apply

Initializing the application

The deployment is created and the initjob should have created the databases and done the initialization of the site settings.

If that didn't complete successfully, try to restart the job.

Once the deployment is complete and the initjob succeeded, we can set the install to that for false in config.yaml:

initjob:
  install: false

Alternatively, here are the manual steps to achieve the same:

Establish a shell into the <deployment>-dfshell container. It is likely that the scheduler and server containers are crashing in a loop.

All we need to is to run these commands:

1. ./manage.py clickhouse create-tables
2. ./manage.py database create-or-upgrade
3. ./manage.py installation set-new-deployment-params

Now all containers should be up and running.

Requirements

Name	Version
dns	3.2.1
google	>= 4.80.0

Providers

Name	Version
google	>= 4.80.0
random	n/a

Modules

Name	Source	Version
clickhouse_backup	./modules/clickhouse_backup	n/a
database	./modules/database	n/a
gke	./modules/gke	n/a
load_balancer	./modules/load_balancer	n/a
networking	./modules/networking	n/a
project-iam-bindings	terraform-google-modules/iam/google//modules/projects_iam	n/a
project_factory_project_services	terraform-google-modules/project-factory/google//modules/project_services	~> 14.4.0

Resources

Name	Type

Inputs

Name	Description	Type	Default	Required
add_onprem_support_group	Flag to add onprem support group for datafold-onprem-support@datafold.com	bool	true	no
clickhouse_backup_sa_key	SA key from secrets	string	""	no
clickhouse_data_disk_size	Data volume size clickhouse	number	40	no
clickhouse_db	Db for clickhouse.	string	"clickhouse"	no
clickhouse_gcs_bucket	GCS Bucket for clickhouse backups.	string	"clickhouse-backups-abcguo23"	no
clickhouse_get_backup_sa_from_secrets_yaml	Flag to toggle getting clickhouse backup SA from secrets.yaml instead of creating new one	bool	false	no
clickhouse_username	Username for clickhouse.	string	"clickhouse"	no
common_tags	Common tags to apply to any resource	map(string)	n/a	yes
create_ssl_cert	True to create the SSL certificate, false if not	bool	false	no
database_name	The name of the database	string	"datafold"	no
database_version	Version of the database	string	"POSTGRES_15"	no
datafold_intercom_app_id	The app id for the intercom. A value other than "" will enable this feature. Only used if the customer doesn't use slack.	string	""	no
db_deletion_protection	A flag that sets delete protection (applied in terraform only, not on the cloud).	bool	true	no
default_node_disk_size	Disk size for a node	number	40	no
deploy_neg_backend	Set this to true to connect the backend service to the NEG that the GKE cluster will create	bool	true	no
deploy_vpc_flow_logs	Flag weither or not to deploy vpc flow logs	bool	false	no
deployment_name	Name of the current deployment.	string	n/a	yes
domain_name	Provide valid domain name (used to set host in GCP)	string	n/a	yes
environment	Global environment tag to apply on all datadog logs, metrics, etc.	string	n/a	yes
gcs_path	Path in the GCS bucket to the backups	string	"backups"	no
github_endpoint	URL of Github enpoint to connect to. Useful for GH Enterprise.	string	""	no
gitlab_endpoint	URL of Gitlab enpoint to connect to. Useful for GH Enterprise.	string	""	no
host_override	A valid domain name if they provision their own DNS / routing	string	""	no
lb_app_rules	Extra rules to apply to the application load balancer for additional filtering	list(object({ action = string priority = number description = string match_type = string # can be either "src_ip_ranges" or "expr" versioned_expr = string # optional, only used if match_type is "src_ip_ranges" src_ip_ranges = list(string) # optional, only used if match_type is "src_ip_ranges" expr = string # optional, only used if match_type is "expr" }))	n/a	yes
lb_layer_7_ddos_defence	Flag to toggle layer 7 ddos defence	bool	false	no
legacy_naming	Flag to toggle legacy behavior - like naming of resources	bool	true	no
mig_disk_type	https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/compute_instance_template#disk_type	string	"pd-balanced"	no
postgres_allocated_storage	The amount of allocated storage for the postgres database	number	20	no
postgres_instance	GCP instance type for PostgreSQL database. Available instance groups: . Available instance classes: .	string	"db-custom-2-7680"	no
postgres_ro_username	Postgres read-only user name	string	"datafold_ro"	no
postgres_username	The username to use for the postgres CloudSQL database	string	"datafold"	no
project_id	The project to deploy to, if not set the default provider project is used.	string	n/a	yes
provider_azs	Provider AZs list, if empty we get AZs dynamically	list(string)	n/a	yes
provider_region	Region for deployment in GCP	string	n/a	yes
redis_data_size	Redis volume size	number	10	no
remote_storage	Type of remote storage for clickhouse backups.	string	"gcs"	no
restricted_roles	Flag to stop certain IAM related resources from being updated/changed	bool	false	no
restricted_viewer_role	Flag to stop certain IAM related resources from being updated/changed	bool	false	no
ssl_cert_name	Provide valid SSL certificate name in GCP OR ssl_private_key_path and ssl_cert_path	string	""	no
ssl_cert_path	SSL certificate path	string	""	no
ssl_private_key_path	Private SSL key path	string	""	no
vpc_cidr	Network CIDR for VPC	string	"10.0.0.0/16"	no
vpc_flow_logs_interval	Interval for vpc flow logs	string	"INTERVAL_5_SEC"	no
vpc_flow_logs_sampling	Sampling for vpc flow logs	string	"0.5"	no
vpc_id	Provide ID of existing VPC if you want to omit creation of new one	string	""	no
vpc_master_cidr_block	cidr block for k8s master, must be a /28 block.	string	"192.168.0.0/28"	no
vpc_secondary_cidr_pods	Network CIDR for VPC secundary subnet 1	string	"/17"	no
vpc_secondary_cidr_services	Network CIDR for VPC secundary subnet 2	string	"/17"	no
whitelist_all_ingress_cidrs_lb	Normally we filter on the load balancer, but some customers want to filter at the SG/Firewall. This flag will whitelist 0.0.0.0/0 on the load balancer.	bool	false	no
whitelisted_egress_cidrs	List of Internet addresses to which the application has access	list(string)	n/a	yes
whitelisted_ingress_cidrs	List of CIDRs that can access the HTTP/HTTPS	list(string)	n/a	yes

Outputs

Name	Description
clickhouse_backup_sa	Name of the clickhouse backup Service Account
clickhouse_data_size	Size in GB of the clickhouse data volume
clickhouse_data_volume_id	Volume ID of the clickhouse data PD volume
clickhouse_gcs_bucket	Name of the GCS bucket for the clickhouse backups
clickhouse_logs_size	Size in GB of the clickhouse logs volume
clickhouse_logs_volume_id	Volume ID of the clickhouse logs PD volume
clickhouse_password	Password to use for clickhouse
cloud_provider	The cloud provider creating all the resources
cluster_name	The name of the GKE cluster that was created
db_instance_id	The database instance ID
deployment_name	The name of the deployment
domain_name	The domain name on the HTTPS certificate
lb_external_ip	The load balancer IP when it was provisioned.
neg_name	The name of the Network Endpoint Group where pods need to be registered from kubernetes.
postgres_database_name	The name of the postgres database
postgres_host	The hostname of the postgres database
postgres_password	The postgres password
postgres_port	The port of the postgres database
postgres_username	The postgres username
redis_data_size	The size in GB of the redis data volume
redis_data_volume_id	The volume ID of the Redis PD data volume
redis_password	The Redis password
vpc_cidr	The CIDR range of the VPC
vpc_id	The ID of the Google VPC the cluster runs in.
vpc_subnetwork	The subnet in which the cluster is created