The world of Kubernetes management is experiencing an Operator revolution.
“The Operator pattern aims to capture the key aim of a human operator who is managing a service or set of services. Human operators who look after specific applications and services have deep knowledge of how the system ought to behave, how to deploy it, and how to react if there are problems.
“People who run workloads on Kubernetes often like to use automation to take care of repeatable tasks. The Operator pattern captures how you can write code to automate a task beyond what Kubernetes itself provides.”
Unfortunately, operators are all too often wresting control away from the mere humans who would manage their own resources. Most Operators embed their logic in compiled container images. The typical human systems operator does not program in Go and does not have a container image build pipeline. Human application operators may have experince with builds, but often do not have cluster administrative rights required to deploy operators.
Anarchy democratizes the potential of the Operator pattern by putting the operator logic in the operator configuration. Users write their own logic in Ansible, which requires little more programming experience than using a command line. Through configuration, the Ansible code is directly controlled at runtime to automate anything you can do in the cluster. Anarchy makes your cluster truly self-governing.
TODO - Add example anarchy operator for managing an application with a Helm chart.
Each AnarchySubject is managed by an AnarchyGovernor. Each AnarchyGovernor is configured to run Ansible tasks on a single AnarchyRunner. AnarchySubjects can have AnarchyActions created to schedule task execution. AnarchyRuns are created for specific Ansible task execution, for processing an subject event handler, run an action, or handle an action callback event.
-
An action defined in the AnarchyGovernor config is scheduled for an AnarchySubject, creating an AnarchyAction
-
An AnarchyRun is created to execute the AnarchyAction’s tasks
-
Ansible in the AnarchyRun calls a Job Template or Survey in Ansible Tower, passing the callback URL and token for the AnarchyAction
-
Ansible executing in Ansible Tower calls the AnarchyAction callback
-
A new AnarchyRun is created to process the callback from Tower
-
The AnarchyRun for the callback updates the status of the AnarchySubject
Anarchy is using [Kopf](https://github.com/nolar/kopf) and needs the kopfpeerings.kopf.dev CustomResourceDefinition must be configured before deploying Anarchy.
oc apply -f kopfpeerings.crd.yaml
Clone this repo to your local workstation and change directory:
git clone https://github.com/redhat-cop/anarchy.git cd anarchy
Install with Helm
helm template anarchy helm/ --include-crds | oc apply -f -
Check your to make sure the anarchy-operator pod is running and the default runner starts soon after:
$ oc get pods NAME READY STATUS RESTARTS AGE anarchy-5f7d7898d7-f6dww 1/1 Running 0 11m anarchy-runner-default-9767c79b6-vt6b4 1/1 Running 0 12m
Anarchy can be installed even if you do not have cluster admin privileges. This will require the CRDs and ClusterRoles to already have been installed.
helm template anarchy helm/ | oc apply -f -
The Anarchy Operator is configured with custom resource types AnarchyAPI and AnarchyGovernor in order to manage AnarchySubject resources. AnarchyAPI resources define how to communicate with an API endpoint while AnarchyGovernor resources define how to interact with APIs to create and manage AnarchySubject resources. Each AnarchySubject is managed according to a single AnarchyGovernor. The AnarchyGovernor defines actions to perform against APIs to instantiate and manage the AnarchySubject. Each action performed for an AnarchySubject according to the AnarchyGovernor definition is represented as an AnarchyAction custom resource. An AnarchyAction always begins with an call to an API. The Anarchy operator listens for callbacks to its own API for events relating to actions such as notifications that an action has completed, or encountered an error. The AnarchyGovernor defines event handlers for actions which may include scheduling further AnarchyActions to occur for the AnarchySubject.
This repository includes a test suite that demonstrates these capabilities by calling a test API. The usage of the test suite is explained in the "Testing" section below. The conceptual overview of the test design is described here.
Let’s start with the AnarchySubject definition:
apiVersion: anarchy.gpte.redhat.com/v1
kind: AnarchySubject
metadata:
generateName: test-
namespace: anarchy-operator
spec:
desiredState: started (1)
governor: test (2)
parameters: (3)
openshift_release: "4.1"
aws_region: us-east-1
repo_version: "3.11"
subdomain_base_suffix: .example.opentlc.com
-
The desired state of the resource, this is an arbitrary string which should be implemented by the AnarchyGovernor.
-
The test AnarchySubject references the name of the AnarchyGovernor that will manage it.
-
Each subject may include a list of parameters to pass to the API, though the governor and API get the final say in how and when the parameters are used.
The test AnarchyGovernor definition is shown here:
apiVersion: anarchy.gpte.redhat.com/v1
kind: AnarchyGovernor
metadata:
name: test
spec:
# Ansible processing for this governor will occur on the default runner.
runner: default
var:
ansible_tower_hostname: tower.example.com
cloud_provider: ec2
varSecrets:
- name: api-creds
var: api_creds
- name: aws-credentials
# The `subjectEventHandlers` provide configuration for how to respond to
# AnarchySubjects being added, updated, and deleted.
subjectEventHandlers:
# The `create` event is processed only for subjects that are newly created.
create:
tasks:
# The `anarchy_subject_update` module is provided to make it easy to
# update the AnarchySubject relating to the current action.
- name: Set state provision-scheduled in subject status
anarchy_subject_update:
metadata:
labels:
state: provision-scheduled
status:
state: provision-scheduled
# The `anarchy_schedule_action` module is used to create AnarchyActions
# for the current AnarchySubject. In this case it schedules an
# AnarchyAction to be processed immediately.
- name: Start Provision
anarchy_schedule_action:
action: provision
# The `update` event is processed when a resource changes.
update:
# The `anarchy_subject` variable stores the state of the AnarchySubject
# which triggered this update. A useful pattern is to implement state
# handling using `spec.desiredState` and `status.state`.
- when: >-
anarchy_subject.spec.desiredState|default('') == 'started' and
(anarchy_subject.status|default({})).state|default('') == 'stopped'
block:
- name: Set state start-scheduled in subject status
anarchy_subject_update:
metadata:
labels:
state: start-scheduled
status:
state: start-scheduled
- name: Schedule start
anarchy_schedule_action:
action: start
- when: >-
anarchy_subject.spec.desiredState|default('stopped') == 'stopped' and
(anarchy_subject.status|default({})).state|default('') == 'started'
block:
- name: Set state stop-scheduled in subject status
anarchy_subject_update:
metadata:
labels:
state: stop-scheduled
status:
state: stop-scheduled
- name: Schedule stop
anarchy_schedule_action:
action: stop
# The `delete` event is processed when a subject delete is requsted. This
# is detected by the presence of a `metadata.deletionTimestamp`. This should
# schedule an action that will result in removing the finalizer from the
# subject when complete.
delete:
tasks:
- name: Schedule destroy
anarchy_schedule_action:
action: destroy
# Actions represent entry points for doing something related to a resource.
# Each action here consists of an API request followed by `callbackHandlers`
# to respond to callbacks from the API endpoint.
actions:
provision:
tasks:
- name: Call API
uri:
url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
url_username: "{{ api_creds.user }}"
url_password: "{{ api_creds.password }}"
validate_certs: false
method: POST
return_content: true
body_format: json
body:
extra_vars:
job_vars: >-
{{ anarchy_subject.vars.job_vars | default({})
| combine(anarchy_governor.vars.job_vars, recursive=True)
| combine({
'ACTION': 'provision',
'__meta__': {
'deployer': {'entry_point': 'ansible/main.yml'},
'tower': {'action': 'provision'}
}
}, recursive=True)
}}
ignore_errors: true
callbackHandlers:
started:
tasks:
- name: Set state provisioning in subject status
anarchy_subject_update:
metadata:
labels:
state: provisioning
status:
state: provisioning
- event: complete
tasks:
- name: Set state started in subject status
anarchy_subject_update:
metadata:
labels:
state: started
status:
state: started
# Subsequent actions are scheduled to run later with the `after` parameter.
- name: Schedule stop
anarchy_schedule_action:
action: stop
after: 8h
- name: Schedule destroy
anarchy_schedule_action:
action: destroy
after: 6d
stop:
tasks:
- name: Call API for stop
uri:
url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
url_username: "{{ api_creds.user }}"
url_password: "{{ api_creds.password }}"
validate_certs: false
method: POST
return_content: true
body_format: json
body:
extra_vars:
job_vars: >-
{{ anarchy_subject.vars.job_vars | default({})
| combine(anarchy_governor.vars.job_vars, recursive=True)
| combine({
'ACTION': 'stop',
'__meta__': {
'deployer': {'entry_point': 'ansible/lifecycle.yml'},
'tower': {'action': 'stop'}
}
}, recursive=True)
}}
ignore_errors: true
callbackHandlers:
started:
tasks:
- name: Set state stopping in subject status
anarchy_subject_update:
spec:
desiredState: stopped
metadata:
labels:
state: stopping
status:
state: stopping
complete:
tasks:
- name: Set state stopped in subject status
anarchy_subject_update:
metadata:
labels:
state: stopped
status:
state: stopped
start:
tasks:
- name: Call API
uri:
url: https://{{ ansible_tower_hostname }}/api/v2/job_templates/job-runner/launch/
url_username: "{{ api_creds.user }}"
url_password: "{{ api_creds.password }}"
validate_certs: false
method: POST
return_content: true
body_format: json
body:
extra_vars:
job_vars: >-
{{ anarchy_subject.vars.job_vars | default({})
| combine(anarchy_governor.vars.job_vars, recursive=True)
| combine({
'ACTION': 'start',
'__meta__': {
'deployer': {'entry_point': 'ansible/lifecycle.yml'},
'tower': {'action': 'start'}
}
}, recursive=True)
}}
ignore_errors: true
callbackHandlers:
started:
tasks:
- name: Set state starting in subject status
anarchy_subject_update:
metadata:
labels:
state: starting
status:
state: starting
complete:
tasks:
- name: Set state started in subject status
anarchy_subject_update:
metadata:
labels:
state: started
status:
state: started
- name: Schedule stop
anarchy_schedule_action:
action: stop
after: 8h
destroy:
tasks:
- name: Call API for destroy
uri:
url: https://{{ babylon_tower_hostname }}/api/v2/job_templates/job-runner/launch/
url_username: "{{ api_creds.user }}"
url_password: "{{ api_creds.password }}"
validate_certs: false
method: POST
return_content: true
body_format: json
body:
extra_vars:
job_vars: >-
{{ anarchy_subject.vars.job_vars | default({})
| combine(anarchy_governor.vars.job_vars, recursive=True)
| combine({
'ACTION': 'destroy',
'__meta__': {
'deployer': {'entry_point': 'ansible/destroy.yml'},
'tower': {'action': 'destroy'}
}
}, recursive=True)
}}
ignore_errors: true
callbackHandlers:
complete:
tasks:
- name: Delete anarchy subject
anarchy_subject_delete:
remove_finalizers: true