Merge pull request #4973 from wozniakjan/logging_eventrouter

Automatic merge from submit-queue

logging: introducing event router

Initial concept integrating kubernetes events to EFK stack using [eventrouter](https://github.com/heptio/eventrouter)
- **eventrouter** is deployed to logging project, has a service account and its own role to read events
- **eventrouter** watches kubernetes events, marshalls them to JSON and outputs to its STDOUT
- **fluentd** picks them up and inserts to elastic search logging project index

Kubernetes events could be easily stored into different index. Among other ways, it could be achieved either by a fluentd filter plugin or custom eventrouter sink.

Pending actions:
- [x] - get our downstream and upstream images for eventrouter, don't use heptio's

cc: @josefkarasek , @jcantrill , @richm

12 files changed, 374 insertions, 2 deletions

diff --git a/roles/openshift_logging/README.md b/roles/openshift_logging/README.md
index de3d19858..9386da8c2 100644
--- a/roles/openshift_logging/README.md
+++ b/roles/openshift_logging/README.md
@@ -12,13 +12,13 @@ generation for Elasticsearch (it uses JKS) as well as openssl to sign certificat
 As part of the installation, it is recommended that you add the Fluentd node selector label
 to the list of persisted [node labels](https://docs.openshift.org/latest/install_config/install/advanced_install.html#configuring-node-host-labels).
 
-###Required vars:
+### Required vars:
 
 - `openshift_logging_install_logging`: When `True` the `openshift_logging` role will install Aggregated Logging.
 
 When `openshift_logging_install_logging` is set to `False` the `openshift_logging` role will uninstall Aggregated Logging.
 
-###Optional vars:
+### Optional vars:
 - `openshift_logging_purge_logging`: When `openshift_logging_install_logging` is set to 'False' to trigger uninstalation and `openshift_logging_purge_logging` is set to 'True', it will completely and irreversibly remove all logging persistent data including PVC. Defaults to 'False'.
 - `openshift_logging_image_prefix`: The prefix for the logging images to use. Defaults to 'docker.io/openshift/origin-'.
 - `openshift_logging_curator_image_prefix`: Setting the image prefix for Curator image. Defaults to `openshift_logging_image_prefix`.
@@ -90,6 +90,12 @@ When `openshift_logging_install_logging` is set to `False` the `openshift_loggin
 - `openshift_logging_es_number_of_shards`: The number of primary shards for every new index created in ES. Defaults to '1'.
 - `openshift_logging_es_number_of_replicas`: The number of replica shards per primary shard for every new index. Defaults to '0'.
 
+- `openshift_logging_install_eventrouter`: Coupled with `openshift_logging_install_logging`. When both are 'True', eventrouter will be installed. When both are 'False', eventrouter will be uninstalled.
+Other combinations will keep the eventrouter untouched.
+
+Detailed eventrouter configuration can be found in
+- `roles/openshift_logging_eventrouter/README.md`
+
 When `openshift_logging_use_ops` is `True`, there are some additional vars. These work the
 same as above for their non-ops counterparts, but apply to the OPS cluster instance:
 - `openshift_logging_es_ops_host`: logging-es-ops
diff --git a/roles/openshift_logging/tasks/delete_logging.yaml b/roles/openshift_logging/tasks/delete_logging.yaml
index 45298e345..3040d15ca 100644
--- a/roles/openshift_logging/tasks/delete_logging.yaml
+++ b/roles/openshift_logging/tasks/delete_logging.yaml
@@ -105,3 +105,9 @@
     - logging-elasticsearch
     - logging-fluentd
     - logging-mux
+
+## EventRouter
+- include_role:
+    name: openshift_logging_eventrouter
+  when:
+    not openshift_logging_install_eventrouter | default(false) | bool
diff --git a/roles/openshift_logging/tasks/install_logging.yaml b/roles/openshift_logging/tasks/install_logging.yaml
index de5e25061..2695ef030 100644
--- a/roles/openshift_logging/tasks/install_logging.yaml
+++ b/roles/openshift_logging/tasks/install_logging.yaml
@@ -270,4 +270,12 @@
     openshift_logging_fluentd_master_url: "{{ openshift_logging_master_url }}"
     openshift_logging_fluentd_namespace: "{{ openshift_logging_namespace }}"
 
+
+## EventRouter
+- include_role:
+    name: openshift_logging_eventrouter
+  when:
+    openshift_logging_install_eventrouter | default(false) | bool
+
+
 - include: update_master_config.yaml
diff --git a/roles/openshift_logging_eventrouter/README.md b/roles/openshift_logging_eventrouter/README.md
new file mode 100644
index 000000000..da313d68b
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/README.md
@@ -0,0 +1,20 @@
+Event router
+------------
+
+A pod forwarding kubernetes events to EFK aggregated logging stack.
+
+- **eventrouter** is deployed to logging project, has a service account and its own role to read events
+- **eventrouter** watches kubernetes events, marshalls them to JSON and outputs to its sink, currently only various formatting to STDOUT
+- **fluentd** picks them up and inserts to elasticsearch *.operations* index
+
+- `openshift_logging_install_eventrouter`: When 'True', eventrouter will be installed. When 'False', eventrouter will be uninstalled.
+
+Configuration variables:
+
+- `openshift_logging_eventrouter_image_prefix`: The prefix for the eventrouter logging image. Defaults to `openshift_logging_image_prefix`.
+- `openshift_logging_eventrouter_image_version`: The image version for the logging eventrouter. Defaults to 'latest'.
+- `openshift_logging_eventrouter_sink`: Select a sink for eventrouter, supported 'stdout' and 'glog'. Defaults to 'stdout'.
+- `openshift_logging_eventrouter_nodeselector`: A map of labels (e.g. {"node":"infra","region":"west"} to select the nodes where the pod will land.
+- `openshift_logging_eventrouter_cpu_limit`: The amount of CPU to allocate to eventrouter. Defaults to '100m'.
+- `openshift_logging_eventrouter_memory_limit`: The memory limit for eventrouter pods. Defaults to '128Mi'.
+- `openshift_logging_eventrouter_namespace`: The namespace where eventrouter is deployed. Defaults to 'default'.
diff --git a/roles/openshift_logging_eventrouter/defaults/main.yaml b/roles/openshift_logging_eventrouter/defaults/main.yaml
new file mode 100644
index 000000000..34e33f75f
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/defaults/main.yaml
@@ -0,0 +1,9 @@
+---
+openshift_logging_eventrouter_image_prefix: "{{ openshift_logging_image_prefix | default(__openshift_logging_image_prefix) }}"
+openshift_logging_eventrouter_image_version: "{{ openshift_logging_image_version | default('latest') }}"
+openshift_logging_eventrouter_replicas: 1
+openshift_logging_eventrouter_sink: stdout
+openshift_logging_eventrouter_nodeselector: ""
+openshift_logging_eventrouter_cpu_limit: 100m
+openshift_logging_eventrouter_memory_limit: 128Mi
+openshift_logging_eventrouter_namespace: default
diff --git a/roles/openshift_logging_eventrouter/files/eventrouter-template.yaml b/roles/openshift_logging_eventrouter/files/eventrouter-template.yaml
new file mode 100644
index 000000000..91708e54b
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/files/eventrouter-template.yaml
@@ -0,0 +1,103 @@
+# this openshift template should match (except nodeSelector) jinja2 template in
+# ../templates/eventrouter-template.j2
+kind: Template
+apiVersion: v1
+metadata:
+  name: eventrouter-template
+  annotations:
+    description: "A pod forwarding kubernetes events to EFK aggregated logging stack."
+    tags: "events,EFK,logging"
+objects:
+  - kind: ServiceAccount
+    apiVersion: v1
+    metadata:
+      name: aggregated-logging-eventrouter
+  - kind: ClusterRole
+    apiVersion: v1
+    metadata:
+      name: event-reader
+    rules:
+    - apiGroups: [""]
+      resources: ["events"]
+      verbs: ["get", "watch", "list"]
+  - kind: ConfigMap
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+    data:
+      config.json: |- 
+        {
+          "sink": "${SINK}"
+        }
+  - kind: DeploymentConfig
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+      labels:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+    spec:
+      selector:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+      replicas: ${REPLICAS}
+      template:
+        metadata:
+          labels:
+            component: eventrouter
+            logging-infra: eventrouter
+            provider: openshift
+          name: logging-eventrouter
+        spec:
+          serviceAccount: aggregated-logging-eventrouter
+          serviceAccountName: aggregated-logging-eventrouter
+          containers:
+            - name: kube-eventrouter
+              image: ${IMAGE}
+              imagePullPolicy: Always
+              resources:
+                limits:
+                  memory: ${MEMORY} 
+                  cpu: ${CPU}
+                requires:
+                  memory: ${MEMORY}
+              volumeMounts:
+              - name: config-volume
+                mountPath: /etc/eventrouter
+          volumes:
+            - name: config-volume
+              configMap:
+                name: logging-eventrouter
+  - kind: ClusterRoleBinding
+    apiVersion: v1
+    metadata:
+      name: event-reader-binding
+    subjects:
+    - kind: ServiceAccount
+      name: aggregated-logging-eventrouter
+      namespace: ${NAMESPACE}
+    roleRef:
+      kind: ClusterRole
+      name: event-reader
+
+parameters:
+  - name: SINK
+    displayName: Sink
+    value: stdout
+  - name: REPLICAS
+    displayName: Replicas
+    value: "1"
+  - name: IMAGE
+    displayName: Image
+    value: "docker.io/openshift/origin-logging-eventrouter:latest"
+  - name: MEMORY
+    displayName: Memory
+    value: "128Mi"
+  - name: CPU
+    displayName: CPU
+    value: "100m"
+  - name: NAMESPACE
+    displayName: Namespace
+    value: default
diff --git a/roles/openshift_logging_eventrouter/tasks/delete_eventrouter.yaml b/roles/openshift_logging_eventrouter/tasks/delete_eventrouter.yaml
new file mode 100644
index 000000000..cf0abbde9
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/tasks/delete_eventrouter.yaml
@@ -0,0 +1,40 @@
+---
+# delete eventrouter
+- name: Delete EventRouter service account
+  oc_serviceaccount:
+    state: absent
+    name: "aggregated-logging-eventrouter"
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+
+- name: Delete event-reader cluster role
+  oc_clusterrole:
+    state: absent
+    name: event-reader
+
+- name: Unset privileged permissions for EventRouter
+  oc_adm_policy_user:
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    resource_kind: cluster-role
+    resource_name: event-reader
+    state: absent
+    user: "system:serviceaccount:{{ openshift_logging_eventrouter_namespace }}:aggregated-logging-eventrouter"
+
+- name: Delete EventRouter configmap
+  oc_configmap:
+    state: absent
+    name: logging-eventrouter
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+
+- name: Delete EventRouter DC
+  oc_obj:
+    state: absent
+    name: logging-eventrouter
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: dc
+
+- name: Delete EventRouter Template
+  oc_obj:
+    state: absent
+    name: eventrouter-template
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: template
diff --git a/roles/openshift_logging_eventrouter/tasks/install_eventrouter.yaml b/roles/openshift_logging_eventrouter/tasks/install_eventrouter.yaml
new file mode 100644
index 000000000..8df7435e2
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/tasks/install_eventrouter.yaml
@@ -0,0 +1,59 @@
+---
+# initial checks
+- assert:
+    msg: Invalid sink type "{{openshift_logging_eventrouter_sink}}", only one of "{{__eventrouter_sinks}}" allowed
+    that: openshift_logging_eventrouter_sink in __eventrouter_sinks
+
+# allow passing in a tempdir
+- name: Create temp directory for doing work in
+  command: mktemp -d /tmp/openshift-logging-ansible-XXXXXX
+  register: mktemp
+  changed_when: False
+
+- set_fact:
+    tempdir: "{{ mktemp.stdout }}"
+
+- name: Create templates subdirectory
+  file:
+    state: directory
+    path: "{{ tempdir }}/templates"
+    mode: 0755
+  changed_when: False
+
+# create EventRouter deployment config
+- name: Generate EventRouter template
+  template:
+    src: eventrouter-template.j2
+    dest: "{{ tempdir }}/templates/eventrouter-template.yaml"
+  vars:
+    node_selector: "{{ openshift_logging_eventrouter_nodeselector | default({}) }}"
+
+- name: Create EventRouter template
+  oc_obj:
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    kind: template
+    name: eventrouter-template
+    state: present
+    files:
+    - "{{ tempdir }}/templates/eventrouter-template.yaml"
+
+- name: Process EventRouter template
+  oc_process:
+    state: present
+    template_name: eventrouter-template
+    namespace: "{{ openshift_logging_eventrouter_namespace }}"
+    params:
+      IMAGE: "{{openshift_logging_eventrouter_image_prefix}}logging-eventrouter:{{openshift_logging_eventrouter_image_version}}"
+      REPLICAS: "{{ openshift_logging_eventrouter_replicas }}"
+      CPU: "{{ openshift_logging_eventrouter_cpu_limit }}"
+      MEMORY: "{{ openshift_logging_eventrouter_memory_limit }}"
+      NAMESPACE: "{{ openshift_logging_eventrouter_namespace }}"
+      SINK: "{{ openshift_logging_eventrouter_sink }}"
+
+## Placeholder for migration when necessary ##
+
+- name: Delete temp directory
+  file:
+    name: "{{ tempdir }}"
+    state: absent
+  changed_when: False
diff --git a/roles/openshift_logging_eventrouter/tasks/main.yaml b/roles/openshift_logging_eventrouter/tasks/main.yaml
new file mode 100644
index 000000000..58e5a559f
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/tasks/main.yaml
@@ -0,0 +1,6 @@
+---
+- include: "{{ role_path }}/tasks/install_eventrouter.yaml"
+  when: openshift_logging_install_eventrouter | default(false) | bool
+
+- include: "{{ role_path }}/tasks/delete_eventrouter.yaml"
+  when: not openshift_logging_install_eventrouter | default(false) | bool
diff --git a/roles/openshift_logging_eventrouter/templates/eventrouter-template.j2 b/roles/openshift_logging_eventrouter/templates/eventrouter-template.j2
new file mode 100644
index 000000000..9ff4c7e80
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/templates/eventrouter-template.j2
@@ -0,0 +1,109 @@
+# this jinja2 template should always match (except nodeSelector) openshift template in
+# ../files/eventrouter-template.yaml
+kind: Template
+apiVersion: v1
+metadata:
+  name: eventrouter-template
+  annotations:
+    description: "A pod forwarding kubernetes events to EFK aggregated logging stack."
+    tags: "events,EFK,logging"
+objects:
+  - kind: ServiceAccount
+    apiVersion: v1
+    metadata:
+      name: aggregated-logging-eventrouter
+  - kind: ClusterRole
+    apiVersion: v1
+    metadata:
+      name: event-reader
+    rules:
+    - apiGroups: [""]
+      resources: ["events"]
+      verbs: ["get", "watch", "list"]
+  - kind: ConfigMap
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+    data:
+      config.json: |- 
+        {
+          "sink": "${SINK}"
+        }
+  - kind: DeploymentConfig
+    apiVersion: v1
+    metadata:
+      name: logging-eventrouter
+      labels:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+    spec:
+      selector:
+        component: eventrouter
+        logging-infra: eventrouter
+        provider: openshift
+      replicas: ${REPLICAS}
+      template:
+        metadata:
+          labels:
+            component: eventrouter
+            logging-infra: eventrouter
+            provider: openshift
+          name: logging-eventrouter
+        spec:
+          serviceAccount: aggregated-logging-eventrouter
+          serviceAccountName: aggregated-logging-eventrouter
+{% if node_selector is iterable and node_selector | length > 0 %}
+      nodeSelector:
+{% for key, value in node_selector.iteritems() %}
+        {{ key }}: "{{ value }}"
+{% endfor %}
+{% endif %}
+          containers:
+            - name: kube-eventrouter
+              image: ${IMAGE}
+              imagePullPolicy: Always
+              resources:
+                limits:
+                  memory: ${MEMORY} 
+                  cpu: ${CPU}
+                requires:
+                  memory: ${MEMORY}
+              volumeMounts:
+              - name: config-volume
+                mountPath: /etc/eventrouter
+          volumes:
+            - name: config-volume
+              configMap:
+                name: logging-eventrouter
+  - kind: ClusterRoleBinding
+    apiVersion: v1
+    metadata:
+      name: event-reader-binding
+    subjects:
+    - kind: ServiceAccount
+      name: aggregated-logging-eventrouter
+      namespace: ${NAMESPACE}
+    roleRef:
+      kind: ClusterRole
+      name: event-reader
+
+parameters:
+  - name: SINK
+    displayName: Sink
+    value: stdout
+  - name: REPLICAS
+    displayName: Replicas
+    value: "1"
+  - name: IMAGE
+    displayName: Image
+    value: "docker.io/openshift/origin-logging-eventrouter:latest"
+  - name: MEMORY
+    displayName: Memory
+    value: "128Mi"
+  - name: CPU
+    displayName: CPU
+    value: "100m"
+  - name: NAMESPACE
+    displayName: Namespace
+    value: default
diff --git a/roles/openshift_logging_eventrouter/vars/main.yaml b/roles/openshift_logging_eventrouter/vars/main.yaml
new file mode 100644
index 000000000..bdf561fe3
--- /dev/null
+++ b/roles/openshift_logging_eventrouter/vars/main.yaml
@@ -0,0 +1,2 @@
+---
+__eventrouter_sinks: ["glog", "stdout"]
diff --git a/roles/openshift_logging_fluentd/templates/fluentd.j2 b/roles/openshift_logging_fluentd/templates/fluentd.j2
index 1c0d1089f..b64393831 100644
--- a/roles/openshift_logging_fluentd/templates/fluentd.j2
+++ b/roles/openshift_logging_fluentd/templates/fluentd.j2
@@ -118,6 +118,10 @@ spec:
         - name: "MUX_CLIENT_MODE"
           value: "{{ openshift_logging_mux_client_mode }}"
 {% endif %}
+{% if openshift_logging_install_eventrouter is defined and openshift_logging_install_eventrouter %}
+        - name: "TRANSFORM_EVENTS"
+          value: "true"
+{% endif %}
       volumes:
       - name: runlogjournal
         hostPath: