Stackdriver Kubernetes Monitoring
Scenario Context - 1
We want to monitor not only the platform our services are running on, but also our services themselves. Plus setting up alerts when certain errors or spike happens.
Tutorial to Follow
Google Cloud’s default Stackdriver K8s Monitoring
Steps to do:
- Prerequisites
- Role: roles/owner
- Kubernetes version: v1.10.2 or later
- Create K8s cluster with beta Stackdriver Monitoring
- Using Prometheus as monitoring tool
Issues:
Reference:
Scenario Context - 2
We want to automate the alert policy creation. Right now all the alerting policy in Stackdriver were manually created. We want to apply the policies to new environment programmatically next time. Therefore we are considerring some scripts to do the work.
Tutorial to Follow
GCP supports Managing Alerting Policies by API Managing Notification Channels by API
Steps to do:
- gcloud CLI?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
1. Creating policies
```jshelllanguage
gcloud alpha monitoring policies create --policy-from-file="rising-cpu-usage.yaml"
```
2. Retrieving policies
```jshelllanguage
gcloud alpha monitoring policies list
```
3. Deleting policies
```jshelllanguage
gcloud alpha monitoring policies delete projects/a-gcp-project/alertPolicies/12669073143329903307
```
4. Modifying policies
```jshelllanguage
# Disable policy
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 --no-enabled
# Update policy with notification channel
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 \
--add-notification-channels="projects/a-gcp-project/notificationChannels/1355376463305411567"
# Update policy documents
gcloud alpha monitoring policies update projects/a-gcp-project/alertPolicies/12669073143329903307 \
--documentation-format="text/markdown" \
--documentation-from-file="cpu-usage-doc.md"
```
Issues:
Reference: