Monitoring

Monitoring #

The mirrord Operator can produce logs in JSON format that can be digested by most popular logging solutions (DataDog, Dynatrace, etc). To enable JSON logging, set operator.jsonLog to true in the Operator Helm chart values. The log level is INFO by default, and can be changed using the RUST_LOG environment variable in the Operator container, which takes values in the following format: mirrord={log_level} (e.g. mirrord=debug).

Functional Logs #

The following logs are written with log level INFO, and can be used for dashboards within monitoring solutions in order to monitor mirrord usage within your organization:

Log messages:

  • Target Copied
  • Port Stolen
  • Port Mirrored
  • Port Released
  • Session Started
  • Session Ended

Fields:

field description events
target the session’s target All
client_hostname whoami::hostname of client All
client_name whoami::realname of client All
client_user Kubernetes user of client (via k8s RBAC) All
client_id unique client id produced from client’s certificate All
session_id unique id for individual mirrord sessions Port Steal Port Mirrored Port Released Session Started Session Ended
session_duration the session’s duration in seconds Session Ended
port port number Port Stolen Port Mirrored Port Released
http_filter the client’s configured HTTP Filter Port Stolen
scale_down whether the session’s target was scaled down Target Copied

Prometheus #

mirrord Operator can expose prometheus metrics if enabled. (default endpoint is :9000/metrics)

Helm #

# values.yaml for mirrord-operator helm chart
operator:
  ...
  metrics: true
  ...

Manual #

env description type default
OPERATOR_METRICS_ENABLED enable metrics endpoint “true” | “false” “false”
OPERATOR_METRICS_ADDR metrics http server addr SocketAddr “0.0.0.0:9000”

Exposed metrics #

metric description labels
mirrord_license_valid_seconds Seconds left for current license validity
mirrord_sessions_create_total Count of created sessions client_hostname client_name client_user user_id
mirrord_sessions_duration Histogram for session durations after they are ended client_hostname client_name client_user user_id

DataDog Dashboard #

We offer a DataDog dashboard you can import to track statistics.

Download it here

Grafana Dashboard #

Alternatively there is a Grafana dashboard you can import to track statistics.

Download it here

fluentd #

If you are using fluentd you can add a filter to unpack some values from the “log” message

<filter kubernetes.var.log.containers.**_mirrord_mirrord-operator-**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

This will expand all the extra fields stored in “log” field.

fluentd + Elasticsearch #

Assuming you are using logstash_format true and the connected mapping will store the extra fields in a keyword type, we have a ready made dashboard you can simply import.

Download it here (use Saved Objects to import).