The mirrord Operator can produce logs in JSON format that can be digested by most popular logging solutions (DataDog, Dynatrace, etc). To enable JSON logging, set operator.jsonLog to true in the Operator Helm chart values. The log level is INFO by default, and can be changed using the RUST_LOG environment variable in the Operator container, which takes values in the following format: mirrord={log_level} (e.g. mirrord=debug).

This feature is only relevant for users on the Team and Enterprise pricing plans.

Functional Logs

The following logs are written with log level INFO, and can be used for dashboards within monitoring solutions in order to monitor mirrord usage within your organization:

Log messages:

• Target Copied

• Port Stolen

• Port Mirrored

• Port Released

• Session Started

• Session Ended

Fields:

Prometheus

mirrord Operator can expose prometheus metrics if enabled. (default endpoint is :9000/metrics)

Helm

# values.yaml for mirrord-operator helm chart
operator:
  ...
  metrics: true
  ...

Manual

Exposed metrics

DataDog Dashboard

We offer a DataDog dashboard you can import to track statistics.

Download it here

Grafana Dashboard

Alternatively there is a Grafana dashboard you can import to track statistics.

Download it here

fluentd

If you are using fluentd you can add a filter to unpack some values from the "log" message

<filter kubernetes.var.log.containers.**_mirrord_mirrord-operator-**>
  @type parser
  key_name log
  reserve_data true
  remove_key_name_field true
  <parse>
    @type json
  </parse>
</filter>

This will expand all the extra fields stored in "log" field.

fluentd + Elasticsearch

Assuming you are using logstash_format true and the connected mapping will store the extra fields in a keyword type, we have a ready made dashboard you can simply import.

Download it here (use Saved Objects to import).

The license server enables you to manage your organization’s seats without sending any data to mirrord’s servers. It can aggregate license metrics from multiple operators (useful if you’re running mirrord across multiple clusters) and provides visibility into seat usage across your organization. This feature is only relevant for users on the Team and Enterprise pricing plans.

Basic Setup

The license server is installable via Helm. First, add the MetalBear Helm repository:

helm repo add metalbear-co https://metalbear-co.github.io/charts

Next, save the following yaml as values.yaml on your machine.

# ./values.yaml
createNamespace: true

service:
  type: ClusterIP

license:
  key: secret
  file:
    data:
      license.pem: |
        ----- ... 
        MIRRORD-LICENSE 
        ... -----

Fill in the license.key and license.pem fields according to the following guidelines:

• License key - Can be any string of your choosing. We recommend using random characters or a UUID.

• License file - Must be a valid operator license. This can also be a secret under the license.pem key.

You can customize the license server deployment further - all values.yaml configuration options can found here

NOTE: The license server needs to be accessible to any mirrord operators you want to track. To that end, the default value for service.type is ClusterIP, but can be changed to NodePort or LoadBalancer, according to your requirements.

Next, install the license server on your cluster:

helm install metalbear-co/mirrord-operator-license-server -f ./values.yaml --generate-name --wait

To make sure it's been installed successfully and is running:

kubectl get deployment -n mirrord mirrord-license-server

If your operator(s) are running at on a different cluster, make sure the mirrord-operator-license-server service is exposed to them via ingress.

Connecting Operators to the License Server

First update your operator values.yaml for quickstart helm setup for operator) file:

# ./values.yaml
license:
  key: secret
  licenseServer: http://<license-server-addr>

NOTE: The server value must contain the protocol and the prefix for any ingress that the the license server can be behind.

Then run:

helm install metalbear-co/mirrord-operator -f ./values.yaml --generate-name --wait

The installation of the mirrord operator defines a new clusterwide in your cluster, called MirrordProfile.

This feature is only relevant for users on the Team and Enterprise pricing plans.

This resource can be used to provide mirrord users with a unified base for their mirrord configs. Users can reference an available profile in their configs, and they will be modified accordingly.

The complete list of allowed values for the featureAdjustments.[].change field is as follows:

1. incoming-mirror - incoming traffic will be mirrored

2. incoming-steal - incoming traffic will be stolen

3. incoming-off - incoming traffic will not be intercepted

4. dns-remote - all DNS resolution will be remote

5. dns-off - all DNS resolution will be local

6. outgoing-remote - all outgoing traffic will be remote

7. outgoing-off - all outgoing traffic will be local

Selecting a profile

Starting from mirrord version 3.136.0, the user can select a mirrord profile in their mirrord config. The profile is referenced by its name.

Enforcing profiles

Use of mirrord profiles can be enforced with .

Important: mirrord profiles are applied to the session on the user machine, and should not be used as security features.

With the mirrord Jira integration you can track how much mirrord has been used for each task on a per-issue basis, both as total time of use and the number of times it was used.

The operator is able to report mirrord session times to the mirrord app installed on your Jira instance, which allows you to view the total time and number of sessions that developers have spent using mirrord for each Jira issue.

Setting up Jira integration

1. Go to the installation link to install the mirrord for Jira app (you must be a Jira admin).

2. In Jira, go to the mirrord admin page under Jira admin settings > Apps > mirrord by MetalBear and choose which projects to display the panel on. If you skip this step, the metrics will not be displayed on any issues in your Jira instance.

3. Update the operator according to the instructions on the admin page.

4. Ensure everything is up to date (mirrord IDE plugins and mirrord binary version, as well as the CLI tool):

To verify that the app in installed properly, navigate to an issue in a project you selected in step 2 and check for the mirrord by MetalBear context panel in the right sidebar.

Using the Jira integration

When viewing an issue in Jira, the number of sessions and total session duration will be shown in a context panel in the right sidebar called mirrord by MetalBear. This panel will be displayed on all issues belonging to the projects that were selected in the admin page.

To use the Jira integration while running mirrord, users must be on a git branch containing the (case sensitive) Jira issue key of the issue they're working on, eg. my-new-branch-KEY-123-latest for issue KEY-123.

Troubleshooting

The operator will emit logs with details upon successful session reporting, or upon encountering an error. When metrics are successfully reported, the operator will emit a DEBUG log with a link to the Jira issue that was updated. If the operator is up to date with the Jira webhook configured but did not successfully report metrics:

• If no branch name was recieved by the operator, a DEBUG log will be emitted. This can happen if a mirrord user is not using the latest version of the mirrord CLI or plugin, or if they are not currently on a git branch.

• If the branch name was present but the operator still fails to report metric to the Jira app, a WARN log will be emitted with more details.

Data security

When a session ends, the session data is sent by the operator via HTTP request to a Jira Forge webtrigger URL and is not stored by the operator.

The Jira app stores usage data for each issue in encrypted key-value storage. It is not visible to anyone outside those with access to the current Jira instance; the only way to access the data is through the context panel on an issue.

Known Issues

• Metrics reporting will not work for jj users, as jj operates in detached HEAD mode.

• Fetching the user's branch name may rarely be unreliable with the IntelliJ plugin, causing metrics reporting to be skipped. In this case you can use the CLI instead, and in the future users will be able to manually override the branch name in config.

This discussion is only relevant for users on the Team and Enterprise pricing plans.

I'm a Security Engineer evaluating mirrord for Teams, what do I need to know?

• mirrord for Teams is completely on-prem. The only data sent to our cloud is analytics and license verification which can be customized or disabled upon request. The analytics don't contain PII or any sensitive information.

• mirrord for Teams uses Kubernetes RBAC, meaning it doesn't add a new attack vector to your cluster.

• The Kubernetes operator installed in the cluster as part of mirrord for Teams is licensed as Source Available (but not yet public) and we'll be happy to share the code if needed for review.

• mirrord for Teams defines a new CRD that can be used to limit access and use of mirrord, with plans of more fine-grained permissions in the future.

• The operator requires permissions to create a pod with the following capabilities in its Kubernetes namespace:

  • CAP_NET_ADMIN - for modifying routing tables

  • CAP_SYS_PTRACE - for reading the target pod's environment variables

  • CAP_SYS_ADMIN - for joining the target pod's network namespace

• The operator requires exclusions from the following gatekeeper policies:

  • runAsNonRoot - to access target pod's filesystem

  • HostPath volume/Sharing the host namespace - to access target pod's file system and networking

• mirrord doesn't copy remote files or secrets to the local filesystem. The local app only gets access to remote files and secrets in memory, and so they'll only be written to the local filesystem if done by the local app, or if mirrord was explicitly configured to log to files with a log level of debug/trace.

• Missing anything? Feel free to ask us on Discord or [email protected]

Are you SOC2/GDPR compliant?

mirrord for Teams is completely on-prem and doesn't process your customer data, so SOC2 and GDPR don't apply to it.

How do I configure Role Based Access Control for mirrord for Teams?

mirrord for Teams works on top of Kubernetes' built-in RBAC with the following resources, mirrordoperators, mirrordoperators/certificate, targets, and targets/port-locks under the operator.metalbear.co apiGroup. The first two resources are required at a cluster level, and the last two can be allowed at a namespace level.

You can limit a user's ability to use mirrord on specific targets by limiting their access to the target resource. The specific verbs for rules to our resources can be copied from the examples below.

For your convenience, mirrord for Teams includes a built-in ClusterRole called mirrord-operator-user, which controls access to the Operator API. To grant access to the Operator API, you can create a ClusterRoleBinding like this:

In addition, the Operator impersonates any user that calls its API, and thus only operates on pods or deployments for which the user has get permissions.

To see the latest definition, we recommend checking our .

How do I limit user access to a specific namespace?

Create a ClusterRoleBinding between the user and the mirrord-operator-user-basic role, then create a (easiest via Helm chart by specifying roleNamespaces) and bind create RoleBinding in the namespace.

How do I limit user access to a specific target?

If the user doesn't have get access to the targets, then they won't be able to target them with mirrord. However, if you want to allow get access to targets but disallow using mirrord on them, we recommend creating a new role based on the mirrord-operator-user namespaced role above, and adding a resourceNames field to the targets resource. This will limit the user to only using the Operator on the specified targets. For example:

How can I prevent users in my team from stealing or mirroring traffic from a target?

You can define that prevent stealing (or only prevent stealing without setting a filter) and/or mirroring for selected targets. Let us know if there are more features you would like to be able to limit using policies.

How can I prevent users from using mirrord without going through the Operator?

When the mirrord CLI starts, it checks if an Operator is installed in the cluster and uses it if it's available. However, if the user lacks access to the Operator or if the Operator doesn't exist, mirrord attempts to create an agent directly.

To prevent clients from attempting to create an agent without the Operator, you can add the to the mirrord configuration file:

To prevent mirrord clients from directly creating agents at the cluster level, we recommend disallowing the creation of pods with extra capabilities by using . Apply a baseline or stricter policy to all namespaces while excluding the mirrord namespace.

Note: before adding a new Pod Admission Policy, you should make sure it doesn't limit any functionality required by your existing workloads.

By default the in-cluster traffic between the operator and its agents isn't encrypted nor authenticated. To ensure encryption and authentication you can enable TLS protocol for the operator–agent connections. You can do this in the operator by setting agent.tls to true or manually by setting OPERATOR_AGENT_CONNECTION_TLS=true in the operator container environment. TLS connections are supported from agent version 3.97.0.

Security hardening with the mirrord operator

Here is a quick checklist you may wish to follow in order to improve the security posture of your cluster when using the operator:

Enabling TLS

TLS can be enabled between the operator and mirrord agents to encrypt the traffic they send to each other. From the :

By default the in-cluster traffic between the operator and its agents isn’t encrypted nor authenticated. To ensure encryption and authentication you can enable TLS protocol for the operator–agent connections. You can do this in the operator by setting agent.tls to true or manually by setting OPERATOR_AGENT_CONNECTION_TLS=true in the operator container environment. TLS connections are supported from agent version 3.97.0.

Reducing access to the mirrord namespace

Users have no need to access to the namespace where mirrord resources are created. By default, this is the mirrord namespace.

Using a certificate for mirrord APIService

By using either your own certificate or one provided by a certificate manager, you can secure access to mirrord's APIService - you will need to set insecureSkipTLSVerify to false in the mirrord-operator Helm chart.

NB: If you are using a certificate manager, make sure you set up reminders for certificate renewal.

Set up network policies for communication

Access to the operator can be further restricted by setting up in the cluster to limit the operator to communicate only with mirrord agents (this is not possible if running agents in ).

The installation of the mirrord operator defines two custom resources in your cluster: the namespaced MirrordPolicy and the clusterwide MirrordClusterPolicy. With these policies you can limit the use of some features of mirrord for selected targets.

• MirrordPolicy and MirrordClusterPolicy have the exact same specification (spec field);

• MirrordPolicy applies only to targets living in the same namespace;

• MirrordClusterPolicy applies to all targets in the cluster.

This feature is only relevant for users on the Team and Enterprise pricing plans.

Blockable features

Currently the set of blockable features contains:

• steal - prevents stealing traffic from the targeted pods;

• steal-without-filter - prevents stealing traffic from the targeted pods, unless HTTP filter is used;

• mirror - prevents mirroring traffic from the targeted pods.

If you are not using the latest operator version, the set of supported blockable features might be different. In order to see the exact set of features you can block, use the following kubectl command:

kubectl get crd mirrordpolicies.policies.mirrord.metalbear.co -o jsonpath='{.spec.versions[-1].schema.openAPIV3Schema.properties.spec.properties.block.items.enum}'

Controllable features

Some policies are not for outright blocking features, instead they change behaviour, overriding what the user has set in their mirrord config file.

You may use these features to change which files may be accessed in the target, or which environment variables may be retrieved. These policies should not be relied upon for security, and should instead be considered convenience policies.

env policy

Added in mirrord Operator version 3.103.0

Changes how environment variables may be retrieved from the target, overriding what the user has set in their mirrord.json config file.

• exclude - the environment variables in this list WON'T be retrieved from the target, instead mirrord will either use the locally available env vars (if they exist in the user's machine), or these env vars will be missing completely;

The policy takes priority over a user's mirrord config, which means that if the user has a config:

{ "feature": { "env": { "include": "*_URL" } } }

If the policy is set with exclude: ["*_URL"], then mirror will NOT retrieve env vars that match *_URL, even though the user explicitly wanted that in their config.

If you are not using the latest operator version, the env policy options might be different. In order to see the latest options, use the following kubectl command:

kubectl get crd mirrordpolicies.policies.mirrord.metalbear.co -o jsonpath='{.spec.versions[-1].schema.openAPIV3Schema.properties.spec.properties.env}'

fs policy

Added in mirrord Operator version 3.103.0

Changes file operations behaviour, giving the operator control over which files may be accessed from the target, and in which modes. Overrides what the user has set in their mirrord.json config file.

• readOnly - files that match any of the patterns specified here must be opened as read-only, otherwise the operation will fail;

• local - matching files will be forced to be opened locally, on the user's machine, instead of in the target;

• notFound - any matching files will return a not found error as if the file is not present in the target, even if it exists there;

The policy takes priority over a user's mirrord config, which means that if the user has a config:

{ "feature": { "fs": { "read_write": ".+\\.json" } } }

If the policy is set with readOnly: [".+\\.json"], and the user tries to open a file that matches this regex in write mode, then mirrord will return an error to the user app, as if the file could not be found, even though the user wanted it to be read_write.

kubectl get crd mirrordpolicies.policies.mirrord.metalbear.co -o jsonpath='{.spec.versions[-1].schema.openAPIV3Schema.properties.spec.properties.fs}'

network policy

Added in mirrord Operator version 3.105.0

Allows the operator to control which patterns may be used as HTTP header filters. Header filters specified by the user must match the regex specified in the network policy.

apiVersion: policies.mirrord.metalbear.co/v1alpha
kind: MirrordPolicy
metadata: { ... }
spec:
  ...
  network:
    incoming:
      httpFilter:
        headerFilter: "^username: .+"

If the policy is set with headerFilter: "^username: .+" at least one header filter must match the ^username: .+ regex when user is using the steal mode for incoming traffic.

{ "feature": { "network": { "incoming": { "http_filter": { "header_filter": "username: foobar" } } } } }

this also works any of or all of patterns

{ "feature": { "network": { "incoming": { "http_filter": { "all_of": [
  { "header": "username: foobar" },
  { "path": "/api.*" }
] } } } } }

{ "feature": { "network": { "incoming": { "http_filter": { "any_of": [
  { "header": "username: foobar" },
  { "header": "username: baz2000" }
] } } } } }

Important: steal-without-filter will be automatically enabled once any http filter is specified.

profile policy

Added in mirrord Operator version 3.108.0

Allows the operator to enforce using a mirrord profile and to specify a set of allowed profiles.

apiVersion: policies.mirrord.metalbear.co/v1alpha
kind: MirrordPolicy
metadata: { ... }
spec:
  ...
  # If this is set, the user must select a mirrord profile for their session.
  #
  # If multiple policies apply to the given session,
  # a profile is required if at least one of them require it.
  #
  # Optional, defaults to false.
  requireProfile: true
  # A list of allowed mirrord profiles.
  #
  # If multiple policies apply to the given session,
  # user's selected profile must be present in all allowlists.
  #
  # Optional. If not present, this policy will not enforce any allowlist.
  profileAllowlist:
  - my-profile-1
  - my-profile-2

The example above will enforce that the user selects either my-profile-1 or my-profile-2 for their session.

Important: mirrord profiles are applied to the session on the user machine, and should not be used as security features.

Restricting targets affected by mirrord policies

By default, mirrord policies apply to all targets in the namespace or cluster. You can use a target path pattern (.spec.targetPath) and/or a label selector (.spec.selector) in order to limit the targets to which a policy applies.

The target path of a mirrord run is either targetless or has the form <TARGET_TYPE>/<NAME> followed by an optional /container/<CONTAINER_NAME>, where <TARGET_TYPE> is one of deploy, pod, rollout and statefulset.

Examples for possible target paths:

• deploy/boats

• pod/boats-5fffb9767c-w92qh

• pod/boats-5fffb9767c-w92qh/container/appcontainer

• targetless

By specifying a targetPath pattern in the policy, you limit the policy to only apply to runs that have a target path that matches the specified pattern.

The target path pattern can contain ?, which will match a single character, and *, which will match arbitrarily many characters. For example, "deploy/*" will make a policy apply for any run with a deployment target. "*boats*" will make a policy apply to any target with boats in its name, e.g. pod/boats-2kljw9, pod/whatever-23oije2/container/boats-container, etc.

Note: when mirrord user specifies a container for the mirrord run, the target path ends with /container/<CONTAINER_NAME>.

This means the pattern deploy/my-deployment will not match when a container is specified. That pattern can be changed to deploy/my-deployment* to also match on runs with a specified container (but will then also match deploy/my-deployment-1 etc.)

Please note that the policy is applied according to the target given to mirrord. It is possible for a policy to apply to a deployment target, but not to apply to the deployment's pods when targeted directly. For example, the following policy:

apiVersion: policies.mirrord.metalbear.co/v1alpha
kind: MirrordPolicy
metadata:
  name: block-stealing-from-boats-deployment
  namespace: default
spec:
  targetPath: "deploy/boats*"
  block:
    - steal

prevents mirrord users from stealing traffic when using the whole boats deployment as a target. However, a user could still use a specific pod out of that deployment as a target for mirrord and steal its traffic. In order to prevent that, the targetPath pattern or the label selector needs to be changed to match the pods of that deployment.

If a workload is used as a target, this workload's labels will be used to match against policies' selector, if set. If a pod is used as a target, the pod's labels will be used.

Another example of a policy:

apiVersion: policies.mirrord.metalbear.co/v1alpha
kind: MirrordPolicy
metadata:
  name: block-unfiltered-stealing-from-webserver-deployments
  namespace: books
spec:
  targetPath: "deploy/*"
  selector:
    matchLabels:
      component: webserver
  block:
    - steal-without-filter
    - mirror

This policy blocks mirroring and unfiltered stealing of traffic coming to all deployments in the namespace books which are marked with label component: webserver.

This section covers how to manage and administer mirrord in your organization or team. Here, you'll find information on topics such as setting up and using the License Server, monitoring mirrord usage and performance, configuring policies to control access and behavior, working with profiles for different environments or teams, and understanding security considerations. These guides are intended for administrators and advanced users who want to ensure mirrord is deployed, maintained, and governed effectively in production or shared environments.