This section provides in-depth technical reference for mirrord. Here you'll find detailed documentation on configuration options, supported targets, third-party integrations, network traffic handling, architecture, environment variables, and file operations. Use this section as a comprehensive resource for understanding and customizing mirrord's behavior in your environment.

Overview

mirrord lets you run a local process in the context of remote environment i.e. environment variables present in the remote pod will be loaded into the local process.

For example, if you want your local process to access a remote database, the connection string configured in the remote pod's environment variable can be used by your local process.

How does it work?

mirrord-layer sends a message to mirrord-agent requesting remote environment variables, which are then set before the local process starts.

Usage

To include/exclude environment variables selectively, use the --override-env-vars-include flag to include and --override-env-vars-exclude to exclude with environment variables specified in a semicolon separated list.

Note: These flags are mutually exclusive. For example, if one chooses to exclude using the --override-env-vars-exclude flag, then there is no need to use --override-env-vars-include="*" to include all other environment variables.

By default, all environment variables are included.

Example

If on our target pod, we have the environment variable ENV_VAR1 with the value remote-value and on our local machine we have ENV_VAR1 with value local-value, then Running the python interpreter with mirrord would look like this:

MIRRORD_AGENT_IMAGE=test MIRRORD_AGENT_RUST_LOG=trace RUST_LOG=debug target/debug/mirrord exec -c --target pod/py-serv-deployment-ff89b5974-x9tjx python3

Python 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:23:25)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.environ['ENV_VAR1'])
remote-value

Logs

❯ MIRRORD_AGENT_IMAGE=test MIRRORD_AGENT_RUST_LOG=trace RUST_LOG=debug target/debug/mirrord exec -c --target pod/py-serv-deployment-ff89b5974-x9tjx python3
...
2022-07-01T17:18:33.744996Z DEBUG mirrord_layer: ClientMessage::GetEnvVarsRequest codec_result Ok(
    (),
)
2022-07-01T17:18:33.754270Z DEBUG mirrord_layer: DaemonMessage::GetEnvVarsResponse Ok(
    {
        "KUBERNETES_PORT": "tcp://10.96.0.1:443",
        "LANG": "C.UTF-8",
        "KUBERNETES_SERVICE_PORT": "443",
        "PY_SERV_PORT": "tcp://10.96.139.36:80",
        "KUBERNETES_PORT_443_TCP": "tcp://10.96.0.1:443",
        "PY_SERV_SERVICE_PORT": "80",
        "KUBERNETES_SERVICE_PORT_HTTPS": "443",
        "PYTHON_SETUPTOOLS_VERSION": "58.1.0",
        "PY_SERV_PORT_80_TCP_ADDR": "10.96.139.36",
        "PYTHON_GET_PIP_SHA256": "ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d",
        "ENV_VAR1": "remote-value",
        "KUBERNETES_SERVICE_HOST": "10.96.0.1",
        "KUBERNETES_PORT_443_TCP_PORT": "443",
        "HOSTNAME": "py-serv-deployment-ff89b5974-x9tjx",
        "KUBERNETES_PORT_443_TCP_ADDR": "10.96.0.1",
        "GPG_KEY": "E3FF2839C048B25C084DEBE9B26995E310250568",
        "PYTHON_GET_PIP_URL": "https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py",
        "PY_SERV_PORT_80_TCP": "tcp://10.96.139.36:80",
        "KUBERNETES_PORT_443_TCP_PROTO": "tcp",
        "PYTHON_VERSION": "3.9.13",
        "PY_SERV_PORT_80_TCP_PROTO": "tcp",
        "PY_SERV_PORT_80_TCP_PORT": "80",
        "PY_SERV_SERVICE_HOST": "10.96.139.36",
        "PYTHON_PIP_VERSION": "22.0.4",
    },
)!
...

mirrord is composed of the following components:

• mirrord-agent - Rust binary that is packaged as a container image. mirrord-agent runs in the cloud and acts as a proxy for the local process.

• mirrord-layer - Rust dynamic library (so/dylib) that loads to the local process, hooks its filesystem, network APIs and relays them to the agent.

• mirrord-cli - Rust binary that wraps the behavior of mirrord-layer in a user friendly CLI.

• VS Code extension - Exposes the same functionality as - mirrord-cli within the VS Code IDE.

• IntelliJ plugin - Exposes the same functionality as - mirrord-cli within the IntelliJ IDEs.

mirrord-agent

mirrord-agent is a Kubernetes job that runs in the same Linux namespace as the pod being impersonated in the cluster. This lets the mirrored-agent sniff the network traffic and gain access to the filesystem of the impersonated pod. It then relays file operations from the local process to the impersonated pod and incoming traffic from the impersonated pod to the local process. Outgoing traffic is intercepted at the local process and emitted by the agent as if originating from the impersonated pod. The connection between the agent and the impersonated pod is terminated if the agent pod hits a timeout.

mirrord-agent does not run as a privileged container in the cluster. However, it requires some Linux capabilities to be able to impersonate the targeted pod. These capabilities are:

• CAP_NET_ADMIN and CAP_NET_RAW - required for modifying routing tables

• CAP_SYS_PTRACE - required for reading target pod environment

• CAP_SYS_ADMIN - required for joining target pod network namespace

However, you can disable any subset of those in the configuration using agent.disabled_capabilities option. This will possibly limit mirrord functionalities or even make it unusable in some setups.

mirrord-layer

mirrord-layer is a .dylib file for OSX systems and .so file on Linux distributions. mirrord-layer is loaded through LD_PRELOAD/DYLD_INSERT_LIBRARIES environment variable with the local process, which lets mirrord-layer selectively override libc functions. The overridden functions are then responsible for maintaining coordination between the process and incoming/outgoing requests for network traffic/file access. mirrord-layer sends and receives events from the agent using port-forwarding.

mirrord-cli

mirrord-cli is a user friendly interface over the essential functionality provided by mirrord-layer. When you run mirrord-cli, it runs the process provided as an argument with mirrord-layer loaded into it.

VS Code Extension

mirrord’s VS Code extension provides mirrord’s functionality within VS Code’s UI. When you debug a process with mirrord enabled in VS Code, it prompts you for a pod to impersonate, then runs the debugged process with mirrord-layer loaded into it.

IntelliJ Plugin

mirrord’s IntelliJ Plugin provides mirrord’s functionality within the IntelliJ UI. When you debug a process with mirrord enabled in IntelliJ, it prompts you for a pod to impersonate, then runs the debugged process with mirrord-layer loaded into it.

List of third-party guides and videos about mirrord:

Overview

mirrord will relay file access (except for some exceptions) to the target pod by default. (this functionality can be disabled using --fs-mode local flag on the command line or by setting mode in the configuration file in the IDE plugin.)

For example, the following python script calls the built-in open function which translate to something like openat(AT_FDCWD, "/tmp/test", O_RDWR|O_CLOEXEC) at a lower level:

with open("/tmp/test", "r+") as rw_file:
    read_str = rw_file.read(42)
print(read_str)

When we run that python script with mirrord:

mirrord exec -c --target py-serv-deployment-cfc458fd4-bjzjx python3 test.py

mirrord overrides that openat call and opens /tmp/test on the remote pod.

How does it work?

Once a request to open a new file is received by mirrord-agent from mirrord-layer, the agent forwards the request to the container in the remote pod in context of the provided path for the open system call, prefixed with path to the root directory of the container.

mirrord-agent uses APIs provided by docker and containerd runtimes to get the PID of the remote container, and refers to the root directory of the remote container through /proc/container_pid/root

Syscalls

mirrord overrirdes calls to the following libc functions/system calls:

open

int open(const char *pathname, int flags);

Open files on the remote pod. Functionality when opening with different types of paths might differ. In the case when pathname is specified to be a relative path, the call to open is sent to libc instead of the remote pod.

Example:

import os
fd = os.open("/tmp/test", os.O_WRONLY | os.O_CREAT)

openat

int openat(int dirfd, const char *pathname, int flags);

openat works the same as open when dirfd is specified as AT_FDCWD or if the path is absolute. If a valid dirfd is provided, files relative to the directory referred to by the dirfd can be opened.

Example:

dir = os.open("/tmp", os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC | os.O_DIRECTORY)

os.open("test", os.O_RDWR | os.O_NONBLOCK | os.O_CLOEXEC, dir_fd=dir)

read

ssize_t read(int fd, void *buf, size_t count);

Read from a file on the remote pod.

Example:

fd = os.open("/tmp/test, os.O_RDWR | os.O_NONBLOCK | os.O_CLOEXEC)
read = os.read(fd, 1024)

write

ssize_t write(int fd, const void *buf, size_t count);

Write to a file on the remote pod.

Example:

with open("/tmp/test", "w") as file:
    file.write(TEXT)

lseek

off_t lseek(int fd, off_t offset, int whence);

Reposition the file offset of an open file on the remote pod. lseek through mirrord-layer supports all valid options for whence as specified in the Linux manpages.

Example:

with open("/tmp/test", "w") as file:
    file.seek(10)
    file.write(TEXT)

Note: For read, write, and lseek if the provided fd is a valid file descriptor i.e. it refers to a file opened on the remote pod then the call is forwarded to the remote pod, otherwise the call is sent to libc.

Incoming

mirrord lets users debug incoming network traffic by mirroring or stealing the traffic sent to the remote pod.

Mirroring

mirrord's default configuration is to mirror incoming TCP traffic from the remote pod, i.e. run the local process in the context of cloud environment without disrupting incoming traffic for the remote pod. Any responses by the local process to the mirrored requests are dropped, and so whatever application is running on the remote pod continues to operate normally while the traffic is mirrored to the local process.

Example - user-service a simple Kubernetes deployment and service that stores registered users.

bigbear@metalbear:~/mirrord$ minikube service list
|-------------|-------------------|--------------|---------------------------|
|  NAMESPACE  |       NAME        | TARGET PORT  |            URL            |
|-------------|-------------------|--------------|---------------------------|
| default     | kubernetes        | No node port |
| default     | user-service      |           80 | http://192.168.49.2:32000 |
| kube-system | kube-dns          | No node port |
|-------------|-------------------|--------------|---------------------------|

bigbear@metalbear:~/mirrord-demo$ curl -X POST -H "Content-type: application/json" -d "{\"Name\" : \"Metal\", \"Last\" : \"Bear\"}" http://192.168.49.2:32000/user
{"Last":"Bear","Name":"Metal"}

bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:31000/index.html
<html> <head>USERS</head><body><h1> MetalBear Users</h1><p>[{"Last":"Bear","Name":"Metal"}]</p></body></html>

bigbear@metalbear:~/mirrord$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS      AGE
metalbear-bff-deployment-597cb4f957-485t5   1/1     Running   1 (15h ago)   16h
metalbear-deployment-85c754c75f-6k7mg       1/1     Running   1 (15h ago)   16h

To mirror traffic from remote services to the local development environment, run the services locally with mirrord

• Serving Flask app 'service' (lazy loading)

• Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

• Debug mode: off

• Running on all addresses (0.0.0.0) WARNING: This is a development server. Do not use it in a production deployment.

• Running on http://127.0.0.1:33695

• Running on http://172.16.0.4:33695 (Press CTRL+C to quit) 127.0.0.1 - - [08/Sep/2022 15:34:34] "GET /users HTTP/1.1" 200 // ^ Received mirrored traffic from the remote pod | |

  Window 2

  Copy

  bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
  [{"Last":"Bear","Name":"Metal"}]

  |

Stealing

mirrord can steal network traffic, i.e. intercept it and send it to the local process instead of the remote pod. This means that all incoming traffic is only handled by the local process.

Example - running user-service with mirrord and --tcp-steal on:

• Serving Flask app 'service' (lazy loading)

• Environment: production WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.

• Debug mode: off

• Running on all addresses (0.0.0.0) WARNING: This is a development server. Do not use it in a production deployment.

• Running on http://127.0.0.1:35215

• Running on http://172.16.0.4:35215 (Press CTRL+C to quit) 127.0.0.1 - - [08/Sep/2022 15:48:40] "GET /users HTTP/1.1" 200 - 127.0.0.1 - - [08/Sep/2022 15:50:40] "POST /user HTTP/1.1" 200 - 127.0.0.1 - - [08/Sep/2022 15:50:55] "GET /users HTTP/1.1" 200 - 127.0.0.1 - - [08/Sep/2022 16:57:51] "POST /user HTTP/1.1" 200 - 127.0.0.1 - - [08/Sep/2022 16:57:54] "GET /users HTTP/1.1" 200 - ^Cbigbear@metalbear:/mirrord-demo$ | |Window 2// Before running mirrord with --tcp-stealbigbear@metalbear:/mirrord-demo$ curl http://192.168.49.2:32000/users[{"Last":"Bear","Name":"Metal"}]

// After running with mirrord and --tcp-steal - local process responds instead of the remote bigbear@metalbear:/mirrord-demo$ curl http://192.168.49.2:32000/users [] bigbear@metalbear:/mirrord-demo$ curl -X POST -H "Content-type: application/json" -d "{"Name" : "Mehul", "Last" : "Arora"}" http://192.168.49.2:32000/user{"Last":"Arora","Name":"Mehul"} bigbear@metalbear:/mirrord-demo$ curl http://192.168.49.2:32000/users [{"Last":"Arora","Name":"Mehul"}] bigbear@metalbear:/mirrord-demo$ curl -X POST -H "Content-type: application/json" -d "{"Name" : "Alex", "Last" : "C"}" http://192.168.49.2:32000/user {"Last":"C","Name":"Alex"} bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users [{"Last":"Arora","Name":"Mehul"},{"Last":"C","Name":"Alex"}]

// After sending SIGINT to the local process bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users [{"Last":"Bear","Name":"Metal"}] |

Filtering Incoming Traffic by HTTP Headers

Currently only supported in steal mode: mirrord lets you specify a regular expression to filter HTTP requests with. When specified, all the headers of each HTTP request that arrives at the remote target are checked against the regular expression. If any of the headers match, the request will be stolen, otherwise, it will be sent to the remote target. For each Header-Name, Header-Value pair, your regular expression will be matched against Header-Name: Header-Value. For example, the filter MyHeader would match requests with MyHeader in any of their header names or header values. The filter ^X-MyFilter: would match only requests that have a header with the header name X-MyFilter (or x-myfilter or with any other capitalization). The regular expression is evaluated with the fancy_regex rust crate.

Specifying a Filter

An HTTP filter can be specified in the mirrord configuration file by setting the incoming mode to steal and specifying a filter in feature.network.incoming.http_filter.header_filter or feature.network.incoming.http_filter.path_filter.

Setting Custom HTTP Ports

The configuration also allows specifying custom HTTP ports under feature.network.incoming.http_filter.ports. By default, ports 80 and 8080 are used as HTTP ports if a filter is specified, which means that the mirrord agent checks each new connection on those ports for HTTP, and if the connection has valid HTTP messages, they are filtered with the header filter.

Outgoing

mirrord's outgoing traffic feature intercepts outgoing requests from the local process and sends them through the remote pod instead. Responses are then routed back to the local process. A simple use case of this feature is enabling the local process to make an API call to another service in the k8s cluster, for example, a database read/write.

For UDP, outgoing traffic is currently only intercepted and forwarded by mirrord if the application binds a non-0 port and makes a connect call on the socket before sending out messages. Outgoing TCP and UDP forwarding are both enabled by default. It can be controlled individually for TCP and UDP or disabled altogether (see mirrord exec --help).

Note: If the handling of incoming requests by your app involves outgoing API calls to other services, and mirrord is configured to mirror incoming traffic, then it might be the case that both the remote pod and the local process (which receives mirrored requests) make an outgoing API call to another service for the same incoming request. If that call is a write operation to a database, this could lead e.g. to duplicate lines in the database. You can avoid such an effect by switching from traffic mirroring to traffic stealing mode. Alternatively, if the service your application makes an API call to is only reachable from within the Kubernetes cluster, you can disable outgoing traffic forwarding, which would make it impossible for your local process to reach that service.

DNS Resolution

mirrord can resolve DNS queries in the context of the remote pod

Example - calling getaddrinfo to see if the query is resolved:

Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getaddrinfo('localhost', None)
2022-09-08T17:37:50.735532Z  INFO mirrord_layer::socket::ops: getaddrinfo -> result Ok(
    0x00007f5508004760,
)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::7074:e00d:557f:0', 0, 0, 97)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('::', 0, 0, 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_RAW: 3>, 0, '', ('::90bf:f401:0:0', 0, 0, 245652448)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('127.0.0.1', 0))]
>>> socket.getaddrinfo('user-service', None)
2022-09-08T17:38:17.556108Z  INFO mirrord_layer::socket::ops: getaddrinfo -> result Ok(
    0x00007f5508003610,
)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('10.106.158.180', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('10.106.158.180', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('10.106.158.180', 0))]
>>> 

Configuration

mirrord allows for a high degree of customization when it comes to which features you want to enable, and how they should function.

All of the configuration fields have a default value, so a minimal configuration would be no configuration at all.

The configuration supports templating using the Tera template engine. Currently we don't provide additional values to the context, if you have anything you want us to provide please let us know.

To use a configuration file in the CLI, use the -f <CONFIG_PATH> flag. Or if using VSCode Extension or JetBrains plugin, simply create a .mirrord/mirrord.json file or use the UI.

To help you get started, here are examples of a basic configuration file, and a complete configuration file containing all fields.

Basic config.json

{
  "target": "pod/bear-pod",
  "feature": {
    "env": true,
    "fs": "read",
    "network": true
  }
}

Basic config.json with templating

{
  "target": "{{ get_env(name="TARGET", default="pod/fallback") }}",
  "feature": {
    "env": true,
    "fs": "read",
    "network": true
  }
}

Complete config.json

Don't use this example as a starting point, it's just here to show you all the available options.

{
  "accept_invalid_certificates": false,
  "skip_processes": "ide-debugger",
  "target": {
    "path": "pod/bear-pod",
    "namespace": "default"
  },
  "connect_tcp": null,
  "agent": {
    "log_level": "info",
    "json_log": false,
    "labels": { "user": "meow" },
    "annotations": { "cats.io/inject": "enabled" },
    "namespace": "default",
    "image": "ghcr.io/metalbear-co/mirrord:latest",
    "image_pull_policy": "IfNotPresent",
    "image_pull_secrets": [ { "secret-key": "secret" } ],
    "ttl": 30,
    "ephemeral": false,
    "communication_timeout": 30,
    "startup_timeout": 360,
    "network_interface": "eth0",
    "flush_connections": true,
    "metrics": "0.0.0.0:9000",
  },
  "feature": {
    "env": {
      "include": "DATABASE_USER;PUBLIC_ENV",
      "exclude": "DATABASE_PASSWORD;SECRET_ENV",
      "override": {
        "DATABASE_CONNECTION": "db://localhost:7777/my-db",
        "LOCAL_BEAR": "panda"
      },
      "mapping": {
        ".+_TIMEOUT": "1000"
      }
    },
    "fs": {
      "mode": "write",
      "read_write": ".+\\.json" ,
      "read_only": [ ".+\\.yaml", ".+important-file\\.txt" ],
      "local": [ ".+\\.js", ".+\\.mjs" ]
    },
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "host: api\\..+"
        },
        "port_mapping": [[ 7777, 8888 ]],
        "ignore_localhost": false,
        "ignore_ports": [9999, 10000]
      },
      "outgoing": {
        "tcp": true,
        "udp": true,
        "filter": {
          "local": ["tcp://1.1.1.0/24:1337", "1.1.5.0/24", "google.com", ":53"]
        },
        "ignore_localhost": false,
        "unix_streams": "bear.+"
      },
      "dns": {
        "enabled": true,
        "filter": {
          "local": ["1.1.1.0/24:1337", "1.1.5.0/24", "google.com"]
        }
      }
    },
    "copy_target": {
      "scale_down": false
    }
  },
  "operator": true,
  "kubeconfig": "~/.kube/config",
  "sip_binaries": "bash",
  "telemetry": true,
  "kube_context": "my-cluster"
}

Options

accept_invalid_certificates

Controls whether or not mirrord accepts invalid TLS certificates (e.g. self-signed certificates).

If not provided, mirrord will use value from the kubeconfig.

agent

Configuration for the mirrord-agent pod that is spawned in the Kubernetes cluster.

Note: this configuration is ignored when using the mirrord Operator. Agent configuration is done by the cluster admin.

We provide sane defaults for this option, so you don't have to set up anything here.

{
  "agent": {
    "log_level": "info",
    "json_log": false,
    "namespace": "default",
    "image": "ghcr.io/metalbear-co/mirrord:latest",
    "image_pull_policy": "IfNotPresent",
    "image_pull_secrets": [ { "secret-key": "secret" } ],
    "ttl": 30,
    "ephemeral": false,
    "communication_timeout": 30,
    "startup_timeout": 360,
    "network_interface": "eth0",
    "flush_connections": false
  }
}

agent.annotations

Allows setting up custom annotations for the agent Job and Pod.

{
  "annotations": {
    "cats.io/inject": "enabled"
    "prometheus.io/scrape": "true",
    "prometheus.io/port": "9000"
  }
}

agent.check_out_of_pods

Determine if to check whether there is room for agent job in target node. (Not applicable when using ephemeral containers feature)

Can be disabled if the check takes too long and you are sure there is enough resources on each node

agent.communication_timeout

Controls how long the agent lives when there are no connections.

Each connection has its own heartbeat mechanism, so even if the local application has no messages, the agent stays alive until there are no more heartbeat messages.

agent.disabled_capabilities

Disables specified Linux capabilities for the agent container. If nothing is disabled here, agent uses NET_ADMIN, NET_RAW, SYS_PTRACE and SYS_ADMIN.

Has no effect when using the targetless mode, as targetless agent containers have no capabilities.

agent.dns

agent.ephemeral

Runs the agent as an ephemeral container.

Not compatible with targetless runs.

Defaults to false.

agent.flush_connections

Flushes existing connections when starting to steal, might fix issues where connections aren't stolen (due to being already established)

Defaults to true.

agent.image

Name of the agent's docker image.

Useful when a custom build of mirrord-agent is required, or when using an internal registry.

Defaults to the latest stable image "ghcr.io/metalbear-co/mirrord:latest".

{
  "image": "internal.repo/images/mirrord:latest"
}

Complete setup:

{
  "image": {
    "registry": "internal.repo/images/mirrord",
    "tag": "latest"
  }
}

agent.image_pull_policy

Controls when a new agent image is downloaded.

Supports "IfNotPresent", "Always", "Never", or any valid kubernetes image pull policy

Defaults to "IfNotPresent"

agent.image_pull_secrets

List of secrets the agent pod has access to.

Takes an array of entries with the format { name: <secret-name> }.

Read more here.

{
  "agent": {
    "image_pull_secrets": [
      { "name": "secret-key-1" },
      { "name": "secret-key-2" }
    ]
  }
}

agent.json_log

Controls whether the agent produces logs in a human-friendly format, or json.

{
  "agent": {
    "json_log": true
  }
}

agent.labels

Allows setting up custom labels for the agent Job and Pod.

{
  "labels": { "user": "meow", "state": "asleep" }
}

agent.log_level

Log level for the agent.

Supports "trace", "debug", "info", "warn", "error", or any string that would work with RUST_LOG.

{
  "agent": {
    "log_level": "mirrord=debug,warn"
  }
}

agent.metrics

Enables prometheus metrics for the agent pod.

You might need to add annotations to the agent pod depending on how prometheus is configured to scrape for metrics.

{
  "metrics": "0.0.0.0:9000"
}

agent.namespace

Namespace where the agent shall live.

Note: ignored in targetless runs or when the agent is run as an ephemeral container.

Defaults to the current kubernetes namespace.

agent.network_interface

Which network interface to use for mirroring.

The default behavior is try to access the internet and use that interface. If that fails it uses eth0.

agent.nftables

Use iptables-nft instead of iptables-legacy. Defaults to false.

Needed if your mesh uses nftables instead of iptables-legacy,

agent.node_selector

Allows setting up custom node selector for the agent Pod. Applies only to targetless runs, as targeted agent always runs on the same node as its target container.

{
  "node_selector": { "kubernetes.io/hostname": "node1" }
}

agent.privileged

Run the mirror agent as privileged container. Defaults to false.

Might be needed in strict environments such as Bottlerocket.

Has no effect when using the targetless mode, as targetless agent containers are never privileged.

agent.resources

Set pod resource reqirements. (not with ephemeral agents) Default is

{
  "requests":
  {
    "cpu": "1m",
    "memory": "1Mi"
  },
  "limits":
  {
    "cpu": "100m",
      "memory": "100Mi"
  }
}

agent.service_account

Allows setting up custom Service Account for the agent Job and Pod.

{
  "service_account": "my-service-account"
}

agent.startup_timeout

Controls how long to wait for the agent to finish initialization.

If initialization takes longer than this value, mirrord exits.

Defaults to 60.

agent.tolerations

Set pod tolerations. (not with ephemeral agents).

Defaults to operator: Exists.

[
  {
    "key": "meow", "operator": "Exists", "effect": "NoSchedule"
  }
]

Set to an empty array to have no tolerations at all

agent.ttl

Controls how long the agent pod persists for after the agent exits (in seconds).

Can be useful for collecting logs.

Defaults to 1.

container

Unstable: mirrord container command specific config.

container.cli_extra_args

Any extra args to use when creating the sidecar mirrord-cli container.

This is useful when you want to use portforwarding, passing -p local:container won't work for main command but adding them here will work

{
  "container": {
    "cli_extra_args": ["-p", "local:container"]
  }
}

container.cli_image

Tag of the mirrord-cli image you want to use.

Defaults to "ghcr.io/metalbear-co/mirrord-cli:<cli version>".

container.cli_image_lib_path

Path of the mirrord-layer lib inside the specified mirrord-cli image.

Defaults to "/opt/mirrord/lib/libmirrord_layer.so".

container.cli_prevent_cleanup

Don't add --rm to sidecar command to prevent cleanup.

container.override_host_ip

Allows to override the IP address for the internal proxy to use when connecting to the host machine from within the container.

{
  "container": {
    "override_host_ip": "172.17.0.1" // usual resolution of value from `host.docker.internal`
  }
}

This should be useful if your host machine is exposed with a different IP address than the one bound as host.

experimental

mirrord Experimental features. This shouldn't be used unless someone from MetalBear/mirrord tells you to.

experimental disable_reuseaddr

Disables the SO_REUSEADDR socket option on sockets that mirrord steals/mirrors. On macOS the application can use the same address many times but then we don't steal it correctly. This probably should be on by default but we want to gradually roll it out. https://github.com/metalbear-co/mirrord/issues/2819 This option applies only on macOS.

experimental enable_exec_hooks_linux

Enables exec hooks on Linux. Enable Linux hooks can fix issues when the application shares sockets with child commands (e.g Python web servers with reload), but the feature is not stable and may cause other issues.

experimental hide_ipv6_interfaces

Enables getifaddrs hook that removes IPv6 interfaces from the list returned by libc.

experimental idle_local_http_connection_timeout

Sets a timeout for idle local HTTP connections (in milliseconds).

HTTP requests stolen with a filter are delivered to the local application from a HTTP connection made from the local machine. Once a request is delivered, the connection is cached for some time, so that it can be reused to deliver the next request.

This timeout determines for how long such connections are cached.

Set to 0 to disable caching local HTTP connections (connections will be dropped as soon as the request is delivered).

Defaults to 3000ms.

experimental readlink

DEPRECATED, WILL BE REMOVED

experimental readonly_file_buffer

DEPRECATED, WILL BE REMOVED: moved to feature.fs.readonly_file_buffer as part of stabilisation. See https://github.com/metalbear-co/mirrord/issues/2069.

experimental tcp_ping4_mock

https://github.com/metalbear-co/mirrord/issues/2421#issuecomment-2093200904

experimental trust_any_certificate

Enables trusting any certificate on macOS, useful for https://github.com/golang/go/issues/51991#issuecomment-2059588252

experimental use_dev_null

Uses /dev/null for creating local fake files (should be better than using /tmp)

external_proxy

Configuration for the external proxy mirrord spawns when using the mirrord container command. This proxy is used to allow the internal proxy running in sidecar to connect to the mirrord agent.

If you get ConnectionRefused errors, increasing the timeouts a bit might solve the issue.

{
  "external_proxy": {
    "start_idle_timeout": 30,
    "idle_timeout": 5
  }
}

external_proxy.host_ip

Specify a custom host ip addr to listen on.

This address must be accessible from within the container. If not specified, mirrord will try and resolve a local address to use.

external_proxy.idle_timeout

How much time to wait while we don't have any active connections before exiting.

Common cases would be running a chain of processes that skip using the layer and don't connect to the proxy.

{
  "external_proxy": {
    "idle_timeout": 30
  }
}

external_proxy.json_log

Whether the proxy should output logs in JSON format. If false, logs are output in human-readable format.

Defaults to true.

external_proxy.log_destination

Set the log file destination for the external proxy.

Defaults to a randomized path inside the temporary directory.

external_proxy.log_level

Set the log level for the external proxy.

The value should follow the RUST_LOG convention (i.e mirrord=trace).

Defaults to mirrord=info,warn.

external_proxy.start_idle_timeout

How much time to wait for the first connection to the external proxy in seconds.

Common cases would be running with dlv or any other debugger, which sets a breakpoint on process execution, delaying the layer startup and connection to the external proxy.

{
  "external_proxy": {
    "start_idle_timeout": 60
  }
}

feature

Controls mirrord features.

See the using mirrord section to learn more about what each feature does.

The env, fs and network options have support for a shortened version, that you can see here.

{
  "feature": {
    "env": {
      "include": "DATABASE_USER;PUBLIC_ENV",
      "exclude": "DATABASE_PASSWORD;SECRET_ENV",
      "override": {
        "DATABASE_CONNECTION": "db://localhost:7777/my-db",
        "LOCAL_BEAR": "panda"
      }
    },
    "fs": {
      "mode": "write",
      "read_write": ".+\\.json" ,
      "read_only": [ ".+\\.yaml", ".+important-file\\.txt" ],
      "local": [ ".+\\.js", ".+\\.mjs" ]
    },
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "host: api\\..+"
        },
        "port_mapping": [[ 7777, 8888 ]],
        "ignore_localhost": false,
        "ignore_ports": [9999, 10000]
      },
      "outgoing": {
        "tcp": true,
        "udp": true,
        "filter": {
          "local": ["tcp://1.1.1.0/24:1337", "1.1.5.0/24", "google.com", ":53"]
        },
        "ignore_localhost": false,
        "unix_streams": "bear.+"
      },
      "dns": false
    },
    "copy_target": false,
    "hostname": true
  }
}

feature.copy_target

Creates a new copy of the target. mirrord will use this copy instead of the original target (e.g. intercept network traffic). This feature requires a mirrord operator.

This feature is not compatible with rollout targets and running without a target (targetless mode).

Allows the user to target a pod created dynamically from the orignal target. The new pod inherits most of the original target's specification, e.g. labels.

{
  "feature": {
    "copy_target": {
      "scale_down": true
    }
  }
}

{
  "feature": {
    "copy_target": true
  }
}

feature.copy_target.scale_down

If this option is set, mirrord will scale down the target deployment to 0 for the time the copied pod is alive.

This option is compatible only with deployment targets.

    {
      "scale_down": true
    }

feature.env

Allows the user to set or override the local process' environment variables with the ones from the remote pod.

Can be set to one of the options:

1. false - Disables the feature, won't have remote environment variables.

2. true - Enables the feature, will obtain remote environment variables.

3. object - see below (means true + additional configuration).

Which environment variables to load from the remote pod are controlled by setting either include or exclude.

See the environment variables reference for more details.

{
  "feature": {
    "env": {
      "include": "DATABASE_USER;PUBLIC_ENV;MY_APP_*",
      "exclude": "DATABASE_PASSWORD;SECRET_ENV",
      "override": {
        "DATABASE_CONNECTION": "db://localhost:7777/my-db",
        "LOCAL_BEAR": "panda"
      },
      "mapping": {
        ".+_TIMEOUT": "1000"
      }
    }
  }
}

feature.env_file

Allows for passing environment variables from an env file.

These variables will override environment fetched from the remote target.

feature.env.exclude

Include the remote environment variables in the local process that are NOT specified by this option. Variable names can be matched using * and ? where ? matches exactly one occurrence of any character and * matches arbitrary many (including zero) occurrences of any character.

Some of the variables that are excluded by default: PATH, HOME, HOMEPATH, CLASSPATH, JAVA_EXE, JAVA_HOME, PYTHONPATH.

Can be passed as a list or as a semicolon-delimited string (e.g. "VAR;OTHER_VAR").

feature.env.include

Include only these remote environment variables in the local process. Variable names can be matched using * and ? where ? matches exactly one occurrence of any character and * matches arbitrary many (including zero) occurrences of any character.

Can be passed as a list or as a semicolon-delimited string (e.g. "VAR;OTHER_VAR").

Some environment variables are excluded by default (PATH for example), including these requires specifying them with include

feature.env.load_from_process

Allows for changing the way mirrord loads remote environment variables. If set, the variables are fetched after the user application is started.

This setting is meant to resolve issues when using mirrord via the IntelliJ plugin on WSL and the remote environment contains a lot of variables.

feature.env.mapping

Specify map of patterns that if matched will replace the value according to specification.

Capture groups are allowed.

Example:

{
  ".+_TIMEOUT": "10000"
  "LOG_.+_VERBOSITY": "debug"
  "(\w+)_(\d+)": "magic-value"
}

Will do the next replacements for environment variables that match:

• CONNECTION_TIMEOUT: 500 => CONNECTION_TIMEOUT: 10000

• LOG_FILE_VERBOSITY: info => LOG_FILE_VERBOSITY: debug

• DATA_1234: common-value => DATA_1234: magic-value

feature.env.override

Allows setting or overriding environment variables (locally) with a custom value.

For example, if the remote pod has an environment variable REGION=1, but this is an undesirable value, it's possible to use override to set REGION=2 (locally) instead.

Environment specified here will also override variables passed via the env file.

feature.env.unset

Allows unsetting environment variables in the executed process.

This is useful for when some system/user-defined environment like AWS_PROFILE make the application behave as if it's running locally, instead of using the remote settings. The unsetting happens from extension (if possible)/CLI and when process initializes. In some cases, such as Go the env might not be able to be modified from the process itself. This is case insensitive, meaning if you'd put AWS_PROFILE it'd unset both AWS_PROFILE and Aws_Profile and other variations.

feature.fs

Allows the user to specify the default behavior for file operations:

1. "read" or true - Read from the remote file system (default)

2. "write" - Read/Write from the remote file system.

3. "local" or false - Read from the local file system.

4. "localwithoverrides" - perform fs operation locally, unless the path matches a pre-defined or user-specified exception.

Note: by default, some paths are read locally or remotely, regardless of the selected FS mode. This is described in further detail below.

Besides the default behavior, the user can specify behavior for specific regex patterns. Case insensitive.

1. "read_write" - List of patterns that should be read/write remotely.

2. "read_only" - List of patterns that should be read only remotely.

3. "local" - List of patterns that should be read locally.

4. "not_found" - List of patters that should never be read nor written. These files should be treated as non-existent.

5. "mapping" - Map of patterns and their corresponding replacers. The replacement happens before any specific behavior as defined above or mode (uses Regex::replace)

The logic for choosing the behavior is as follows:

1. Check agains "mapping" if path needs to be replaced, if matched then continue to next step with new path after replacements otherwise continue as usual.

2. Check if one of the patterns match the file path, do the corresponding action. There's no specified order if two lists match the same path, we will use the first one (and we do not guarantee what is first).

   Warning: Specifying the same path in two lists is unsupported and can lead to undefined behaviour.

3. There are pre-defined exceptions to the set FS mode.

   1. Paths that match the patterns defined here are read locally by default.

   2. Paths that match the patterns defined here are read remotely by default when the mode is localwithoverrides.

   3. Paths that match the patterns defined here under the running user's home directory will not be found by the application when the mode is not local.

   In order to override that default setting for a path, or a pattern, include it the appropriate pattern set from above. E.g. in order to read files under /etc/ remotely even though it is covered by the set of patterns that are read locally by default, add "^/etc/." to the read_only set.

4. If none of the above match, use the default behavior (mode).

For more information, check the file operations technical reference.

{
  "feature": {
    "fs": {
      "mode": "write",
      "read_write": ".+\\.json" ,
      "read_only": [ ".+\\.yaml", ".+important-file\\.txt" ],
      "local": [ ".+\\.js", ".+\\.mjs" ],
      "not_found": [ "\\.config/gcloud" ]
    }
  }
}

feature.fs.local

Specify file path patterns that if matched will be opened locally.

feature.fs.mapping

Specify map of patterns that if matched will replace the path according to specification.

Capture groups are allowed.

Example:

{
  "^/home/(?<user>\\S+)/dev/tomcat": "/etc/tomcat"
  "^/home/(?<user>\\S+)/dev/config/(?<app>\\S+)": "/mnt/configs/${user}-$app"
}

Will do the next replacements for any io operaton

/home/johndoe/dev/tomcat/context.xml => /etc/tomcat/context.xml /home/johndoe/dev/config/api/app.conf => /mnt/configs/johndoe-api/app.conf

• Relative paths: this feature (currently) does not apply mappings to relative paths, e.g. ../dev.

feature.fs.mode

Configuration for enabling read-only or read-write file operations.

These options are overriden by user specified overrides and mirrord default overrides.

If you set "localwithoverrides" then some files can be read/write remotely based on our default/user specified. Default option for general file configuration.

The accepted values are: "local", "localwithoverrides, "read", or "write.

feature.fs.not_found

Specify file path patterns that if matched will be treated as non-existent.

feature.fs.read_only

Specify file path patterns that if matched will be read from the remote. if file matching the pattern is opened for writing or read/write it will be opened locally.

feature.fs.read_write

Specify file path patterns that if matched will be read and written to the remote.

feature.fs.readonly_file_buffer

Sets buffer size for read-only remote files in bytes. By default, the value is 128000 bytes, or 128 kB.

Setting the value to 0 disables file buffering. Otherwise, read-only remote files will be read in chunks and buffered locally. This improves performance when the user application reads data in small portions.

feature.hostname

Should mirrord return the hostname of the target pod when calling gethostname

feature.network

Controls mirrord network operations.

See the network traffic reference for more details.

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "host: api\\..+"
        },
        "port_mapping": [[ 7777, 8888 ]],
        "ignore_localhost": false,
        "ignore_ports": [9999, 10000]
      },
      "outgoing": {
        "tcp": true,
        "udp": true,
        "filter": {
          "local": ["tcp://1.1.1.0/24:1337", "1.1.5.0/24", "google.com", ":53"]
        },
        "ignore_localhost": false,
        "unix_streams": "bear.+"
      },
      "dns": {
        "enabled": true,
        "filter": {
          "local": ["1.1.1.0/24:1337", "1.1.5.0/24", "google.com"]
        }
      }
    }
  }
}

feature.network.dns

Resolve DNS via the remote pod.

Defaults to true.

Mind that:

• DNS resolving can be done in multiple ways. Some frameworks use getaddrinfo/gethostbyname functions, while others communicate directly with the DNS server at port 53 and perform a sort of manual resolution. Just enabling the dns feature in mirrord might not be enough. If you see an address resolution error, try enabling the fs feature, and setting read_only: ["/etc/resolv.conf"].

• DNS filter currently works only with frameworks that use getaddrinfo/gethostbyname functions.

feature.network.dns.filter

Unstable: the precise syntax of this config is subject to change.

List of addresses/ports/subnets that should be resolved through either the remote pod or local app, depending how you set this up with either remote or local.

You may use this option to specify when DNS resolution is done from the remote pod (which is the default behavior when you enable remote DNS), or from the local app (default when you have remote DNS disabled).

Takes a list of values, such as:

• Only queries for hostname my-service-in-cluster will go through the remote pod.

{
  "remote": ["my-service-in-cluster"]
}

• Only queries for addresses in subnet 1.1.1.0/24 with service port `1337`` will go through the remote pod.

{
  "remote": ["1.1.1.0/24:1337"]
}

• Only queries for hostname google.com with service port 1337 or 7331 will go through the remote pod.

{
  "remote": ["google.com:1337", "google.com:7331"]
}

• Only queries for localhost with service port 1337 will go through the local app.

{
  "local": ["localhost:1337"]
}

• Only queries with service port 1337 or 7331 will go through the local app.

{
  "local": [":1337", ":7331"]
}

Valid values follow this pattern: [name|address|subnet/mask][:port].

feature.network.incoming

Controls the incoming TCP traffic feature.

See the incoming traffic reference for more details.

Incoming traffic supports 3 modes of operation:

1. Mirror (default): Sniffs the TCP data from a port, and forwards a copy to the interested listeners;

2. Steal: Captures the TCP data from a port, and forwards it to the local process.

3. Off: Disables the incoming network feature.

This field can either take an object with more configuration fields (that are documented below), or alternatively -

• A boolean:

  • true: use the default configuration, same as not specifying this field at all.

  • false: disable incoming configuration.

• One of the incoming modes (lowercase).

Examples:

Steal all the incoming traffic:

{
  "feature": {
    "network": {
      "incoming": "steal"
    }
  }
}

Disable the incoming traffic feature:

{
  "feature": {
    "network": {
      "incoming": false
    }
  }
}

Steal only traffic that matches the http_filter (steals only HTTP traffic).

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "host: api\\..+"
        },
        "port_mapping": [[ 7777, 8888 ]],
        "ignore_localhost": false,
        "ignore_ports": [9999, 10000],
        "listen_ports": [[80, 8111]]
      }
    }
  }
}

feature.network.incoming.http_filter

Filter configuration for the HTTP traffic stealer feature.

Allows the user to set a filter (regex) for the HTTP headers, so that the stealer traffic feature only captures HTTP requests that match the specified filter, forwarding unmatched requests to their original destinations.

Only does something when feature.network.incoming.mode is set as "steal", ignored otherwise.

For example, to filter based on header:

{
  "header_filter": "host: api\\..+"
}

Setting that filter will make mirrord only steal requests with the host header set to hosts that start with "api", followed by a dot, and then at least one more character.

For example, to filter based on path:

{
  "path_filter": "^/api/"
}

Setting this filter will make mirrord only steal requests to URIs starting with "/api/".

This can be useful for filtering out Kubernetes liveness, readiness and startup probes. For example, for avoiding stealing any probe sent by kubernetes, you can set this filter:

{
  "header_filter": "^User-Agent: (?!kube-probe)"
}

Setting this filter will make mirrord only steal requests that do have a user agent that does not begin with "kube-probe".

Similarly, you can exclude certain paths using a negative look-ahead:

{
  "path_filter": "^(?!/health/)"
}

Setting this filter will make mirrord only steal requests to URIs that do not start with "/health/".

With all_of and any_of, you can use multiple HTTP filters at the same time.

If you want to steal HTTP requests that match every pattern specified, use all_of. For example, this filter steals only HTTP requests to endpoint /api/my-endpoint that contain header x-debug-session with value 121212.

{
  "all_of": [
    { "header": "^x-debug-session: 121212$" },
    { "path": "^/api/my-endpoint$" }
  ]
}

If you want to steal HTTP requests that match any of the patterns specified, use any_of. For example, this filter steals HTTP requests to endpoint /api/my-endpoint and HTTP requests that contain header x-debug-session with value 121212.

{
 "any_of": [
   { "path": "^/api/my-endpoint$"},
   { "header": "^x-debug-session: 121212$" }
 ]
}

feature.network.incoming.http_filter.all_of

An array of HTTP filters.

Each inner filter specifies either header or path regex. Requests must match all of the filters to be stolen.

Cannot be an empty list.

Example:

{
  "all_of": [
    { "header": "x-user: my-user$" },
    { "path": "^/api/v1/my-endpoint" }
  ]
}

feature.network.incoming.http_filter.any_of

An array of HTTP filters.

Each inner filter specifies either header or path regex. Requests must match at least one of the filters to be stolen.

Cannot be an empty list.

Example:

{
  "any_of": [
    { "header": "^x-user: my-user$" },
    { "path": "^/api/v1/my-endpoint" }
  ]
}

feature.network.incoming.http_filter.header_filter

Supports regexes validated by the fancy-regex crate.

The HTTP traffic feature converts the HTTP headers to HeaderKey: HeaderValue, case-insensitive.

feature.network.incoming.http_filter.path_filter

Supports regexes validated by the fancy-regex crate.

Case-insensitive. Tries to find match in the path (without query) and path+query. If any of the two matches, the request is stolen.

feature.network.incoming.http_filter.ports

Activate the HTTP traffic filter only for these ports.

Other ports will not be stolen, unless listed in feature.network.incoming.ports.

Set to [80, 8080] by default.

feature.network.incoming.https_delivery

(Operator Only): configures how mirrord delivers stolen HTTPS requests to the local application.

Stolen HTTPS requests can be delivered to the local application either as HTTPS or as plain HTTP requests. Note that stealing HTTPS requests requires mirrord Operator support.

To have the stolen HTTPS requests delivered with plain HTTP, use:

{
  "protocol": "tcp"
}

To have the requests delivered with HTTPS, use:

{
  "protocol": "tls"
}

By default, the local mirrord TLS client will trust any certificate presented by the local application's HTTP server. To override this behavior, you can either:

1. Specify a list of paths to trust roots. These paths can lead either to PEM files or PEM file directories. Each found certificate will be used as a trust anchor.

2. Specify a path to the cartificate chain used by the server.

Example with trust roots:

{
  "protocol": "tls",
  "trust_roots": ["/path/to/cert.pem", "/path/to/cert/dir"]
}

Example with certificate chain:

{
  "protocol": "tls",
  "server_cert": "/path/to/cert.pem"
}

To make a TLS connection to the local application's HTTPS server, mirrord's TLS client needs a server name. You can supply it manually like this:

{
  "protocol": "tls",
  "server_name": "my.test.server.name"
}

If you don't supply the server name:

1. If server_cert is given, and the found end-entity certificate contains a valid server name, this server name will be used;

2. Otherwise, if the original client supplied an SNI extension, the server name from that extension will be used;

3. Otherwise, if the stolen request's URL contains a valid server name, that server name will be used;

4. Otherwise, localhost will be used.

feature.network.incoming.https_delivery.protocol

Protocol to use when delivering the HTTPS requests locally.

Path to a PEM file containing the certificate chain used by the local application's HTTPS server.

This file must contain at least one certificate. It can contain entries of other types, e.g private keys, which are ignored.

feature.network.incoming.https_delivery.server_name

Server name to use when making a connection.

Must be a valid DNS name or an IP address.

feature.network.incoming.https_delivery.trust_roots

Paths to PEM files and directories with PEM files containing allowed root certificates.

Directories are not traversed recursively.

Each certificate found in the files is treated as an allowed root. The files can contain entries of other types, e.g private keys, which are ignored.

feature.network.incoming.ignore_localhost

feature.network.incoming.ignore_ports

Ports to ignore when mirroring/stealing traffic, these ports will remain local.

Can be especially useful when feature.network.incoming.mode is set to "steal", and you want to avoid redirecting traffic from some ports (for example, traffic from a health probe, or other heartbeat-like traffic).

Mutually exclusive with feature.network.incoming.ports.

feature.network.incoming.listen_ports

Mapping for local ports to actually used local ports. When application listens on a port while steal/mirror is active we fallback to random ports to avoid port conflicts. Using this configuration will always use the specified port. If this configuration doesn't exist, mirrord will try to listen on the original port and if it fails it will assign a random port

This is useful when you want to access ports exposed by your service locally For example, if you have a service that listens on port 80 and you want to access it, you probably can't listen on 80 without sudo, so you can use [[80, 4480]] then access it on 4480 while getting traffic from remote 80. The value of port_mapping doesn't affect this.

feature.network.incoming.mode

Allows selecting between mirrorring or stealing traffic.

Can be set to either "mirror" (default), "steal" or "off".

• "mirror": Sniffs on TCP port, and send a copy of the data to listeners.

• "off": Disables the incoming network feature.

• "steal": Supports 2 modes of operation:

1. Port traffic stealing: Steals all TCP data from a port, which is selected whenever the user listens in a TCP socket (enabling the feature is enough to make this work, no additional configuration is needed);

2. HTTP traffic stealing: Steals only HTTP traffic, mirrord tries to detect if the incoming data on a port is HTTP (in a best-effort kind of way, not guaranteed to be HTTP), and steals the traffic on the port if it is HTTP;

feature.network.incoming.on_concurrent_steal

(Operator Only): Allows overriding port locks

Can be set to either "continue" or "override".

• "continue": Continue with normal execution

• "override": If port lock detected then override it with new lock and force close the original locking connection.

feature.network.incoming.port_mapping

Mapping for local ports to remote ports.

This is useful when you want to mirror/steal a port to a different port on the remote machine. For example, your local process listens on port 9333 and the container listens on port 80. You'd use [[9333, 80]]

feature.network.incoming.ports

List of ports to mirror/steal traffic from. Other ports will remain local.

Mutually exclusive with feature.network.incoming.ignore_ports.

feature.network.ipv6

Enable ipv6 support. Turn on if your application listens to incoming traffic over IPv6, or connects to other services over IPv6.

feature.network.outgoing

Tunnel outgoing network operations through mirrord.

See the outgoing traffic reference for more details.

The remote and local config for this feature are mutually exclusive.

{
  "feature": {
    "network": {
      "outgoing": {
        "tcp": true,
        "udp": true,
        "ignore_localhost": false,
        "filter": {
          "local": ["tcp://1.1.1.0/24:1337", "1.1.5.0/24", "google.com", ":53"]
        },
        "unix_streams": "bear.+"
      }
    }
  }
}

feature.network.outgoing.filter

Filters that are used to send specific traffic from either the remote pod or the local app

List of addresses/ports/subnets that should be sent through either the remote pod or local app, depending how you set this up with either remote or local.

You may use this option to specify when outgoing traffic is sent from the remote pod (which is the default behavior when you enable outgoing traffic), or from the local app (default when you have outgoing traffic disabled).

Takes a list of values, such as:

• Only UDP traffic on subnet 1.1.1.0/24 on port 1337 will go through the remote pod.

{
  "remote": ["udp://1.1.1.0/24:1337"]
}

• Only UDP and TCP traffic on resolved address of google.com on port 1337 and 7331 will go through the remote pod.

{
  "remote": ["google.com:1337", "google.com:7331"]
}

• Only TCP traffic on localhost on port 1337 will go through the local app, the rest will be emmited remotely in the cluster.

{
  "local": ["tcp://localhost:1337"]
}

• Only outgoing traffic on port 1337 and 7331 will go through the local app.

{
  "local": [":1337", ":7331"]
}

Valid values follow this pattern: [protocol]://[name|address|subnet/mask]:[port].

feature.network.outgoing.ignore_localhost

Defaults to false.

feature.network.outgoing.tcp

Defaults to true.

feature.network.outgoing.udp

Defaults to true.

feature.network.outgoing.unix_streams

Connect to these unix streams remotely (and to all other paths locally).

You can either specify a single value or an array of values. Each value is interpreted as a regular expression (Supported Syntax).

When your application connects to a unix socket, the target address will be converted to a string (non-utf8 bytes are replaced by a placeholder character) and matched against the set of regexes specified here. If there is a match, mirrord will connect your application with the target unix socket address on the target pod. Otherwise, it will leave the connection to happen locally on your machine.

feature.split_queues

Define filters to split queues by, and make your local application consume only messages that match those filters. If you don't specify any filter for a queue that is however declared in the MirrordWorkloadQueueRegistry of the target you're using, a match-nothing filter will be used, and your local application will not receive any messages from that queue.

{
  "feature": {
    "split_queues": {
      "first-queue": {
        "queue_type": "SQS",
        "message_filter": {
          "wows": "so wows",
          "coolz": "^very"
        }
      },
      "second-queue": {
        "queue_type": "SQS",
        "message_filter": {
          "who": "you$"
        }
      },
      "third-queue": {
        "queue_type": "Kafka",
        "message_filter": {
          "who": "you$"
        }
      },
      "fourth-queue": {
        "queue_type": "Kafka",
        "message_filter": {
          "wows": "so wows",
          "coolz": "^very"
        }
      },
    }
  }
}

internal_proxy

Configuration for the internal proxy mirrord spawns for each local mirrord session that local layers use to connect to the remote agent

This is seldom used, but if you get ConnectionRefused errors, you might want to increase the timeouts a bit.

{
  "internal_proxy": {
    "start_idle_timeout": 30,
    "idle_timeout": 5
  }
}

internal_proxy.idle_timeout

How much time to wait while we don't have any active connections before exiting.

Common cases would be running a chain of processes that skip using the layer and don't connect to the proxy.

{
  "internal_proxy": {
    "idle_timeout": 30
  }
}

internal_proxy.json_log

Whether the proxy should output logs in JSON format. If false, logs are output in human-readable format.

Defaults to true.

internal_proxy.log_destination

Set the log file destination for the internal proxy.

Defaults to a randomized path inside the temporary directory.

internal_proxy.log_level

Set the log level for the internal proxy.

The value should follow the RUST_LOG convention (i.e mirrord=trace).

Defaults to mirrord=info,warn.

internal_proxy.start_idle_timeout

How much time to wait for the first connection to the proxy in seconds.

Common cases would be running with dlv or any other debugger, which sets a breakpoint on process execution, delaying the layer startup and connection to proxy.

{
  "internal_proxy": {
    "start_idle_timeout": 60
  }
}

kube_context

Kube context to use from the kubeconfig file. Will use current context if not specified.

{
  "kube_context": "mycluster"
}

kubeconfig

Path to a kubeconfig file, if not specified, will use KUBECONFIG, or ~/.kube/config, or the in-cluster config.

{
  "kubeconfig": "~/bear/kube-config"
}

operator

Whether mirrord should use the operator. If not set, mirrord will first attempt to use the operator, but continue without it in case of failure.

profile

Name of the mirrord profile to use.

sip_binaries

Binaries to patch (macOS SIP).

Use this when mirrord isn't loaded to protected binaries that weren't automatically patched.

Runs endswith on the binary path (so bash would apply to any binary ending with bash while /usr/bin/bash would apply only for that binary).

{
  "sip_binaries": ["bash", "python"]
}

skip_build_tools

Allows mirrord to skip build tools. Useful when running command lines that build and run the application in a single command.

Defaults to true.

Build-Tools: ["as", "cc", "ld", "go", "air", "asm", "cc1", "cgo", "dlv", "gcc", "git", "link", "math", "cargo", "hpack", "rustc", "compile", "collect2", "cargo-watch", "debugserver"]

skip_extra_build_tools

Allows mirrord to skip the specified build tools. Useful when running command lines that build and run the application in a single command.

Must also enable skip_build_tools for this to take an effect.

It's similar to skip_processes, except that here it also skips SIP patching.

Accepts a single value, or an array of values.

{
 "skip_extra_build_tools": ["bash", "node"]
}

skip_processes

Allows mirrord to skip unwanted processes.

Useful when process A spawns process B, and the user wants mirrord to operate only on process B. Accepts a single value, or an array of values.

{
 "skip_processes": ["bash", "node"]
}

skip_sip

Allows mirrord to skip patching (macOS SIP) unwanted processes.

When patching is skipped, mirrord will no longer be able to load into the process and its child processes.

target

Specifies the target and namespace to target.

The simplified configuration supports:

• targetless

• pod/{pod-name}[/container/{container-name}];

• deployment/{deployment-name}[/container/{container-name}];

• rollout/{rollout-name}[/container/{container-name}];

• job/{job-name}[/container/{container-name}];

• cronjob/{cronjob-name}[/container/{container-name}];

• statefulset/{statefulset-name}[/container/{container-name}];

• service/{service-name}[/container/{container-name}];

Please note that:

• job, cronjob, statefulset and service targets require the mirrord Operator

• job and cronjob targets require the copy_target feature

Shortened setup with a target:

{
 "target": "pod/bear-pod"
}

The setup above will result in a session targeting the bear-pod Kubernetes pod in the user's default namespace. A target container will be chosen by mirrord.

Shortened setup with a target container:

{
  "target": "pod/bear-pod/container/bear-pod-container"
}

The setup above will result in a session targeting the bear-pod-container container in the bear-pod Kubernetes pod in the user's default namespace.

Complete setup with a target container:

{
 "target": {
   "path": {
     "pod": "bear-pod",
     "container": "bear-pod-container"
   },
   "namespace": "bear-pod-namespace"
 }
}

The setup above will result in a session targeting the bear-pod-container container in the bear-pod Kubernetes pod in the bear-pod-namespace namespace.

Setup with a namespace for a targetless run:

{
  "target": {
    "path": "targetless",
    "namespace": "bear-namespace"
  }
}

The setup above will result in a session without any target. Remote outgoing traffic and DNS will be done from the bear-namespace namespace.

target.namespace

Namespace where the target lives.

For targetless runs, this the namespace in which remote networking is done.

Defaults to the Kubernetes user's default namespace (defined in Kubernetes context).

target.path

Specifies the Kubernetes resource to target.

If not given, defaults to targetless.

Note: targeting services and whole workloads is available only in mirrord for Teams. If you target a workload without the mirrord Operator, it will choose a random pod replica to work with.

Supports:

• targetless

• pod/{pod-name}[/container/{container-name}];

• deployment/{deployment-name}[/container/{container-name}];

• rollout/{rollout-name}[/container/{container-name}];

• job/{job-name}[/container/{container-name}]; (requires mirrord Operator and the copy_target feature)

• cronjob/{cronjob-name}[/container/{container-name}]; (requires mirrord Operator and the copy_target feature)

• statefulset/{statefulset-name}[/container/{container-name}]; (requires mirrord Operator)

• service/{service-name}[/container/{container-name}]; (requires mirrord Operator)

• replicaset/{replicaset-name}[/container/{container-name}]; (requires mirrord Operator)

telemetry

Controls whether or not mirrord sends telemetry data to MetalBear cloud. Telemetry sent doesn't contain personal identifiers or any data that should be considered sensitive. It is used to improve the product. For more information

use_proxy

When disabled, mirrord will remove HTTP[S]_PROXY env variables before doing any network requests. This is useful when the system sets a proxy but you don't want mirrord to use it. This also applies to the mirrord process (as it just removes the env). If the remote pod sets this env, the mirrord process will still use it.

Overview

You can specify a target on your cluster for mirrord, giving your local application access to the remote target's network environment, file system and environment variables, according to the configuration. When a target is specified, a mirrord-agent pod will be created on the same node as the target pod. The several kinds of supported targets are detailed below. There are also multiple ways to specify a target for mirrord: you can do it in a configuration file, in an IDE dialog, or in the CLI with an argument or an environment variable.

Possible targets

mirrord OSS supports the following Kubernetes objects as targets:

• Pods

• Deployments

• Argo Rollouts

In mirrord OSS, mirrord will always target a random pod when a workload with multiple pods is used as the remote target.

mirrord for Teams adds support for the following workloads:

• Jobs

• CronJobs

• StatefulSets

In mirrord for Teams, mirrord will always target all pods when a workload with multiple pods is used as the remote target.

Both in mirrord OSS and mirrord for Teams, if you don't name any specific container to be targeted, mirrord will pick the first container from the pod spec. Some containers, like service mesh proxies, will be automatically ignored.

You can specify a target namespace if the target should be found in that namespace instead of the namespace that is currently used by kubectl. See the different interfaces below for possible ways of specifying the target and its namespace.

Specifying a target

There are multiple ways to specify a target. In all the possible interfaces for specifying a target, the basic format is <resource-type>/<resource-name> optionally followed by /container/<container-name>. So for specifying a target without specifying a container you can pass

deploy/<YOUR-DEPLOYMENT-NAME>

e.g. deploy/lolz,

or

pod/<YOUR-POD-NAME>

e.g. pod/lolz-64698df9b7-6plq8,

And for also specifying a container, you just add /container/<CONTAINER-NAME> at the end, e.g. pod/lolz-64698df9b7-6plq8/container/main-container.

Using a configuration file

The target path from the last section is set under the target.path field. The target's namespace can be set under target.namespace. By default, the namespace currently specified in the local kubeconfig is used.

{
  "target": {
    "path": "pod/lolz-64698df9b7-6plq8/container/main-container",
    "namespace": "lolzspace"
  }
}

Using an IDE's dialog

If you are running one of mirrord's IDE extensions and you didn't specify a target via a configuration file, a dialog will pop up for you to pick a target. If you want to choose a target from a different namespace you can set a target namespace in the configuration file, and the dialog will then contain targets in that namespace. Choose the No Target ("targetless") option in the dialog in order to run without a target.

Using a command line argument

If you are running mirrord from the command line, you can specify the target via -t and its namespace via -n, e.g. mirrord exec -t deploy/lolz -n lolzspace my-app. Values specified by command line arguments will be used even if other values are set in a configuration file or in environment variables.

Using an environment variable

You can set the target using the environment variable MIRRORD_IMPERSONATED_TARGET and the target's namespace using the environment variable MIRRORD_TARGET_NAMESPACE. Values specified by environment variables will be used even if other values are set in a configuration file.

Running without a target

When no target is specified, mirrord will start a targetless agent. That can be useful when you want to connect to services from within the cluster, but you don't have any target that you want to "impersonate" - like when running an external utility or a new microservice. When running targetless, mirrord will forward any connections initiated by your application to be sent out of the cluster, but it will not mirror or steal incoming traffic, as a targetless agent is not connected to any Kubernetes service and does not expose any ports. This means that if your application binds a port and listens on it, that will all happen locally, on your machine. So if you're using a management program that exposes a web interface, you can have it listen for connections on localhost, and connect to remote services in the cluster.

If you're using a mirrord configuration file and want to run targetless, you can either leave the target key out completely or specify it explicitly. Note that if you want to skip the target dialog in the IDE plugins, you have to specify it explicitly. You can do it with the following configuration:

{
  "target": "targetless"
}

In your IDE you can pick the No Target ("targetless") option in the target selection dialog (or just not make a selection). Moreover, you should make sure the environment variable used to specify a target isn't set (or is set to an empty value).

Note: In order to set the namespace the agent is going to be created in, set the agent namespace, not the target namespace. That value can be set via the agent.namespace configuration file field, the -a CLI argument, or the MIRRORD_AGENT_NAMESPACE environment variable.