mirrord helps developers improve their development experience by seamlessly mirroring Kubernetes traffic and configuration from staging or testing environments directly into their local environment. This allows developers to test their locally written code in a production-like environment without going through tedious CI pipelines or repeatedly having to build and deploy container images.

mirrord is a tool that lets developers run local processes in the context of their cloud environment. It makes it incredibly easy to test your code on a cloud environment (e.g. staging) without actually going through the hassle of Dockerization, CI, or deployment, and without disrupting the environment by deploying untested code. Instead of saving it for the last step, now you can shift-left on cloud testing and test your code in the cloud from the very beginning of your development process.

Want to see mirrord in action? Check out our demo.

Why?

Traditionally, software development happens in loops. Developers write and test their code locally, then deploy it to a staging/pre-production environment in the cloud, where they perform additional tests. These tests often fail, because the code is meeting a production-like environment for the first time, and encounters new conditions. The code must then be fixed/rewritten, tested locally again, deployed to staging again, and so on, until the tests on staging pass.

Deployment to staging can be costly for several reasons:

1. It often involves a CI process, which may be slow (because of e.g. a long automated test suite having to pass in order to progress) and sometimes broken.

2. Since staging environments are shared, the environment is occasionally broken when an engineer deploys unstable code.

mirrord removes the costs associated with deployment to staging, by taking 'deployment' out of the process completely. By plugging your local process directly into the staging environment, you can test your code in cloud conditions without having to go through a long CI process, and without the risk of breaking the environment for other developers.

However, the point of mirrord is not just to make that final step in the dev loop of testing in staging quicker. mirrord makes running your code in the cloud easy, fast and safe, so you can shift left on cloud testing, and test your code in the cloud from the very beginning of your development process. Instead of spending your time running local environments, writing mocks, test fixtures, and so on - why not just test your code on staging itself?

How it works

mirrord runs in two places - in the memory of your local process (mirrord-layer), and as a pod in your cloud environment (mirrord-agent).

When you start your local process with mirrord, it creates a pod in your cloud environment, which listens in on the pod you've passed as an argument. mirrord-layer then does the following:

• Override the process' syscalls to:

  • Listen to incoming traffic from the agent, instead of local sockets.

  • Intercept outgoing traffic and send it out from the remote pod, instead of locally.

  • Read and write files to the remote file system.

• Merge the process' environment variables with those of the remote pod.

The remote part of this logic is handled by the agent, which runs in the network namespace of the remote pod, and can access its file system and environment variables.

For further details, see the architecture section.

How it's different from other remocal solutions

mirrord is not the first tool to allow you to run your code in the cloud. However, it does it in a way that's completely different from all the other solutions. While all other remocal solutions use some version of a VPN to connect your local machine (or local Docker container) to the cluster, mirrord works at the level of your local process, overriding its syscalls and proxying them to the cloud. Similarly, at the cluster level, it runs at the level of the target pod, running on the same node and executing the syscalls received from the client.

This gives mirrord some unique advantages over its alternatives:

• You can configure exactly what functionality happens remotely, and what stays local. For example:

  • You can read some files or environment variables from your local machine and some from the remote pod

  • You can make requests to certain hosts or IPs happen locally, and others be sent out from the remote pod

• At the local level, it doesn't require root access

• At the local level, it takes 15 seconds at most to start up

• At the local level, you can run multiple processes simultaneously, each in the context of a different remote pod

• At the cluster level, it's agnostic to the cluster's network setup - whether it includes a service mesh, a VPN, or anything else

• At the cluster level, it's agnostic to the cluster's size - mirrord has been tested on clusters running 10,000+ pods

There are a lot of ways you can contribute to mirrord, like:

• Write code! See our guide . Most of our code is in Rust, but we also have some TypeScript and Kotlin (in our and extensions, respectively), and even some Assembly if you're feeling particularly brave. To start writing code for mirrord, find an issue you'd like to fix or implement in our . Please let us know that you're working on an issue beforehand, so we can assign it to you, provide help and reduce double-work.

• Report a bug you found in our .

• Suggest a feature or improvement in a discussion on our .

• Write a blog post about your experience with mirrord. We'd be happy to share it, or even host it on the - just open a PR with the new post the .

• Contribute to our docs and website by sending in a PR to the .

Community is at the core of the mirrord project, and we welcome anyone and everyone that wants to join. We mainly hang out on - if you need help or want to chat, feel free to join us there!

Ready to start using mirrord for Teams? Register to get started.

Why mirrord for Teams?

So you've tried out mirrord on your cloud development environment and you liked it. You've even shown it to your teammates, and they tried it out on their own personal environments. Great! Now all of you might want to use mirrord together on the same environment (your shared staging environment, for example). This is where things might get tricky:

1. Your DevOps team might not be too happy about giving everyone privileged permissions to the shared environment.

2. Two or more developers might want to run mirrord simultaneously on the same pod or deployment without clashing.

3. You might be sharing the staging environment with other teams, who wouldn't want you to mess with their components.

4. Your shared environment is more likely to have multi-pod deployments, which you might want to mirror entirely, rather than just the first pod.

As you might have guessed, this is where mirrord for Teams comes in.

How It Works

In the basic version of mirrord, mirrord injects itself into the local process, and creates a pod in the Kubernetes cluster. It's completely standalone, and is unaware of other instances of mirrord running on the same cluster.

In mirrord for Teams, we introduce a new component - the mirrord Operator. The Operator is a Kubernetes operator that runs persistently in the cluster and manages the mirrord instances trying to access it. The Operator itself is the one creating mirrord agents, so individual users no longer need elevated Kubernetes permissions to use mirrord. Additionally, a centralized component makes possible things like concurrent use, or limiting access or specific actions to certain cluster components.

Supported Features

The following functionality is currently available in mirrord for Teams that isn't available in the open-source version:

• Concurrent use - mirrord for Teams includes various features that allow your entire team to work on a shared environment without conflicts.

• Better security and RBAC - with the mirrord Operator, users no longer need permissions to create privileged pods - only the Operator does. In addition, permissions can be managed within Kubernetes to allow or prevent users from impersonating specific targets.

• Advanced features - support for new resource types like Jobs and StatefulSets, multi-pod deployments, and more.

For the full list of features, see the mirrord for Teams .

This section provides detailed guides and explanations on how to use mirrord in various scenarios. You'll find instructions and best practices for features such as copying targets, running local containers, filtering outgoing traffic, port forwarding, managing sessions, stealing HTTPS and general traffic, using targetless mode, web browsing, and integrating with popular development tools like IntelliJ and VSCode. Whether you're new to mirrord or looking to leverage its advanced capabilities, these topics will help you get the most out of your development workflow.

This section provides in-depth technical reference for mirrord. Here you'll find detailed documentation on configuration options, supported targets, third-party integrations, network traffic handling, architecture, environment variables, and file operations. Use this section as a comprehensive resource for understanding and customizing mirrord's behavior in your environment.

The common use case for mirrord is testing out modifications to an existing application. In this case, the stable version of the service is running in the cloud, and the new code runs locally, using the stable cloud version as its remote target. However, sometimes you want to test a brand new application that has never been deployed to the cloud. Or you might not want to run an application at all - maybe you just want to run a tool, like Postman or pgAdmin, in the context of your cluster.

This is where targetless mode comes in. When running in targetless mode, mirrord doesn't impersonate a remote target. There's no incoming traffic functionality in this mode, since there's no remote target receiving traffic, but everything else works exactly the same.

To run mirrord in targetless mode, just don't specify a target! For example:

mirrord exec /bin/my-tool

IDE

If you want to run in targetless mode using the IntelliJ or VSCode plugin, you can select the No Target ("targetless") option from the target selection dialog, or you can add

{
  "target": "targetless"
}

to your mirrord configuration file.

What's next?

1. If you'd like to intercept traffic rather than mirror it so that your local process is the one answering the remote requests, check out this guide. Note that you can even filter which traffic you intercept!

2. Want to use Targetless mode to run a web browser in the context of your cluster? Check out this guide.

3. If you just want to learn more about mirrord, why not check out our architecture or configuration sections?

Regular port-forwarding

The port-forward command allows you to forward traffic from a local port to any destination that the mirrord targeted pod has access to, in a similar way to kubectl port-forward. The traffic is forwarded as-if it was coming from the target pod, meaning it has access to destinations that might be outside the cluster, like third-party APIs, depending on what's accessible by the target pod.

You can use the command like so:

mirrord port-forward --target <target-path> -L <local port>:<remote address>:<remote port>

For example, to forward traffic from localhost:8080 to an incluster service py-serv listening on port 80:

mirrord port-forward -L 8080:py-serv:80

Reverse port-forwarding

It also allows for reverse port forwarding, where traffic is redirected from a port on the target pod or workload to a local port, like so:

mirrord port-forward --target <target-path> -R <remote port>:<local port>

For example, to forward traffic from an incluster deployment py-serv listening on port 80 to localhost:8080:

mirrord port-forward --target deployment/py-serv -R 80:8080

In addition, multiple ports can be forwarded in one direction or both directions simultaneously in the same command by providing each source and destination as a separate -L or -R argument.

Regular port forwarding with an -L can be done in targetless mode and does not require specifying any target. Reverse port forwarding always requires a target.

More details

• The local port component of the -L argument is optional, and without it the same port will be used locally as on the remote.

• The same is true of the -R argument: if one port number is provided, it will be used for both local and remote ports.

• Port-forwarding only supports TCP, not UDP.

• The remote address can be an IPv4 address or a hostname - hostnames are resolved in the cluster.

• In regular port forwarding (-L) connections are made lazily and hostname resolution is attempted only data is sent to the local port.

• Reverse forwarding (-R) can read the feature.network.incoming section of a mirrord config file when the file is passed to the command with -f.

This section covers how to manage and administer mirrord in your organization or team. Here, you'll find information on topics such as setting up and using the License Server, monitoring mirrord usage and performance, configuring policies to control access and behavior, working with profiles for different environments or teams, and understanding security considerations. These guides are intended for administrators and advanced users who want to ensure mirrord is deployed, maintained, and governed effectively in production or shared environments.

One way to use mirrord's targetless mode is to set up your browser to use the IP address of the remote target. This way, you can browse the web as if you were in the same location as the remote target. Below is a guide on how to do this with Google Chrome.

Prerequisites

1. Install microsocks (via brew or apt)

2. Install "Socks5 Configurator" Chrome extension

Steps

1. In a terminal session, trigger microsocks using mirrord.

   • If you want to use a specific target's network: mirrord exec -t deployment/my_deployment microsocks

   • If you just want a specific namespace networking: mirrord exec -a namespace microsocks

   • And you can just do: mirrord exec microsocks if you want to use your current namespace.

2. In a Chrome window:

   1. Open the Socks5 Configurator extension

   2. Make sure the "Socks5 Proxy" is enabled

   3. Type in its respective textbox 127.0.0.1:1080

   4. Hit the save button

3. That's it! You can verify your IP address has changed via a quick "what is my ip address" search in Google

What's next?

1. If you'd like to intercept traffic rather than mirror it so that your local process is the one answering the remote requests, check out this guide. Note that you can even filter which traffic you intercept!

2. If your local process reads from a queue, you might want to test out the copy target feature, which temporarily creates a copy of the mirrord session target. With its scaledown flag it allows you to temporarily delete all replicas in your targeted rollout or deployment, so that none competes with your local process for queue messages.

3. If you just want to learn more about mirrord, why not check out our architecture or configuration sections?

What frameworks/languages does mirrord support?

mirrord works by hooking libc, so it should work with any language/framework that uses libc (vast majority).

This includes: Rust, Node, Python, Java, Kotlin, Ruby, and others (most languages use libc).

mirrord also supports for Go, which doesn't use libc

Does mirrord support clusters with a service mesh like Istio or Linkerd?

Yes, mirrord works exactly the same way with and without a service mesh installed.

Does mirrord support OpenShift?

Yes, mirrord works with OpenShift. However, OpenShift usually ships with a default security policy that doesn't let mirrord create pods. To fix this, you would need to tweak your scc settings - more information here. If you'd rather keep the default security policies, we recommend trying out mirrord for Teams.

Does mirrord support binaries that are statically compiled? (Linux)

No, mirrord needs to be able to leverage dynamic linking in order to work. This means static binaries are not supported.

To check a binary, you can use the file <FILE_NAME> command - dynamically linked binaries will look like this:

marvin@heart-of-gold:~$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=36b86f957a1be53733633d184c3a3354f3fc7b12, for GNU/Linux 3.2.0, stripped

And static binaries will look like this:

marvin@heart-of-gold:~/MetalBear$ file some_static_binary 
some_static_binary: ELF 64-bit LSB executable, x86-64, version 1 (GNU/Linux), statically linked, BuildID[sha1]=2e1eda62d5f755377435c009e856cd7b9836734e, for GNU/Linux 3.2.0, not stripped

Sometimes Go is statically compiled by default, so it's important to check and compile dynamically if necessary. See this section in Common Issues for more info.

This section contains answers to frequently asked questions about mirrord. Here you'll find information on common issues, general usage, limitations, and comparisons with other tools. If you encounter a problem or want to better understand how mirrord works, check here for quick solutions and clarifications.

There are several features underlying mirrord's ability to let your local app send outgoing network requests to cluster resources:

1. By importing the remote target's environment variables, your app will send the request to the remote hostnames configured in them.

2. By intercepting DNS resolution, mirrord will resolve the remote hostnames to the remote pod's IP address.

3. Finally, by intercepting outgoing network requests, mirrord will send the request from the remote pod, allowing it to access resources that are only available from within the cluster.

However, sometimes you might have a resource in the cluster that you don't want to access from your local process - perhaps a shared database. This is what the outgoing filter is for. It allows you to specify a list of hostnames that should be resolved accessed remotely, or a list of hostnames that should be resolved and accessed locally. That way, you can run a local instance of your database and have your local process read and write to it, while still running all other operations against the cluster.

For example, if you want your app to access the hostname example-hostname.svc locally, and everything else remotely, you can do it with the following configuration:

{
  "feature": {
    "network": {
      "outgoing": {
        "filter": {
          "local": ["example-hostname.svc"]
        },
      }
    }
  }
}

You can see all the configuration options for the outgoing filter feature .

What's next?

1. If you'd like to intercept traffic rather than mirror it so that your local process is the one answering the remote requests, check out this guide. Note that you can even filter which traffic you intercept!

2. If you don't want to impersonate a remote target - for example, if you want to run a tool in the context of your cluster - check out our guide on the targetless mode.

3. If you just want to learn more about mirrord, why not check out our architecture or configuration sections?

When you set the configuration field, instead of using the of the run directly, mirrord will create a new pod using the pod spec of the original target, and use that new pod as a target.

This can be useful when you want to run your application with access to the resources and I/O of a target that isn't reliable, for example because the target pod keeps crashing, or because it is managed by a and might terminate before you are done debugging your application with mirrord.

Health Checks

The new, copied pod will not have any even if the original pod spec does define them. This means you can traffic without having to also answer those probes. This might come in handy when debugging with breakpoints with stolen traffic. Without copy_target, if you linger too long on a breakpoint, the application might miss some probes, which could cause a target pod to restart.

Replacing a Whole Workload Using scale_down

When the option is set, mirrord will the target workload down to zero, effectively replacing all existing pods of that workload by the one new copied pod, that is then used as the target for the mirrord run. This feature is supported with Deployment, Argo Rollout, StatefulSet, and ReplicaSet (owned by either a Deployment or an Argo Rollout) targets.

The scale down feature can be useful e.g. when a workload reads from a queue. By scaling it down to zero, the application you run with mirrord does not have to compete with the workload's pods for queue items.

Only one mirrord session can scale down a workload at the same time. If you try to scale down a workload that is already being scaled down in another mirrord session (by you or by a teammate), mirrord will display an error and exit.

You can see active copied targets by running mirrord operator status. When there are no active copy targets, the relevant part of the output will say "No active copy targets".

When there are active copy targets, the relevant section of the output will look like this:

With an asterisk marking copy targets that are also scaling down their original target.

Please note however that you don't necessarily have to check if a target is already being scaled down, as trying to scale it down again will not interrupt the ongoing session, it will just result in your new run exiting with an error.

The core value of mirrord is that it cuts iteration time by letting developers run their code against the cluster directly, instead of having to build, push and deploy images. This significantly cuts down iteration time by letting developers test their code in the cloud from the very first step of the development cycle. However, in order to properly test new code in the cloud, it needs to be able to not only read or receive traffic from the environment, but also to write or send traffic to it, potentially mutating it. This discussion is only relevant for users on the Team and Enterprise pricing plans.

This raises the question, what if I want multiple users in my organization to use the same cluster (e.g. the organization's staging cluster) concurrently? Wouldn't they step on each other toes and affect each other's work?

1. If one developer steals traffic from a remote service, wouldn't that prevent other users from stealing or mirroring traffic from that same service?

2. If a service reads from a queue, wouldn't a developer targeting it with mirrord steal all the messages from the queue, preventing other developers from reading them?

3. If a developer writes to a database, wouldn't that affect the data that other developers see when they read from the same database?

These conflicts and more are resolved by the mirrord Operator, available in the mirrord Team and Enterprise plans. By having a persistent, centralized component in the cluster that can synchronize and orchestrate different instances of mirrord running in the cluster, we can allow developers to use mirrord against the same cluster without affecting each other.

What capabilities does mirrord have to allow concurrent usage of the same cluster?

1. Concurrently debug the same HTTP server with HTTP filters

mirrord's HTTP filters let users only steal a subset of the incoming traffic to the remote service. By adding personalized headers to incoming traffic and then configuring mirrord to only steal traffic with those headers, users can debug the same service concurrently without affecting each other. .

NOTE: While HTTP filters are supported in the OSS version of mirrord, concurrently debugging the same service using HTTP filters is only supported in the Team and Enterprise versions.

2. Concurrently debug the same queue-based service with queue splitting

mirrord's queue splitting feature lets users only steal a subset of the messages from a queue. By configuring mirrord to only steal messages with specific properties, users can debug the same queue-based service concurrently without affecting each other. .

3. Prevent unwanted behavior with mirrord Policies

mirrord Policies let you define rules that prevent users from doing certain actions. For example, you can prevent users from writing to a database, or from stealing traffic without using an HTTP filter. .

4. Communicate with local components using the outgoing traffic filter

Sometimes a database is just too sensitive to write to remotely. Or maybe you want to test a migration, and don't want it to affect your coworkers who are using the same cluster. In these cases, you can use the outgoing traffic filter to send traffic to a locally running component instead of the one that's running in the cluster. Your local process will still communicate with all of its other dependencies remotely in the cluster. .

5. View other sessions running in the cluster (and kill them if necessary)

Sometimes, all you need to avoid clashes is just to see what other users are doing in the cluster. The mirrord operator status command displays a list of all the currently running sessions in the cluster, along with the user who started them. If you see a session that's causing problems, you can kill it using the mirrord operator kill command (given you have the necessary permissions). .

Conclusion

Even though using mirrord with a shared cluster is already safer than actually deploying your code to it, we're constantly working to make it even safer and more seamless for multiple users to use mirrord concurrently on the same environment. If you have any questions or suggestions, please don't hesitate to reach out to us or on our . Happy mirroring!

Whenever a user starts mirrord on a cluster where mirrord for Teams is installed, the Operator assigns a session to this user, until they stop running mirrord, at which point the session is closed in the Operator automatically.

See active Operator sessions

Users can use the command mirrord operator status to see active sessions in the cluster. For example, in the following output, we can see the session ID, the target used, the namespace of the target, the session duration, and the user running that session. We can also see that Ports is empty, meaning the user isn't stealing or mirroring any traffic at the moment.

The User field is generated in the following format - whoami/k8s-user@hostname. whoami and hostname are from the local machine, while k8s-user is the user we see from the operator side.

In this example, we can see that the session has an active steal on port 80, filtering HTTP traffic with the following filter: X-PG-Tenant: Avi.+

Stop active Operator sessions

Users may also forcefully stop a session with the mirrord operator session CLI commands. These allow users to manually close Operator sessions while they're still alive (user is still running mirrord).

The session management commands are:

• mirrord operator session kill-all which will forcefully stop ALL sessions!

• mirrord operator session kill --id {id} which will forcefully stop a session with id, where you may obtain the session id through mirrord operator status;

sessions RBAC

Every mirrord-operator-user has access to all session operations by default, as they come with deletecollection and delete privileges for the sessions resource. You may limit this by changing the RBAC configuration. Here is a sample role.yaml with the other Operator rules omitted:

• mirrord operator session kill-all requires the deletecollection verb;

• mirrord operator session kill --id {id} requires the delete verb;

Why not just use a remote debugger?

When you use a remote debugger, you still have to deploy new code to the cluster. When you plug local code into the cloud with mirrord, you don't have to wait for cloud deployment. Using mirrord is also less disruptive to the cluster, since the stable version of the code is still running and handling requests.

Why not just run a copy of the cluster on my machine with e.g. minikube?

Our assumption is that some environments are too complex to run wholly on your local machine (or their components are just not virtualizable). If that's the case with your environment, you can only run the microservice you're currently working on locally, but connect it to your cloud environment with mirrord. Note that mirrord can also be used to connect your non-containerized process to your local Kubernetes cluster.

How is mirrord different from Telepresence?

mirrord can be a great alternative to Telepresence. The main differences are:

• mirrord works on the process level, meaning it doesn't require you to run a "daemon" locally and it doesn't change your local machine settings. For example, if you run another process, it won't be affected by mirrord.

• This means that you can run multiple services at the same time, each in a different context and without needing to containerize them.

• mirrord doesn't require you to install anything on the cluster.

• mirrord duplicates traffic and doesn't intercept/steal it by default.

• mirrord can be run through one of our IDE extensions: we support and .

More details can be found in this

The common way to use mirrord is on a locally running process. This way you can easily debug it in your IDE, as well as make quick changes and test them out without going through the additional layer of containerization.

However, sometimes you're just not able to run your microservice locally - usually due to complicated dependencies. For these cases, you can run mirrord on a local container instead. To do this, simply run the following command:

For example:

In addition to Docker, Podman and nerdctl are also supported.

Local container execution is currently only supported in the mirrord CLI tool. IDE extension support will be added in the future.

What's next?

1. If you'd like to intercept traffic rather than mirror it so that your local process is the one answering the remote requests, check out . Note that you can even filter which traffic you intercept!

2. If you don't want to impersonate a remote target - for example, if you want to run a tool in the context of your cluster - check out our .

3. If you just want to learn more about mirrord, why not check out our or sections?

The license server enables you to manage your organization’s seats without sending any data to mirrord’s servers. It can aggregate license metrics from multiple operators (useful if you’re running mirrord across multiple clusters) and provides visibility into seat usage across your organization.

Basic Setup

The license server is installable via Helm. First, add the MetalBear Helm repository:

Next, save the following yaml as values.yaml on your machine.

Fill in the license.key and license.pem fields according to the following guidelines:

• License key - Can be any string of your choosing. We recommend using random characters or a UUID.

• License file - Must be a valid operator license. This can also be a secret under the license.pem key.

You can customize the license server deployment further - all values.yaml configuration options can found

NOTE: The license server needs to be accessible to any mirrord operators you want to track. To that end, the default value for service.type is ClusterIP, but can be changed to NodePort or LoadBalancer, according to your requirements.

Next, install the license server on your cluster:

To make sure it's been installed successfully and is running:

If your operator(s) are running at on a different cluster, make sure the mirrord-operator-license-server service is exposed to them via ingress.

Connecting Operators to the License Server

First update your operator for quickstart helm setup for operator) file:

NOTE: The server value must contain the protocol and the prefix for any ingress that the the license server can be behind.

Then run:

By default, mirrord mirrors all incoming traffic into the remote target, and sends a copy to your local process. This is useful when you want the remote target to answer requests, keeping the remote environment completely agnostic to your local code. However, sometimes you do want to test out how your local code responds to requests; or maybe your process writes to a database when receiving a request, and you want to avoid duplicate records (one from your local code, one from the remote target). In these cases, you probably want to steal traffic instead of mirroring it. When you steal traffic, your local process is the one answering the requests, and not the remote target. This guide will show you how to do that.

Stealing all of the remote target's traffic

If you want all traffic arriving at the remote target to be redirected to your local process, change the feature.network.incoming configuration to steal:

{
  "feature": {
    "network": {
      "incoming": "steal"
    }
  }
}

Run your process with mirrord using the steal configuration, then send a request to the remote target. The response you receive will have been sent by the local process. If you're using one of our IDE extensions, set a breakpoint in the function handling the request - your request should hang when the breakpoint is hit and until you continue the process.

Stealing only a subset of the remote target's traffic

For incoming HTTP traffic (including HTTP2 and gRPC), mirrord also supports stealing a subset of the remote target's traffic. You can do this by specifying a filter on either an HTTP header or path. To specify a filter on a header, use the feature.network.incoming.http_filter.header_filter configuration:

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "X-My-Header: my-header-value",
          "ports": [80, 8080]
        }
      }
    }
  }
}

The feature.network.incoming.http_filter.ports configuration lets mirrord know which ports are listening to HTTP traffic and should be filtered. It defaults to [80, 8080].

To specify a filter on a path, use the feature.network.incoming.http_filter.path_filter configuration:

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "my/path",
          "ports": [80, 8080]
        }
      }
    }
  }
}

Note that both header_filter and path_filter take regex value, so for example "header_filter": "X-Header-.+: header-value-.+" would work.

Filtering out healthchecks using a negative look-ahead

The HTTP filters both take "fancy" regexes that support negative look-aheads. This can be useful for avoiding the stealing of Kubernetes liveness, readiness and startup probes.

For filtering out any probes sent to the application by kubernetes, you can use this header filter, to require a user-agent that does not start with "kube-probe":

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "header_filter": "^User-Agent: (?!kube-probe)"
        }
      }
    }
  }
}

To avoid stealing requests sent to URIs starting with "/health/", you can set this filter:

{
  "feature": {
    "network": {
      "incoming": {
        "mode": "steal",
        "http_filter": {
          "path_filter": "^(?!/health/)"
        }
      }
    }
  }
}

Stealing HTTPS traffic with a filter

feature.network.incoming.http_filter allows you to steal a subset of HTTP requests. To apply the filter, the mirrord-agent needs to be able to parse the requests stolen from the target. Most commonly, the incluster traffic is encrypted with TLS, but it is decrypted by a service mesh before it gets to the target service. In this case, mirrord is able to parse the requests out of the box.

However, in some cases the traffic is only decrypted by the target service itself. Using an HTTP filter in this case requires some additional setup. Check out the HTTPS stealing guide for more information. Note that this HTTPS stealing requires mirrord Operator, which is part of mirrord for Teams.

What's next?

1. If your local process reads from a queue, you might want to test out the copy target feature, which temporarily creates a copy of the mirrord session target. With its scaledown flag it allows you to temporarily delete all replicas in your targeted rollout or deployment, so that none competes with your local process for queue messages.

2. If you don't want to impersonate a remote target - for example, if you want to run a tool in the context of your cluster - check out our guide on the targetless mode.

3. If you just want to learn more about mirrord, why not check out our architecture or configuration sections?

The installation of the mirrord operator defines two types of custom resources in your cluster: MirrordClusterProfile and MirrordProfile.

These resources provide a unified base for users' mirrord configurations. Users can reference a profile in their mirrord config, and mirrord will apply the defined feature adjustments accordingly.

Cluster-Wide Profiles (MirrordClusterProfile)

MirrordClusterProfile is a cluster-scoped custom resource that provides a shared base configuration available to all namespaces in the cluster.

apiVersion: profiles.mirrord.metalbear.co/v1alpha
kind: MirrordClusterProfile
metadata:
  # This name can be referenced by the user in their mirrord configs.
  name: example-cluster-profile
spec:
  # A list of adjustments to be made in the user's feature config.
  #
  # The adjustments are applied in order.
  featureAdjustments:
    # Incoming traffic will be stolen.
    - change: incoming-steal
    # All outgoing traffic will be remote.
    - change: outgoing-remote
    # All DNS resolution will be remote.
    - change: dns-remote

Namespaced Profiles (MirrordProfile)

MirrordProfile is a namespaced custom resource. These profiles are defined within a specific namespace and are only available to workloads running in that namespace.

apiVersion: profiles.mirrord.metalbear.co/v1alpha
kind: MirrordProfile
metadata:
  name: example-profile
  namespace: example-namespace
spec:
  featureAdjustments:
    - change: incoming-steal
    - change: outgoing-remote
    - change: dns-remote

Allowed Feature Adjustments

The complete list of allowed values for the featureAdjustments.[].change field is as follows:

1. incoming-mirror - incoming traffic will be mirrored

2. incoming-steal - incoming traffic will be stolen

3. incoming-off - incoming traffic will not be intercepted

4. dns-remote - all DNS resolution will be remote

5. dns-off - all DNS resolution will be local

6. outgoing-remote - all outgoing traffic will be remote

7. outgoing-off - all outgoing traffic will be local

Selecting a profile

Starting from mirrord version 3.136.0, the user can select a mirrord profile in their mirrord config. The profile is referenced by its name.

{
    "profile": "example-profile"
}

Enforcing profiles

Use of mirrord profiles can be enforced with mirrord policies.

Important: mirrord profiles are applied to the session on the user machine, and should not be used as security features.

The mirrord Browser Extension automatically injects an HTTP header into all your browser requests while a mirrord session is running. This solves the hassle of manually adding headers when debugging local services that rely on header-based routing, making it easier to test production-like flows in your browser with zero manual setup.

Prerequisites

Before you start, make sure you have:

1. Header propagation set up in your app.

2. Google Chrome is installed.

3. mirrord Browser Extension for Chrome installed.

4. A valid HTTP header filter defined in your mirrord.json under feature.network.incoming.http_filter.header_filter with mode set to steal.

5. Browser extension config enabled in your mirrord.json. Note: This feature is experimental.

   Copy

   {
   "feature": {
       "network": {
       "incoming": {
           "mode": "steal",
           "http_filter": {
           "header_filter": "X-My-Header: my-header-value"
           }
       }
       }
   },
   "experimental": {
       "browser_extension_config": true
   }
   }

Using mirrord Browser Extension

1. Run mirrord exec with the configured mirrord.json, mirrord will then:

   • Prints the configuration URL to the screen.

   • Opens the URL automatically in a Chrome tab.

2. The extension injects the active session’s header into all browser requests across tabs.

3. You can check the current header and status in the extension popup, the extension can be opened anytime from the Chrome extension icon

4. To stop the extension from injecting HTTP request header, click the extension icon and remove the header from the pop up menu.

Header Filter

The mirrord Browser extenstion will automatically inject the header_filter defined in mirrord.json. In case header_filter configured with regex pattern, you will be prompted in the browser to enter a header that matches it: Please enter a header that matches pattern $HEADER_PATTERN

More details

• In case the mirrord browser extension is not downloaded prior to running mirrord exec with the configured mirrord.json, the URL will fail to open. Google Chrome will display an error page showing the URL is blocked by Google Chrome.

• In case the Browser Extension is enabled in mirrord.json, but no http header filter is configured, mirrord will not initiate the extension in Google Chrome and display a warning in the terminal.

Overview

mirrord lets you run a local process in the context of remote environment i.e. environment variables present in the remote pod will be loaded into the local process.

For example, if you want your local process to access a remote database, the connection string configured in the remote pod's environment variable can be used by your local process.

How does it work?

mirrord-layer sends a message to mirrord-agent requesting remote environment variables, which are then set before the local process starts.

Usage

To include/exclude environment variables selectively, use the --override-env-vars-include flag to include and --override-env-vars-exclude to exclude with environment variables specified in a semicolon separated list.

Note: These flags are mutually exclusive. For example, if one chooses to exclude using the --override-env-vars-exclude flag, then there is no need to use --override-env-vars-include="*" to include all other environment variables.

By default, all environment variables are included.

Example

If on our target pod, we have the environment variable ENV_VAR1 with the value remote-value and on our local machine we have ENV_VAR1 with value local-value, then Running the python interpreter with mirrord would look like this:

MIRRORD_AGENT_IMAGE=test MIRRORD_AGENT_RUST_LOG=trace RUST_LOG=debug target/debug/mirrord exec -c --target pod/py-serv-deployment-ff89b5974-x9tjx python3

Python 3.9.13 (v3.9.13:6de2ca5339, May 17 2022, 11:23:25)
[Clang 6.0 (clang-600.0.57)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import os
>>> print(os.environ['ENV_VAR1'])
remote-value

Logs

❯ MIRRORD_AGENT_IMAGE=test MIRRORD_AGENT_RUST_LOG=trace RUST_LOG=debug target/debug/mirrord exec -c --target pod/py-serv-deployment-ff89b5974-x9tjx python3
...
2022-07-01T17:18:33.744996Z DEBUG mirrord_layer: ClientMessage::GetEnvVarsRequest codec_result Ok(
    (),
)
2022-07-01T17:18:33.754270Z DEBUG mirrord_layer: DaemonMessage::GetEnvVarsResponse Ok(
    {
        "KUBERNETES_PORT": "tcp://10.96.0.1:443",
        "LANG": "C.UTF-8",
        "KUBERNETES_SERVICE_PORT": "443",
        "PY_SERV_PORT": "tcp://10.96.139.36:80",
        "KUBERNETES_PORT_443_TCP": "tcp://10.96.0.1:443",
        "PY_SERV_SERVICE_PORT": "80",
        "KUBERNETES_SERVICE_PORT_HTTPS": "443",
        "PYTHON_SETUPTOOLS_VERSION": "58.1.0",
        "PY_SERV_PORT_80_TCP_ADDR": "10.96.139.36",
        "PYTHON_GET_PIP_SHA256": "ba3ab8267d91fd41c58dbce08f76db99f747f716d85ce1865813842bb035524d",
        "ENV_VAR1": "remote-value",
        "KUBERNETES_SERVICE_HOST": "10.96.0.1",
        "KUBERNETES_PORT_443_TCP_PORT": "443",
        "HOSTNAME": "py-serv-deployment-ff89b5974-x9tjx",
        "KUBERNETES_PORT_443_TCP_ADDR": "10.96.0.1",
        "GPG_KEY": "E3FF2839C048B25C084DEBE9B26995E310250568",
        "PYTHON_GET_PIP_URL": "https://github.com/pypa/get-pip/raw/6ce3639da143c5d79b44f94b04080abf2531fd6e/public/get-pip.py",
        "PY_SERV_PORT_80_TCP": "tcp://10.96.139.36:80",
        "KUBERNETES_PORT_443_TCP_PROTO": "tcp",
        "PYTHON_VERSION": "3.9.13",
        "PY_SERV_PORT_80_TCP_PROTO": "tcp",
        "PY_SERV_PORT_80_TCP_PORT": "80",
        "PY_SERV_SERVICE_HOST": "10.96.139.36",
        "PYTHON_PIP_VERSION": "22.0.4",
    },
)!
...

mirrord is composed of the following components:

• mirrord-agent - Rust binary that is packaged as a container image. mirrord-agent runs in the cloud and acts as a proxy for the local process.

• mirrord-layer - Rust dynamic library (so/dylib) that loads to the local process, hooks its filesystem, network APIs and relays them to the agent.

• mirrord-cli - Rust binary that wraps the behavior of mirrord-layer in a user friendly CLI.

• VS Code extension - Exposes the same functionality as - mirrord-cli within the VS Code IDE.

• IntelliJ plugin - Exposes the same functionality as - mirrord-cli within the IntelliJ IDEs.

mirrord-agent

mirrord-agent is a Kubernetes job that runs in the same Linux namespace as the pod being impersonated in the cluster. This lets the mirrored-agent sniff the network traffic and gain access to the filesystem of the impersonated pod. It then relays file operations from the local process to the impersonated pod and incoming traffic from the impersonated pod to the local process. Outgoing traffic is intercepted at the local process and emitted by the agent as if originating from the impersonated pod. The connection between the agent and the impersonated pod is terminated if the agent pod hits a timeout.

mirrord-agent does not run as a privileged container in the cluster. However, it requires some Linux capabilities to be able to impersonate the targeted pod. These capabilities are:

• CAP_NET_ADMIN and CAP_NET_RAW - required for modifying routing tables

• CAP_SYS_PTRACE - required for reading target pod environment

• CAP_SYS_ADMIN - required for joining target pod network namespace

However, you can disable any subset of those in the configuration using option. This will possibly limit mirrord functionalities or even make it unusable in some setups.

mirrord-layer

mirrord-layer is a .dylib file for OSX systems and .so file on Linux distributions. mirrord-layer is loaded through LD_PRELOAD/DYLD_INSERT_LIBRARIES environment variable with the local process, which lets mirrord-layer selectively override libc functions. The overridden functions are then responsible for maintaining coordination between the process and incoming/outgoing requests for network traffic/file access. mirrord-layer sends and receives events from the agent using port-forwarding.

mirrord-cli

mirrord-cli is a user friendly interface over the essential functionality provided by mirrord-layer. When you run mirrord-cli, it runs the process provided as an argument with mirrord-layer loaded into it.

VS Code Extension

mirrord’s VS Code extension provides mirrord’s functionality within VS Code’s UI. When you debug a process with mirrord enabled in VS Code, it prompts you for a pod to impersonate, then runs the debugged process with mirrord-layer loaded into it.

IntelliJ Plugin

mirrord’s IntelliJ Plugin provides mirrord’s functionality within the IntelliJ UI. When you debug a process with mirrord enabled in IntelliJ, it prompts you for a pod to impersonate, then runs the debugged process with mirrord-layer loaded into it.

Overview

mirrord will relay file access (except for some exceptions) to the target pod by default. (this functionality can be disabled using --fs-mode local flag on the command line or by setting mode in the configuration file in the IDE plugin.)

For example, the following python script calls the built-in open function which translate to something like openat(AT_FDCWD, "/tmp/test", O_RDWR|O_CLOEXEC) at a lower level:

with open("/tmp/test", "r+") as rw_file:
    read_str = rw_file.read(42)
print(read_str)

When we run that python script with mirrord:

mirrord exec -c --target py-serv-deployment-cfc458fd4-bjzjx python3 test.py

mirrord overrides that openat call and opens /tmp/test on the remote pod.

How does it work?

Once a request to open a new file is received by mirrord-agent from mirrord-layer, the agent forwards the request to the container in the remote pod in context of the provided path for the open system call, prefixed with path to the root directory of the container.

mirrord-agent uses APIs provided by docker and containerd runtimes to get the PID of the remote container, and refers to the root directory of the remote container through /proc/container_pid/root

Syscalls

mirrord overrirdes calls to the following libc functions/system calls:

open

int open(const char *pathname, int flags);

Open files on the remote pod. Functionality when opening with different types of paths might differ. In the case when pathname is specified to be a relative path, the call to open is sent to libc instead of the remote pod.

Example:

import os
fd = os.open("/tmp/test", os.O_WRONLY | os.O_CREAT)

openat

int openat(int dirfd, const char *pathname, int flags);

openat works the same as open when dirfd is specified as AT_FDCWD or if the path is absolute. If a valid dirfd is provided, files relative to the directory referred to by the dirfd can be opened.

Example:

dir = os.open("/tmp", os.O_RDONLY | os.O_NONBLOCK | os.O_CLOEXEC | os.O_DIRECTORY)

os.open("test", os.O_RDWR | os.O_NONBLOCK | os.O_CLOEXEC, dir_fd=dir)

read

ssize_t read(int fd, void *buf, size_t count);

Read from a file on the remote pod.

Example:

fd = os.open("/tmp/test, os.O_RDWR | os.O_NONBLOCK | os.O_CLOEXEC)
read = os.read(fd, 1024)

write

ssize_t write(int fd, const void *buf, size_t count);

Write to a file on the remote pod.

Example:

with open("/tmp/test", "w") as file:
    file.write(TEXT)

lseek

off_t lseek(int fd, off_t offset, int whence);

Reposition the file offset of an open file on the remote pod. lseek through mirrord-layer supports all valid options for whence as specified in the Linux manpages.

Example:

with open("/tmp/test", "w") as file:
    file.seek(10)
    file.write(TEXT)

Note: For read, write, and lseek if the provided fd is a valid file descriptor i.e. it refers to a file opened on the remote pod then the call is forwarded to the remote pod, otherwise the call is sent to libc.

If you develop your application in Visual Studio Code, you can debug it with mirrord using our Visual Studio Marketplace . Simply:

1. Download the extension

2. Enable mirrord using the "mirrord" button on the bottom toolbar

3. Run or debug your application as you usually do

When you start a debugging session with mirrord enabled, you'll be prompted with a target selection quick pick. This quick pick will allow you to select the target in your Kubernetes cluster that you want to impersonate.

The toolbar button enables/disables mirrord for all run and debug sessions.

mirrord's initial state on startup can be configured in the VSCode settings:

Enabling/disabling mirrord for a specific launch configuration

mirrord can be persistently enabled or disabled for a specific launch configuration, regardless of the toolbar button state. This is controlled via the MIRRORD_ACTIVE environment variable in your launch configuration. The value "1" keeps mirrord always enabled, while the value "0" disables it.

Selecting session target

mirrord's target can be specified in two ways:

1. with the target selection quick pick

   • The quick pick will only appear if the mirrord config does not specify the target.

   • The quick pick will initially show targets in the namespace specified in the mirrord config (). If the namespace is not specified, your Kubernetes user's default namespace will be used.

   • If you want to see targets in a different namespace, there is an option to "Select Another Namespace".

2. in the mirrord config's

Using the mirrord config

The extension allows for using the . For any run/debug session, the mirrord config to be used can be specified in multiple ways:

Active config

The toolbar dropdown menu allows for specifying a temporary mirrord config override. This config will be used for all run/debug sessions.

To specify the override, use Select active config action.

You will be prompted with a quick pick where you can select a mirrord config from your project files. For the file to be present in the dialog, it must either be located in a directory which name ends with .mirrord, or have a name that ends with mirrord. Accepted config file extensions are: json, toml, yml and yaml.

You can remove the override using the same action.

Config for launch configuration

If no active config is specified, the extension will try to read the config file path from the MIRRORD_CONFIG_FILE environment variable specified in the launch configuration.

This path should be absolute.

Config from default path

If the config file path is not specified in the launch configuration environment, the plugin will try to find a default config.

The default config is the lexicographically first file in <PROJECT ROOT>/.mirrord directory that ends with mirrord. Accepted config file extensions are: json, toml, yml and yaml.

Managing the mirrord binary

The extension relies on the standard mirrord CLI binary.

By default, the extension checks the latest release version and downloads the most up-to-date binary in the background. You can disable this behavior in the VSCode settings:

You can also pin the binary version with:

To use a specific mirrord binary from your filesystem:

WSL

The guide on how to use the extension with remote development on WSL can be found .

What does mirrord actually do?

First and most important, mirrord doesn't just mirror traffic. It does that, but also a lot more.

mirrord lets you connect a process on your development machine to your Kubernetes cluster. It does this by injecting itself into the local process (no code changes needed!), intercepting all of the input and output points of the process - network traffic, file access, and environment variables - and proxying them to the cluster. This mechanism is discussed in more detail here.

When you run mirrord, you select a Target - this is the Kubernetes Pod or Deployment whose context you want your local code to run in. For example, if you have a staging cluster running the latest stable version of all of your microservices, and you're now coding the next version of one of these microservices, you'd select as your Target the Pod or Deployment running the stable version of that microservice in staging. The following things will then happen:

• The Target's environment variables will be made available to the local process.

• When the local process tries to read a file, it will be read from the Target's filesystem instead.

• Traffic reaching the remote Target will reach your locally running process (this incoming traffic can either be mirrored, intercepted entirely, or intercepted based on a filter you define).

• Traffic sent out from your local process will be sent out from the Target instead, letting it reach any endpoint that's accessible to the Target, and the response will be sent back to your local process.

By proxying all of your local process' input and output points in this way, mirrord makes it "think" it's running in the cloud, which lets you test it in cloud conditions:

1. Without having to run your entire architecture locally

2. Without going through lengthy CI and deployment processes

3. Without deploying untested code to the cloud environment - the stable version of the code is still running in the cluster and handling requests - letting multiple users test on the same cluster without queueing to use it or breaking the cluster for everyone else.

Is mirrord free?

mirrord is free and open source (MIT License). Our paid offering, mirrord for Teams, includes a Kubernetes operator that acts as a control plane for mirrord. You can read more about it here.

Can I intercept traffic instead of duplicating it?

Yes, you can use the --steal flag to intercept traffic instead of duplicating it.

Does mirrord install anything on the cluster?

No, mirrord doesn't install anything on the cluster, nor does it have any persistent state. It does spawn a short-living pod/container to run the proxy, which is automatically removed when mirrord exits. mirrord works using the Kubernetes API, and so the only prerequisite to start using mirrord is to have kubectl configured for your cluster.

If you have any restrictions for pulling external images inside your cluster, you have to allow pulling of ghcr.io/metalbear-co/mirrord image.

How does mirrord protect against disrupting my shared environment with my local code?

• By letting you mirror traffic rather than intercept it, the stable version of the code can still run in the cluster and handle requests.

• By letting you control which functionality runs locally and which runs in the cloud, you can configure mirrord in the way that's safest for your architecture. For example, you can configure mirrord to read files and receive incoming traffic from the cloud, but write files and send outgoing traffic locally. Our main goal in future versions of mirrord is to reduce the risk of disruption of the shared environment when using mirrord. This will be achieved by providing more granular configuration options (for example, filtering traffic by hostname or protocol), and advanced functionality like copy-on-write for databases.

Can I use mirrord to run a local container, rather than a local process, in the context of the remote Kubernetes cluster?

Yes! You can use the mirrord container command to run a local container in the context of the remote Kubernetes cluster. You can read more about it here.

What if I can't create containers with the capabilities mirrord requires in my cluster?

mirrord works by creating an agent on a privileged pod in the remote cluster that accesses another pod's namespaces (read more about it here). If you can't give your end users permissions to create pods with the capabilities mirrord needs, we suggest trying out mirrord for Teams. It adds a Kubernetes operator that acts as a control plane for mirrord clients, and lets them work with mirrord without creating pods themselves. If mirrord for Teams doesn't work for you either, let us know and we'll try to figure a solution that matches your security policies.

What kinds of Kubernetes objects can I use as a remote target?

mirrord OSS supports the following Kubernetes objects as targets:

• Pods

• Deployments

• Argo Rollouts

In mirrord OSS, mirrord will always target a random pod when a workload with multiple pods is used as the remote target.

mirrord for Teams adds support for the following workloads:

• Jobs

• CronJobs

• StatefulSets

In mirrord for Teams, mirrord will always target all pods when a workload with multiple pods is used as the remote target.

Both in mirrord OSS and mirrord for Teams, if you don't name any specific container to be targeted, mirrord will pick the first container from the pod spec. Some containers, like service mesh proxies, will be automatically ignored.

Requirements

mirrord runs on your local machine and in your Kubernetes cluster.

Local Requirements

• MacOS (Intel, Apple Silicon) and Linux (x86_64) are supported for the local machine. Windows users can use mirrord using WSL (IDE plugins supported as well).

• kubectl needs to be configured on the local machine.

Remote Requirements

• Docker or containerd runtime (containerd is the most common). If you'd like support for other runtimes to be added, please let us know by opening an issue on GitHub.

• Linux Kernel version 4.20+

mirrord can be used in three ways:

1. CLI Tool

2. VS Code Extension

3. IntelliJ Plugin

If you're planning to use mirrord for Teams, you'll also need to install the mirrord Operator.

CLI Tool

Installation

To install the CLI, run:

brew install metalbear-co/mirrord/mirrord

or

curl -fsSL https://raw.githubusercontent.com/metalbear-co/mirrord/main/scripts/install.sh | bash

Usage

To use mirrord to plug a local process into a pod/deployment in the cluster configured with kubectl, run:

mirrord exec --target <target-path> <command used to run the local process>

For example:

mirrord exec --target pod/app-pod-01 python main.py

Or, if you'd rather run a local container than a native process, run:

mirrord container --target <target-path> -- <command used to run the local container>

For example:

mirrord container -- docker run nginx

Use mirrord exec --help or mirrord container --help to get all possible commands + arguments.

VS Code Extension

Installation

You can install the extension directly in the IDE (Extensions -> search for 'mirrord'), or download it from the marketplace here.

Usage

To use extension, click the 'Enable mirrord' button in the status bar at the bottom of the window. When you next run a debug session, you'll be prompted with a dropdown listing pods in the namespace you've configured (or the 'default' namespace, if you haven't). Select the pod you want to impersonate, and the debugged process will be plugged into that pod by mirrord.

Configuration

The VS Code extension reads its configuration from the following file: <project-path>/.mirrord/mirrord.json. You can also prepend a prefix, e.g. my-config.mirrord.json, or use .toml or .yaml format. Configuration options are listed . The configuration file also supports autocomplete when edited in VS Code when the extension is installed.

IntelliJ Plugin

Installation

You can install the plugin directly in the IDE (Preferences -> Plugins, search for 'mirrord'), or download it from the marketplace here.

Usage

To use extension, click the mirrord icon in the Navigation Toolbar at the top right of the window. When you next run a debug session, you'll be prompted with a dropdown listing namespaces in your cluster, and then another with pods in the namespace you selected. Select the pod you want to impersonate, and the debugged process will be plugged into that pod by mirrord.

Configuration

The IntelliJ plugin reads its configuration from the following file: <project-path>/.mirrord/mirrord.json. You can also prepend a prefix, e.g. my-config.mirrord.json, or use .toml or .yaml format. Configuration options are listed here.

Operator

To install and use the Operator, you'll need a mirrord for Teams license. You can get one here. The Operator can be installed using the mirrord CLI or Helm. This has to be performed by a user with elevated permissions to the cluster.

mirrord CLI

1. Install the mirrord CLI.

2. Run the mirrord operator setup command. The base of the command is: ​ mirrord operator setup [OPTIONS] | kubectl apply -f - ​ Options:

• --accept-tos You accept terms of service for mirrord-operator

• --license-key The license key for the operator

• (Optional) -f, --file Output Kubernetes definitions to file and not to stdout (instead of piping to kubectl apply -f -)

• (Optional) --namespace Set namespace of mirrord operator (default: mirrord) ​ So the final command should look like ​ mirrord operator setup --accept-tos --license-key <license-key> | kubectl apply -f -

You should now be able to see the mirrord-operator deployment when running kubectl get deployments -n mirrord. Also, when you run mirrord, you'll see the connected to operator step in its progress reports.

Helm

To install the mirrord Operator with Helm, first add the MetalBear Helm repository:

helm repo add metalbear https://metalbear-co.github.io/charts

Then download the accompanying values.yaml:

curl https://raw.githubusercontent.com/metalbear-co/charts/main/mirrord-operator/values.yaml --output values.yaml

Set license.key to your key.

Finally, install the chart:

helm install -f values.yaml mirrord-operator metalbear/mirrord-operator 

Using Internal Registry (Optional)

The use of an internal registry for storing mirrord images is useful for:

1. Reducing startup time of agent and operator.

2. Reducing cost of ingress traffic needed to download the images.

3. Ensuring that even if our registry goes down (we use GitHub) your use of mirrord isn't interrupted.

Copying images

The first step would be to copy the needed images to your internal registry. We recommend using regctl because it has a built in copy command, that supports copying multi-arch images so you can use mirrord on a mixed arm/x64 fleet at ease.

Install regctl:

go install github.com/regclient/regclient/cmd/regctl@latest

or using script:

curl -L https://github.com/regclient/regclient/releases/latest/download/regctl-linux-amd64 >regctl
chmod 755 regctl

Note - you might need to login to the registry (it automatically uses docker login if available)

regctl registry login REGISTRY

Get the operator image relevant to the chart version you want to install:

IMAGE_VERSION=$(helm show chart metalbear/mirrord-operator | grep 'appVersion:' | awk '{print $2}')

Copy the image to your registry

regctl image copy ghcr.io/metalbear-co/operator:$IMAGE_VERSION your-registry/operator:$IMAGE_VERSION

Extract agent version that is used with specific operator version:

AGENT_IMAGE_VERSION=$(regctl image config ghcr.io/metalbear-co/operator:$IMAGE_VERSION | jq -r '.config.Labels."metalbear.mirrord.version"')

Copy agent image to your registry

regctl image copy ghcr.io/metalbear-co/mirrord:$AGENT_IMAGE_VERSION your-registry/mirrord:$AGENT_IMAGE_VERSION

Setting chart to use internal registry

In the operator chart, set the following values:

operator:
  image: ghcr.io/metalbear-co/operator # REPLACE TO YOUR REGISTRY.
agent:
  image:
    registry: ghcr.io/metalbear-co/mirrord # REPLACE TO YOUR REGISTRY.

In the license server chart (if used), set the following values:

server:
  image: ghcr.io/metalbear-co/operator # REPLACE TO YOUR REGISTRY

Note: License server uses same image as operator for simplicity in deployment.

OpenShift

In order to make the operator work with OpenShift, you need to apply the following scc:

kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: scc-mirrord
allowHostPID: true
allowPrivilegedContainer: false
allowHostDirVolumePlugin: true
allowedCapabilities: ["SYS_ADMIN", "SYS_PTRACE", "NET_RAW", "NET_ADMIN"]
runAsUser:
  type: RunAsAny
seLinuxContext:
  type: MustRunAs
users:
- system:serviceaccount:mirrord:mirrord-operator
- system:serviceaccount:mirrord:default

Verifying the Installation

After installing the Operator, you can verify it works by running mirrord operator status. All mirrord clients will now use the Operator instead of doing actions on their own when running against the cluster.

Test it out!

Now that you've installed the CLI tool or one of the extensions, lets see mirrord at work. By default, mirrord will mirror incoming traffic to the remote target (this can be changed in the ), sending a duplicate to the same port on your local process. So if your remote target receives traffic on port 80, your local process will receive a copy of that traffic on that same port (this can also be ).

To test this out, enable mirrord in your IDE plugin and start debugging your process (or execute your process with the mirrord CLI). Send a request to your remote target, and you should see that request arriving at your local process as well!

Note that, by default, the following features are also enabled:

1. Environment variables from the remote target will be imported into your local process

2. When reading files, your local process will read them from the remote target

3. DNS resolution for your local process will happen on the remote target

4. Outgoing traffic sent by your local process will be sent out from the remote target instead, and the response will be sent back to your local process

We find that this configuration works for a lot of use cases, but if you'd like to change it, please read about available options in the .

What's next?

Now that you've tried out mirrord, it's time to get acquainted with its different configuration options and tailor it to your needs:

1. If you'd like to intercept traffic rather than mirror it so that your local process is the one answering the remote requests, check out this guide. Note that you can even filter which traffic you intercept!

2. If your local process reads from a queue, you might want to test out the copy target feature, which temporarily creates a copy of the mirrord session target. With its scaledown flag it allows you to temporarily delete all replicas in your targeted rollout or deployment, so that none competes with your local process for queue messages.

3. If you don't want to impersonate a remote target - for example, if you want to run a tool in the context of your cluster - check out our guide on the targetless mode.

4. If you just want to learn more about mirrord, why not check out our architecture or configuration sections?

Overview

You can specify a target on your cluster for mirrord, giving your local application access to the remote target's network environment, file system and environment variables, according to the configuration. When a target is specified, a mirrord-agent pod will be created on the same node as the target pod. The several kinds of supported targets are detailed below. There are also multiple ways to specify a target for mirrord: you can do it in a configuration file, in an IDE dialog, or in the CLI with an argument or an environment variable.

Possible targets

mirrord OSS supports the following Kubernetes objects as targets:

• Pods

• Deployments

• Argo Rollouts

In mirrord OSS, mirrord will always target a random pod when a workload with multiple pods is used as the remote target.

mirrord for Teams adds support for the following workloads:

• Jobs

• CronJobs

• StatefulSets

In mirrord for Teams, mirrord will always target all pods when a workload with multiple pods is used as the remote target.

Both in mirrord OSS and mirrord for Teams, if you don't name any specific container to be targeted, mirrord will pick the first container from the pod spec. Some containers, like service mesh proxies, will be automatically ignored.

You can specify a target namespace if the target should be found in that namespace instead of the namespace that is currently used by kubectl. See the different interfaces below for possible ways of specifying the target and its namespace.

Specifying a target

There are multiple ways to specify a target. In all the possible interfaces for specifying a target, the basic format is <resource-type>/<resource-name> optionally followed by /container/<container-name>. So for specifying a target without specifying a container you can pass

deploy/<YOUR-DEPLOYMENT-NAME>

e.g. deploy/lolz,

or

pod/<YOUR-POD-NAME>

e.g. pod/lolz-64698df9b7-6plq8,

And for also specifying a container, you just add /container/<CONTAINER-NAME> at the end, e.g. pod/lolz-64698df9b7-6plq8/container/main-container.

Using a configuration file

The target path from the last section is set under the field. The target's namespace can be set under . By default, the namespace currently specified in the local kubeconfig is used.

{
  "target": {
    "path": "pod/lolz-64698df9b7-6plq8/container/main-container",
    "namespace": "lolzspace"
  }
}

Using an IDE's dialog

If you are running one of mirrord's IDE extensions and you didn't specify a target via a configuration file, a dialog will pop up for you to pick a target. If you want to choose a target from a different namespace you can set a target namespace in the configuration file, and the dialog will then contain targets in that namespace. Choose the No Target ("targetless") option in the dialog in order to run without a target.

Using a command line argument

If you are running mirrord from the command line, you can specify the target via -t and its namespace via -n, e.g. mirrord exec -t deploy/lolz -n lolzspace my-app. Values specified by command line arguments will be used even if other values are set in a configuration file or in environment variables.

Using an environment variable

You can set the target using the environment variable MIRRORD_IMPERSONATED_TARGET and the target's namespace using the environment variable MIRRORD_TARGET_NAMESPACE. Values specified by environment variables will be used even if other values are set in a configuration file.

Running without a target

When no target is specified, mirrord will start a targetless agent. That can be useful when you want to connect to services from within the cluster, but you don't have any target that you want to "impersonate" - like when running an external utility or a new microservice. When running targetless, mirrord will forward any connections initiated by your application to be sent out of the cluster, but it will not mirror or steal incoming traffic, as a targetless agent is not connected to any Kubernetes service and does not expose any ports. This means that if your application binds a port and listens on it, that will all happen locally, on your machine. So if you're using a management program that exposes a web interface, you can have it listen for connections on localhost, and connect to remote services in the cluster.

If you're using a mirrord configuration file and want to run targetless, you can either leave the target key out completely or specify it explicitly. Note that if you want to skip the target dialog in the IDE plugins, you have to specify it explicitly. You can do it with the following configuration:

{
  "target": "targetless"
}

In your IDE you can pick the No Target ("targetless") option in the target selection dialog (or just not make a selection). Moreover, you should make sure the environment variable used to specify a target isn't set (or is set to an empty value).

Note: In order to set the namespace the agent is going to be created in, set the agent namespace, not the target namespace. That value can be set via the , the -a CLI argument, or the MIRRORD_AGENT_NAMESPACE environment variable.

If you develop your application in one of the JetBrains' IDEs (e.g PyCharm, IntelliJ or GoLand), you can debug it with mirrord using our JetBrains Marketplace plugin. Simply:

1. Download the plugin

2. Enable mirrord using the toolbar button (next to "mirrord" popup menu)

3. Run or debug your application as you usually do

When you start a debugging session with mirrord enabled, you'll be prompted with a target selection dialog. This dialog will allow you to select the target in your Kubernetes cluster that you want to impersonate.

Note: For some projects, the plugin might not be able to present the target selection dialog.

When this happens, you'll see a warning notification and the execution will be cancelled. You can still use mirrord, but you'll have to specify the target in mirrord config.

This is known to happen with Java projects using the IntelliJ build system.

The toolbar button enables/disables mirrord for all run and debug sessions.

mirrord's initial state on startup can be configured in the plugin settings (Settings -> Tools -> mirrord -> Enable mirrord on startup)

Enabling/disabling mirrord for a specific run configuration

mirrord can be persistently enabled or disabled for a specific run configuration, regardless of the toolbar button state. This is controlled via the MIRRORD_ACTIVE environment variable in your run configuration.

To have mirrord always enabled for the given run configuration, set MIRRORD_ACTIVE=1 in the run configuration's environment variables. To have mirrord always disabled, set MIRRORD_ACTIVE=0.

Selecting session target

mirrord's target can be specified in two ways:

1. with the target selection dialog

   • The dialog will only appear if the mirrord config does not specify the target.

   • The dialog will initially show targets in the namespace specified in the mirrord config (). If the namespace is not specified, your Kubernetes user's default namespace will be used.

   • If you want to see targets in a different namespace, there is a dropdown to choose between namespaces.

2. in the mirrord config's

Using the mirrord config

The plugin allows for using the mirrord config. For any run/debug session, the mirrord config to be used can be specified in multiple ways:

Active config

The toolbar dropdown menu allows for specifying a temporary mirrord config override. This config will be used for all run/debug sessions.

To specify the override, use Select Active Config action.

You will be prompted with a dialog where you can select a mirrord config from your project files. For the file to be present in the dialog, its path must contain mirrord and end with either .json, .yaml or .toml.

You can remove the override using the same action.

Config for run configuration

If no active config is specified, the plugin will try to read the config file path from the MIRRORD_CONFIG_FILE environment variable specified in the run configuration.

This path should be absolute.

Config from default path

If the config file path is not specified in the run configuration environment, the plugin will try to find a default config.

The default config is the lexicographically first file in <PROJECT ROOT>/.mirrord directory that ends with either .json, .yaml or .toml.

Managing the mirrord binary

The plugin relies on the standard mirrord CLI binary.

By default, the plugin checks the latest release version and downloads the most up-to-date binary in the background. You can disable this behavior in the plugin settings (Settings -> Tools -> mirrord -> Auto update mirrord binary).

You can also pin the binary version in the plugin settings (Settings -> Tools -> mirrord -> mirrord binary version).

WSL

The guide on how to use the plugin with remote development on WSL can be found here.

Love using mirrord but need help getting your security team on board? Talk to one of our technical experts!

I'm a Security Engineer evaluating mirrord for Teams, what do I need to know?

• mirrord for Teams is completely on-prem. The only data sent to our cloud is analytics and license verification which can be customized or disabled upon request. The analytics don't contain PII or any sensitive information.

• mirrord does not require root permissions on the user's machine.

• mirrord for Teams uses Kubernetes RBAC, meaning it doesn't add a new attack vector to your cluster.

• Communication between the mirrord client and the mirrord Operator takes place over your existing Kubernetes API. If you’ve configured your cluster to encrypt this communication (as is commonly done), then mirrord for Teams’ client-server communication is encrypted as well.

• mirrord for Teams defines a new CRD that can be used to limit access and use of mirrord, with plans of more fine-grained permissions in the future.

• The operator requires permissions to create a pod with the following capabilities in its Kubernetes namespace:

  • CAP_NET_ADMIN - for modifying routing tables

  • CAP_SYS_PTRACE - for reading the target pod's environment variables

  • CAP_SYS_ADMIN - for joining the target pod's network namespace

• The operator requires exclusions from the following gatekeeper policies:

  • runAsNonRoot - to access target pod's filesystem

  • HostPath volume/Sharing the host namespace - to access target pod's file system and networking

• mirrord doesn't copy remote files or secrets to the local filesystem. The local app only gets access to remote files and secrets in memory, and so they'll only be written to the local filesystem if done by the local app, or if mirrord was explicitly configured to log to files with a log level of debug/trace.

• Missing anything? Feel free to ask us on Discord or [email protected]

Are you SOC2/GDPR compliant?

mirrord for Teams is completely on-prem and doesn't process your customer data, so SOC2 and GDPR don't apply to it.

How do I configure Role Based Access Control for mirrord for Teams?

mirrord for Teams works on top of Kubernetes' built-in RBAC with the following resources, mirrordoperators, mirrordoperators/certificate, targets, and targets/port-locks under the operator.metalbear.co apiGroup. The first two resources are required at a cluster level, and the last two can be allowed at a namespace level.

You can limit a user's ability to use mirrord on specific targets by limiting their access to the target resource. The specific verbs for rules to our resources can be copied from the examples below.

For your convenience, mirrord for Teams includes a built-in ClusterRole called mirrord-operator-user, which controls access to the Operator API. To grant access to the Operator API, you can create a ClusterRoleBinding like this:

In addition, the Operator impersonates any user that calls its API, and thus only operates on pods or deployments for which the user has get permissions.

To see the latest definition, we recommend checking our .

How do I limit user access to a specific namespace?

Create a ClusterRoleBinding between the user and the mirrord-operator-user-basic role, then create a (easiest via Helm chart by specifying roleNamespaces) and bind create RoleBinding in the namespace.

How do I limit user access to a specific target?

If the user doesn't have get access to the targets, then they won't be able to target them with mirrord. However, if you want to allow get access to targets but disallow using mirrord on them, we recommend creating a new role based on the mirrord-operator-user namespaced role above, and adding a resourceNames field to the targets resource. This will limit the user to only using the Operator on the specified targets. For example:

How can I prevent users in my team from stealing or mirroring traffic from a target?

You can define that prevent stealing (or only prevent stealing without setting a filter) and/or mirroring for selected targets. Let us know if there are more features you would like to be able to limit using policies.

How can I prevent users from using mirrord without going through the Operator?

When the mirrord CLI starts, it checks if an Operator is installed in the cluster and uses it if it's available. However, if the user lacks access to the Operator or if the Operator doesn't exist, mirrord attempts to create an agent directly.

To prevent clients from attempting to create an agent without the Operator, you can add the to the mirrord configuration file:

To prevent mirrord clients from directly creating agents at the cluster level, we recommend disallowing the creation of pods with extra capabilities by using . Apply a baseline or stricter policy to all namespaces while excluding the mirrord namespace.

Note: before adding a new Pod Admission Policy, you should make sure it doesn't limit any functionality required by your existing workloads.

By default the in-cluster traffic between the operator and its agents isn't encrypted nor authenticated. To ensure encryption and authentication you can enable TLS protocol for the operator–agent connections. You can do this in the operator by setting agent.tls to true or manually by setting OPERATOR_AGENT_CONNECTION_TLS=true in the operator container environment. TLS connections are supported from agent version 3.97.0.

Security hardening with the mirrord operator

Here is a quick checklist you may wish to follow in order to improve the security posture of your cluster when using the operator:

Enabling TLS

TLS can be enabled between the operator and mirrord agents to encrypt the traffic they send to each other. From the :

By default the in-cluster traffic between the operator and its agents isn’t encrypted nor authenticated. To ensure encryption and authentication you can enable TLS protocol for the operator–agent connections. You can do this in the operator by setting agent.tls to true or manually by setting OPERATOR_AGENT_CONNECTION_TLS=true in the operator container environment. TLS connections are supported from agent version 3.97.0.

Reducing access to the mirrord namespace

Users have no need to access to the namespace where mirrord resources are created. By default, this is the mirrord namespace.

Using a certificate for mirrord APIService

By using either your own certificate or one provided by a certificate manager, you can secure access to mirrord's APIService - you will need to set insecureSkipTLSVerify to false in the mirrord-operator Helm chart.

NB: If you are using a certificate manager, make sure you set up reminders for certificate renewal.

Set up network policies for communication

Access to the operator can be further restricted by setting up in the cluster to limit the operator to communicate only with mirrord agents (this is not possible if running agents in ).

With the mirrord Jira integration you can track how much mirrord has been used for each task on a per-issue basis, both as total time of use and the number of times it was used.

The operator is able to report mirrord session times to the mirrord app installed on your Jira instance, which allows you to view the total time and number of sessions that developers have spent using mirrord for each Jira issue.

Setting up Jira integration

1. Go to the to install the mirrord for Jira app (you must be a Jira admin).

2. In Jira, go to the mirrord admin page under Jira admin settings > Apps > mirrord by MetalBear and choose which projects to display the panel on. If you skip this step, the metrics will not be displayed on any issues in your Jira instance.

3. Update the operator according to the instructions on the admin page.

4. Ensure everything is up to date (mirrord IDE plugins and mirrord binary version, as well as the CLI tool):

To verify that the app in installed properly, navigate to an issue in a project you selected in step 2 and check for the mirrord by MetalBear context panel in the right sidebar.

Using the Jira integration

When viewing an issue in Jira, the number of sessions and total session duration will be shown in a context panel in the right sidebar called mirrord by MetalBear. This panel will be displayed on all issues belonging to the projects that were selected in the admin page.

To use the Jira integration while running mirrord, users must be on a git branch containing the (case sensitive) Jira issue key of the issue they're working on, eg. my-new-branch-KEY-123-latest for issue KEY-123.

Troubleshooting

The operator will emit logs with details upon successful session reporting, or upon encountering an error. When metrics are successfully reported, the operator will emit a DEBUG log with a link to the Jira issue that was updated. If the operator is up to date with the Jira webhook configured but did not successfully report metrics:

• If no branch name was recieved by the operator, a DEBUG log will be emitted. This can happen if a mirrord user is not using the latest version of the mirrord CLI or plugin, or if they are not currently on a git branch.

• If the branch name was present but the operator still fails to report metric to the Jira app, a WARN log will be emitted with more details.

Data security

When a session ends, the session data is sent by the operator via HTTP request to a URL and is not stored by the operator.

The Jira app stores usage data for each issue in . It is not visible to anyone outside those with access to the current Jira instance; the only way to access the data is through the context panel on an issue.

Known Issues

• Metrics reporting will not work for jj users, as jj operates in detached HEAD mode.

• Fetching the user's branch name may rarely be unreliable with the IntelliJ plugin, causing metrics reporting to be skipped. In this case you can use the CLI instead, and in the future users will be able to manually override the branch name in config.

Incoming

mirrord lets users debug incoming network traffic by mirroring or stealing the traffic sent to the remote pod.

Mirroring

mirrord's default configuration is to mirror incoming TCP traffic from the remote pod, i.e. run the local process in the context of cloud environment without disrupting incoming traffic for the remote pod. Any responses by the local process to the mirrored requests are dropped, and so whatever application is running on the remote pod continues to operate normally while the traffic is mirrored to the local process.

Example - user-service a simple Kubernetes deployment and service that stores registered users.

bigbear@metalbear:~/mirrord$ minikube service list
|-------------|-------------------|--------------|---------------------------|
|  NAMESPACE  |       NAME        | TARGET PORT  |            URL            |
|-------------|-------------------|--------------|---------------------------|
| default     | kubernetes        | No node port |
| default     | user-service      |           80 | http://192.168.49.2:32000 |
| kube-system | kube-dns          | No node port |
|-------------|-------------------|--------------|---------------------------|

bigbear@metalbear:~/mirrord-demo$ curl -X POST -H "Content-type: application/json" -d "{\"Name\" : \"Metal\", \"Last\" : \"Bear\"}" http://192.168.49.2:32000/user
{"Last":"Bear","Name":"Metal"}

bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:31000/index.html
<html> <head>USERS</head><body><h1> MetalBear Users</h1><p>[{"Last":"Bear","Name":"Metal"}]</p></body></html>

bigbear@metalbear:~/mirrord$ kubectl get pods
NAME                                        READY   STATUS    RESTARTS      AGE
metalbear-bff-deployment-597cb4f957-485t5   1/1     Running   1 (15h ago)   16h
metalbear-deployment-85c754c75f-6k7mg       1/1     Running   1 (15h ago)   16h

To mirror traffic from remote services to the local development environment, run the services locally with mirrord

Window 1

bigbear@metalbear:~/mirrord-demo$ ../mirrord/target/debug/mirrord exec -c
--no-outgoing --target pod/metalbear-deployment-85c754c75f-6k7mg python3
user-service/service.py 
 * Serving Flask app 'service' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses (0.0.0.0)
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://127.0.0.1:33695
 * Running on http://172.16.0.4:33695 (Press CTRL+C to quit)
 127.0.0.1 - - [08/Sep/2022 15:34:34] "GET /users HTTP/1.1" 200
// ^ Received mirrored traffic from the remote pod

Window 2

bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[{"Last":"Bear","Name":"Metal"}]

Stealing

mirrord can steal network traffic, i.e. intercept it and send it to the local process instead of the remote pod. This means that all incoming traffic is only handled by the local process.

Example - running user-service with mirrord and --tcp-steal on:

Window 1

bigbear@metalbear:~/mirrord-demo$ ../mirrord/target/debug/mirrord exec -c 
--tcp-steal --target pod/metalbear-deployment-85c754c75f-6k7mg 
python3 user-service/service.py 
 * Serving Flask app 'service' (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses (0.0.0.0)
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://127.0.0.1:35215
 * Running on http://172.16.0.4:35215 (Press CTRL+C to quit) 
 127.0.0.1 - - [08/Sep/2022 15:48:40] "GET /users HTTP/1.1" 200 -
 127.0.0.1 - - [08/Sep/2022 15:50:40] "POST /user HTTP/1.1" 200 -
 127.0.0.1 - - [08/Sep/2022 15:50:55] "GET /users HTTP/1.1" 200 -
 127.0.0.1 - - [08/Sep/2022 16:57:51] "POST /user HTTP/1.1" 200 -
 127.0.0.1 - - [08/Sep/2022 16:57:54] "GET /users HTTP/1.1" 200 -
 ^Cbigbear@metalbear:~/mirrord-demo$ 

Window 2

// Before running mirrord with `--tcp-steal`
bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[{"Last":"Bear","Name":"Metal"}]

// After running with mirrord and `--tcp-steal` - local process responds
 instead of the remote
bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[]
bigbear@metalbear:~/mirrord-demo$ curl -X POST -H 
"Content-type: application/json" 
-d "{\"Name\" : \"Mehul\", \"Last\" : \"Arora\"}" http://192.168.49.2:32000/user

{"Last":"Arora","Name":"Mehul"}
bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[{"Last":"Arora","Name":"Mehul"}]
bigbear@metalbear:~/mirrord-demo$ curl -X POST -H 
"Content-type: application/json" 
-d "{\"Name\" : \"Alex\", \"Last\" : \"C\"}" http://192.168.49.2:32000/user
{"Last":"C","Name":"Alex"}
bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[{"Last":"Arora","Name":"Mehul"},{"Last":"C","Name":"Alex"}]

// After sending SIGINT to the local process
bigbear@metalbear:~/mirrord-demo$ curl http://192.168.49.2:32000/users
[{"Last":"Bear","Name":"Metal"}]

Filtering Incoming Traffic by HTTP Headers

Currently only supported in steal mode: mirrord lets you specify a regular expression to filter HTTP requests with. When specified, all the headers of each HTTP request that arrives at the remote target are checked against the regular expression. If any of the headers match, the request will be stolen, otherwise, it will be sent to the remote target. For each Header-Name, Header-Value pair, your regular expression will be matched against Header-Name: Header-Value. For example, the filter MyHeader would match requests with MyHeader in any of their header names or header values. The filter ^X-MyFilter: would match only requests that have a header with the header name X-MyFilter (or x-myfilter or with any other capitalization). The regular expression is evaluated with the fancy_regex rust crate.

Specifying a Filter

An HTTP filter can be specified in the mirrord configuration file by setting the incoming mode to steal and specifying a filter in or .

Setting Custom HTTP Ports

The configuration also allows specifying custom HTTP ports under feature.network.incoming.http_filter.ports. By default, ports 80 and 8080 are used as HTTP ports if a filter is specified, which means that the mirrord agent checks each new connection on those ports for HTTP, and if the connection has valid HTTP messages, they are filtered with the header filter.

Outgoing

mirrord's outgoing traffic feature intercepts outgoing requests from the local process and sends them through the remote pod instead. Responses are then routed back to the local process. A simple use case of this feature is enabling the local process to make an API call to another service in the k8s cluster, for example, a database read/write.

For UDP, outgoing traffic is currently only intercepted and forwarded by mirrord if the application binds a non-0 port and makes a connect call on the socket before sending out messages. Outgoing TCP and UDP forwarding are both enabled by default. It can be controlled individually for TCP and UDP or disabled altogether (see mirrord exec --help).

Note: If the handling of incoming requests by your app involves outgoing API calls to other services, and mirrord is configured to mirror incoming traffic, then it might be the case that both the remote pod and the local process (which receives mirrored requests) make an outgoing API call to another service for the same incoming request. If that call is a write operation to a database, this could lead e.g. to duplicate lines in the database. You can avoid such an effect by switching from traffic mirroring to traffic stealing mode. Alternatively, if the service your application makes an API call to is only reachable from within the Kubernetes cluster, you can disable outgoing traffic forwarding, which would make it impossible for your local process to reach that service.

DNS Resolution

mirrord can resolve DNS queries in the context of the remote pod

Example - calling getaddrinfo to see if the query is resolved:

Python 3.8.10 (default, Jun 22 2022, 20:18:18) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> socket.getaddrinfo('localhost', None)
2022-09-08T17:37:50.735532Z  INFO mirrord_layer::socket::ops: getaddrinfo -> result Ok(
    0x00007f5508004760,
)
[(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('::7074:e00d:557f:0', 0, 0, 97)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('::', 0, 0, 0)), (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_RAW: 3>, 0, '', ('::90bf:f401:0:0', 0, 0, 245652448)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('127.0.0.1', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('127.0.0.1', 0))]
>>> socket.getaddrinfo('user-service', None)
2022-09-08T17:38:17.556108Z  INFO mirrord_layer::socket::ops: getaddrinfo -> result Ok(
    0x00007f5508003610,
)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('10.106.158.180', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_DGRAM: 2>, 17, '', ('10.106.158.180', 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_RAW: 3>, 0, '', ('10.106.158.180', 0))]
>>> 

With mirrord for Teams, you can steal a subset of HTTP requests coming to your target, even if the deployed application receives the traffic encrypted with TLS.

Important: stealing HTTPS with a filter requires mirrord-operator version at least 3.106.0 and mirrord-agent version at least 1.134.0.

Configuring HTTPS stealing in the cluster

To enable mirrord users to steal HTTPS requests with a filter, you must provide the mirrord Operator with some insight into your TLS configuration. This can be done with dedicated custom resources: MirrordTlsStealConfig and MirrordClusterTlsStealConfig. These two resources look and work almost the same. The only exception is that MirrordTlsStealConfig is scoped to the namespace in which you create it, while MirrordClusterTlsStealConfig scopes the whole Kubernetes cluster.

An example MirrordTlsStealConfig resource that configures HTTPS stealing from an example-deploy deployment living in namespace example-deploy-namespace:

Each MirrordTlsStealConfig/MirrordClusterTlsStealConfig resource configures HTTPS stealing for some set of available mirrord targets. With the use of spec.targetPath and spec.selector, you can link one configuration resource to multiple pods, deployments, rollouts, etc.

When the mirrord Operator finds multiple configuration resources matching the session target path and labels, it merges their ports lists. The same port cannot be configured multiple times (extra entries are discarded).

Important: mirrord-agent will search for all files and directories referenced by the config resources in the target container filesystem.

Configuring delivery of stolen HTTPS to your local application

By default, when delivering stolen HTTPS requests to your local application, mirrord uses the original protocol - TLS. The connection is be made from your local machine by an anonymous TLS client that does not verify the server certificate.

This behavior can be configured in your mirrord config with .

The mirrord Operator can produce logs in JSON format that can be digested by most popular logging solutions (DataDog, Dynatrace, etc). To enable JSON logging, set operator.jsonLog to true in the Operator Helm chart values. The log level is INFO by default, and can be changed using the RUST_LOG environment variable in the Operator container, which takes values in the following format: mirrord={log_level} (e.g. mirrord=debug).

Functional Logs

The following logs are written with log level INFO, and can be used for dashboards within monitoring solutions in order to monitor mirrord usage within your organization:

Log messages:

• Target Copied

• Port Stolen

• Port Mirrored

• Port Released

• Session Started

• Session Ended

Fields:

Prometheus

mirrord Operator can expose prometheus metrics if enabled. (default endpoint is :9000/metrics)

Helm

Manual

Exposed metrics

DataDog Dashboard

We offer a DataDog dashboard you can import to track statistics.

Download it

Grafana Dashboard

Alternatively there is a Grafana dashboard you can import to track statistics.

Download it

fluentd

If you are using fluentd you can add a filter to unpack some values from the "log" message

This will expand all the extra fields stored in "log" field.

fluentd + Elasticsearch

Assuming you are using logstash_format true and the connected mapping will store the extra fields in a keyword type, we have a ready made dashboard you can simply import.

Download it (use Saved Objects to import).

List of third-party guides and videos about mirrord:

The installation of the mirrord operator defines two in your cluster: the namespaced MirrordPolicy and the clusterwide MirrordClusterPolicy. With these policies you can limit the use of some features of mirrord for selected targets.

• MirrordPolicy and MirrordClusterPolicy have the exact same specification (spec field);

• MirrordPolicy applies only to targets living in the same namespace;

• MirrordClusterPolicy applies to all targets in the cluster.

Blockable features

Currently the set of blockable features contains:

• steal - prevents stealing traffic from the targeted pods;

• steal-without-filter - prevents stealing traffic from the targeted pods, unless HTTP filter is used;

• mirror - prevents mirroring traffic from the targeted pods.

If you are not using the latest operator version, the set of supported blockable features might be different. In order to see the exact set of features you can block, use the following kubectl command:

Controllable features

Some policies are not for outright blocking features, instead they change behaviour, overriding what the user has set in their mirrord config file.

You may use these features to change which files may be accessed in the target, or which environment variables may be retrieved. These policies should not be relied upon for security, and should instead be considered convenience policies.

env policy

Added in mirrord Operator version 3.103.0

Changes how environment variables may be retrieved from the target, overriding what the user has set in their mirrord.json config file.

• exclude - the environment variables in this list WON'T be retrieved from the target, instead mirrord will either use the locally available env vars (if they exist in the user's machine), or these env vars will be missing completely;

The policy takes priority over a user's mirrord config, which means that if the user has a config:

If the policy is set with exclude: ["*_URL"], then mirror will NOT retrieve env vars that match *_URL, even though the user explicitly wanted that in their config.

If you are not using the latest operator version, the env policy options might be different. In order to see the latest options, use the following kubectl command:

fs policy

Added in mirrord Operator version 3.103.0

Changes file operations behaviour, giving the operator control over which files may be accessed from the target, and in which modes. Overrides what the user has set in their mirrord.json config file.

• readOnly - files that match any of the patterns specified here must be opened as read-only, otherwise the operation will fail;

• local - matching files will be forced to be opened locally, on the user's machine, instead of in the target;

• notFound - any matching files will return a not found error as if the file is not present in the target, even if it exists there;

The policy takes priority over a user's mirrord config, which means that if the user has a config:

If the policy is set with readOnly: [".+\\.json"], and the user tries to open a file that matches this regex in write mode, then mirrord will return an error to the user app, as if the file could not be found, even though the user wanted it to be read_write.

network policy

Added in mirrord Operator version 3.105.0

Allows the operator to control which patterns may be used as . Header filters specified by the user must match the regex specified in the network policy.

If the policy is set with headerFilter: "^username: .+" at least one header filter must match the ^username: .+ regex when user is using the steal mode for incoming traffic.

this also works any of or all of patterns

Important: steal-without-filter will be automatically enabled once any http filter is specified.

profile policy

Added in mirrord Operator version 3.108.0

Allows the operator to enforce using a and to specify a set of allowed profiles.

The example above will enforce that the user selects either my-profile-1 or my-profile-2 for their session.

Important: mirrord profiles are applied to the session on the user machine, and should not be used as security features.

Restricting targets affected by mirrord policies

By default, mirrord policies apply to all targets in the namespace or cluster. You can use a target path pattern (.spec.targetPath) and/or a (.spec.selector) in order to limit the targets to which a policy applies.

The target path of a mirrord run is either targetless or has the form <TARGET_TYPE>/<NAME> followed by an optional /container/<CONTAINER_NAME>, where <TARGET_TYPE> is one of deploy, pod, rollout and statefulset.

Examples for possible target paths:

• deploy/boats

• pod/boats-5fffb9767c-w92qh

• pod/boats-5fffb9767c-w92qh/container/appcontainer

• targetless

By specifying a targetPath pattern in the policy, you limit the policy to only apply to runs that have a target path that matches the specified pattern.

The target path pattern can contain ?, which will match a single character, and *, which will match arbitrarily many characters. For example, "deploy/*" will make a policy apply for any run with a deployment target. "*boats*" will make a policy apply to any target with boats in its name, e.g. pod/boats-2kljw9, pod/whatever-23oije2/container/boats-container, etc.

Note: when mirrord user specifies a container for the mirrord run, the target path ends with /container/<CONTAINER_NAME>.

This means the pattern deploy/my-deployment will not match when a container is specified. That pattern can be changed to deploy/my-deployment* to also match on runs with a specified container (but will then also match deploy/my-deployment-1 etc.)

Please note that the policy is applied according to the target given to mirrord. It is possible for a policy to apply to a deployment target, but not to apply to the deployment's pods when targeted directly. For example, the following policy:

prevents mirrord users from stealing traffic when using the whole boats deployment as a target. However, a user could still use a specific pod out of that deployment as a target for mirrord and steal its traffic. In order to prevent that, the targetPath pattern or the label selector needs to be changed to match the pods of that deployment.

If a workload is used as a target, this workload's labels will be used to match against policies' selector, if set. If a pod is used as a target, the pod's labels will be used.

Another example of a policy:

This policy blocks mirroring and unfiltered stealing of traffic coming to all deployments in the namespace books which are marked with label component: webserver.

I've run my program with mirrord, but it seems to have no effect

There are currently two known cases where mirrord cannot load into the application's process:

1. Statically linked binaries. Since mirrord uses the dynamic linker to load into the application's process, it cannot load if the binary is statically linked. Support for statically linked binaries is planned for the long term, but for now you would have to make sure your binaries are dynamically linked in order to run them with mirrord. With Go programs, for example, it is as simple as adding import "C" to your program code. If you don't want to add an import to your Go program, you can alternatively build a dynamically linked binary using go build -ldflags='-linkmode external'. In VSCode, this can be done by adding "buildFlags": "-ldflags='-linkmode external'" to your launch.json.

   On Linux, append -ldflags="-s=false" to instruct go run not to omit the symbol table and debug information required by mirrord.

2. If you are running mirrord on MacOS and the executable you are running is protected by SIP (the application you are developing wouldn't be, but the binary that is used to execute it, e.g. bash for a bash script, might be protected), mirrord might have trouble loading into it (mirrord can generally bypass SIP, but there are still some unhandled edge cases). If that is the case, you could try copying the binary you're trying to run to an unprotected directory (e.g. anywhere in your home directory), changing the IDE run configuration or the CLI to use the copy instead of the original binary, and trying again. If it still doesn't work, also remove the signature from the copy with:

   sudo codesign --remove-signature ./<your-binary>

   Please let us know if you're having trouble with SIP by opening an issue on GitHub or talking to us on Discord.

Another reason that mirrord might seem not to work is if your remote pod has more than one container. mirrord works at the level of the container, not the whole pod. If your pod runs multiple containers, you need to make sure mirrord targets the correct one by by specifying it explicitly in the . Note that we filter out the proxy containers added by popular service meshes automatically.

When running a Go program on Linux, DNS and outgoing traffic filters seem to have no effect

This can be caused when Go resolves DNS without going through libc. Build your Go binary with the following environment variable: GODEBUG=netdns=cgo

I've run my Turbo task with mirrord, but it seems to have no effect

When executing a task Turbo strips most of the existing process environment, including internal mirrord variables required during libc call interception setup. There are two alternative ways to solve this problem:

1. Explicitly tell Turbo to pass mirrord environment to the task. To do this, merge the snippet below into your turbo.json. You should be able to run the task like mirrord exec turbo dev.

{
  "globalPassThroughEnv": ["MIRRORD_*", "LD_PRELOAD", "DYLD_INSERT_LIBRARIES"]
}

1. Invoke mirrord inside the Turbo task command line itself.

Incoming traffic to the remote target doesn't reach my local process

This could happen because the local process is listening on a different port than the remote target. You can either change the local process to listen on the same port as the remote target (don't worry about the port being used locally by other processes), or use the to map the local port to a remote port.

The remote target stops receiving remote traffic, but it doesn't reach my local process either

This can happen in some clusters using a service mesh when stealing incoming traffic. You can use this configuration to fix it:

{"agent": {"flush_connections": false}}

My application is trying to read a file locally instead of from the cluster

mirrord has a list of path patterns that are read locally by default regardless of the configured fs mode. You can override this behavior in the configuration.

Here you can find all the pre-defined exceptions:

1. Paths that match the patterns defined here are read locally by default.

2. Paths that match the patterns defined here are read remotely by default when the mode is localwithoverrides.

3. Paths that match the patterns defined here under the running user's home directory will be failed to be found by default when the mode is not local.

In order to override that settings for a path or a pattern, add it to the appropriate set:

1. feature.fs.read_only if you want read operations to that path to happen remotely, but write operations to happen locally.

2. feature.fs.read_write if you want read and write operations to that path to happen remotely.

3. feature.fs.local if you want read and write operations to that path to happen locally.

4. feature.fs.not_found if you want the application to "think" that file does not exist.

My local process fails to resolve the domain name of a Kubernetes service in the same cluster

If you've set feature.fs.mode to local, try changing it to localwithoverrides.

When the local mode is set, all files will be opened locally. This might prevent your process from resolving cluster-internal domain names correctly, because it can no longer read Kubelet-generated configuration files like /etc/resolv.conf. With localwithoverrides, such files are read from the remote pod instead.

Old mirrord agent pods are not getting deleted after the mirrord run is completed

If an agent pod's status is Running, it means mirrord is probably still running locally as well. Once you terminate the local process, the agent pod's status should change to Completed.

On clusters with Kubernetes version v1.23 or higher, agent pods are automatically cleaned up immediately (or after a ). If your cluster is v1.23 or higher and mirrord agent pods are not being cleaned up automatically, please open an issue on GitHub. As a temporary solution for cleaning up completed agent pods manually, you can run:

kubectl delete jobs --selector=app=mirrord --field-selector=status.successful=1

My local process gets permission (EACCESS) error on file access or DNS can't resolve

If your cluster is running on Bottlerocket or has SELinux enabled, please try enabling the privileged flag in the agent configuration:

{
  "agent": {
    "privileged": true
  }
}

mirrord operator status fails with 503 Service Unavailable on GKE

If private networking is enabled, it is likely due to firewall rules blocking the mirrord operator's API service from the API server. To fix this, add a firewall rule that allows your cluster's master nodes to access TCP port 443 in your cluster's pods. Please refer to the GCP docs for information.

My local process encounters unexpected certificate validation errors

When running processes locally versus in a container within Kubernetes, some languages handle certificate validation differently. For instance, a Go application on macOS will use the macOS Keychain for certificate validation, whereas the same application in a container will use different API calls. This discrepancy can lead to unexpected certificate validation errors when using tools like mirrord.

A specific issue with Go can be found here, where Go encounters certificate validation errors due to certain AWS services serving certificates that are deemed invalid by the macOS Keychain, but not by Go’s certificate validation in other environments.

To work around this issue (on macOS), you can use the following mirrord configuration:

{
   "experimental": {"trust_any_certificate": true}
}

This configuration would make any certificate trusted for the process.

Other alternatives are to either disable certificate validation in your application or import the problematic certificate (or its root CA) into your macOS Keychain. For guidance on how to do this, refer to this Apple support article.

Agent connection fails or drops when using an ephemeral agent with a service mesh

When running the agent as an , the agent shares the network stack with the target pod. This means that incoming connections to the agent are handled by the service mesh, which might drop it for various reasons (lack of TLS, not HTTP, etc.) To work around that, set the agent.port to be static using agent.port in values.yaml when installing the operator, then add a port exclusion for the agent port in your service mesh's configuration. For example, if you use Istio and have set the agent port to 5000, you can add the following annotation for exclusion:

traffic.sidecar.istio.io/excludeInboundPorts: '50000'

My mirrord fails to load mirrord-layer dynamic library with Carbon Black installed

When running mirrord, you might see an error indicating that it failed to load its required dynamic library, for example:

dyld[68792]: terminating because inserted dylib '/tmp/11832501046814586937-libmirrord_layer.dylib' could not be loaded: tried: '/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/System/Volumes/Preboot/Cryptexes/OS/tmp/11832501046814586937-libmirrord_layer.dylib' (no such file), '/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/private/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/System/Volumes/Preboot/Cryptexes/OS/private/tmp/11832501046814586937-libmirrord_layer.dylib' (no such file), '/private/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib')

This error can occur on systems with Carbon Black Endpoint Detection and Response (EDR) software installed. Carbon Black enforces strict controls over code execution, including blocking attempts to load dynamic libraries (shared objects) that are unsigned or located in potentially untrusted or ephemeral directories like /tmp.

mirrord, by design, extracts a temporary dynamic library at runtime — the mirrord-layer — which is written to a randomized path in /tmp (for example, /tmp/11832501046814586937-libmirrord_layer.dylib). This library is then dynamically injected into your local application so mirrord can intercept and redirect its network traffic transparently to a remote Kubernetes environment.

Carbon Black typically does not trust binaries or dynamic libraries residing in ephemeral directories like /tmp. As a result, Carbon Black treats the loading attempt as potentially suspicious and blocks it, which prevents the library from being injected and disrupts mirrord’s operation.

To resolve this problem, you can create an explicit exclusion policy in Carbon Black to permit loading of the mirrord-layer dynamic library. Your policy should permit loading dynamic libraries from a specific predictable directory, such as /private/tmp/mirrord/**.

Using mirrord on Windows requires setting up the Linux Subsystem for Windows (WSL). You’ll also need a Kubernetes cluster. If you don’t have one, you can set one up locally using . mirrord works with any Kubernetes cluster, be it remote or local.

Setting up WSL

You can read about the prerequisites and installation options on the official Microsoft documentation for .

The mirrord guide uses the default installation options, which has Ubuntu as the Linux distro. mirrord itself is not limited to any particular distro.

From the Microsoft Store

To install WSL from the Microsoft Store just open the Microsoft Store app, then search for the name of the Linux distro you want. We recommend installing Ubuntu, but mirrord works with any Linux distro.

After installation is complete, click on the Open button and a terminal window will appear.

From the Terminal

Open a terminal with administrator privileges.

• It doesn’t have to be the Windows Terminal. PowerShell and Command Prompt will also work.

On the terminal, run the wsl –install command to install the default (Ubuntu) Linux distro:

• You can read more about the prerequisites and installation options on the official Microsoft documentation for . This guide uses the default installation options, which has Ubuntu as the Linux distro. mirrord itself is not limited to any particular distro.

After installing WSL, in a terminal window, you should see the following output from executing the wsl --list command:

• If you're not seeing any Linux Distribution listed, please refer back to the , or join our [Discord server](link pending) and we'll be happy to help you.

To start a session in WSL, now enter the wsl command:

Setting up the Linux distro

After starting a new WSL session (either from the command line, or from the Microsoft Store) you’ll be prompted to set up a Linux user. The username and password does not need to match your Windows user.

After setting up your Linux user, it’s time to prepare the Linux environment for development. Install the tools needed to access your Kubernetes cluster (gcloud cli, azure cli, or whatever cli tool you use for authentication and cluster connection). You’ll also need to install the compilers and project management tools (such as nvm, JDK, dotnet cli) necessary to run and debug your project.

• Many of those tools may be installed using the Linux distro package manager, but some might require manual installation and setup.

Some IDEs may support running in WSL from Windows directly (the IDE is installed on Windows), such as VS Code and the IntelliJ family of IDEs, while others may require being installed in Linux itself.

Kubernetes on WSL

• Setting up a Kubernetes cluster is out of scope for this guide - we’re assuming that you have a remote cluster to target with mirrord. If you don’t have a Kubernetes cluster to use and still want to try out mirrord, we recommend checking out the Docker Desktop guide on .

With the tooling out of the way, and after cluster authorization has been set up, you may check cluster access with kubectl get all.

• If you got a command not found error instead, this means that kubectl is not installed. Some Kubernetes tools install it as part of their setup, but you can also manually install it directly, follow the for installing it on Linux. You can also , but this may require changing the KUBECONFIG environment variable.

• If you’re not seeing any of your Kubernetes resources, you might need to change your Kubernetes configuration. Refer to the manual.

Running a project on WSL

Before starting your IDE, it’s recommended that you copy your project files from the Windows file system to Linux, to avoid performance issues. The best practice is to have everything inside Linux.

You can do this from the command line (from within Linux, the Windows file system should be something like /mnt/{windows-drive-letter}, so by default it’ll be /mnt/c), or from File Explorer.

Creating a playground project

• If you already have your own project, you may .

We’ll provide you with a small playground project here, if you don’t already have your own. Let's create a sample NodeJS project to use with mirrord, but bear in mind that mirrord is not limited to any programming languages or frameworks. In the Linux terminal, navigate to the home directory.

Create a new playground directory.

Install NodeJS (if you haven’t already in the Setting up the Linux Distro section). First update the package manager.

Now install the nodejs package.

Create a very simple NodeJS program.

Running node app.mjs should look something like this.

We can finally move on to installing and using mirrord.

Using mirrord in VS Code

• Microsoft provides a very good guide on .

Open VS Code from Windows, as you normally would, and click on the Remote Explorer.

Select the Linux distro you have set up, and click on the Connect in Current Window button that appears.

VS Code will notify you it’s starting WSL, and the Remote Explorer will change to indicate you’re connected.

Now go to the Extensions panel, search for mirrord and install it.

• Some of your extensions may appear as disabled, with a button to Install in WSL. If you want to use these extensions from the WSL VS Code, then you must click the button and install them.

• If you get an error saying that mirrord does not support the Windows platform, this means that you’re trying to install it on the Windows VS Code. Uninstall the mirrord extension, and follow the previous steps to start the WSL VS Code.

With mirrord installed, open up your project.

• Keep in mind that you’ll be navigating the directories with Linux style paths. If you have not copied your project files to WSL, you can navigate the Windows files from the /mnt directory.

Using mirrord in IntelliJ

• Jetbrains provides a very good guide on .

Open the Jetbrains IDE you have installed on Windows (the mirrord plugin is available for every Jetbrains IDE. In this tutorial we’ll show screen caps from IntelliJ Idea Ultimate, but that’s not a requirement).

Select the WSL option under Remote Development.

Click on the + button (if you already have a project, otherwise select New Project).

• Pay attention to the IDE version you’re choosing. The recommendation here is to select the same one that you have installed on Windows, pay close attention to the version numbers as well (sometimes the Beta version comes selected by default).

Either type the path to your project, or click on the ... button to open the path picker.

Now click Download IDE and Connect at the bottom.

The IDE will be downloaded and installed on Linux. After it’s ready, it should automatically open.

Click on the gear button, select Plugins and search the Marketplace for “mirrord”.

After clicking to install it, the install button will change to Restart IDE. Instead of restarting it like that, close the WSL IDE, and in the Windows IDE select to open your project again.

• If you get an error saying that mirrord does not support the Windows platform, this means that you’re trying to install it on the Windows IDE. Uninstall the mirrord extension, and follow the previous steps to start the WSL IDE.

Using mirrord from the CLI

In your WSL terminal, you can download and install mirrord by running the following command:

• You might get prompted to enter your root user password, so we can install it in /usr/local/bin.

• If curl is not installed in the Linux distro, you can use the distro package manager to install it, or download and install it manually from the .

Now to execute your project with mirrord, just run the mirrord exec command:

• If you’re using this guide’s playground project your mirrord exec command should be:

• You can list the available mirrord targets with the mirrord ls command. If no targets are being shown, you might not have any Kubernetes resources that can be targeted by mirrord, or you might not be using the right Kubernetes context. You can check the later with kubectl config view, look at the current-context and see if it’s the intended one. You may change the context with the kubectl config use-context [CONTEXT NAME] command.

You can use mirrord exec –help to list other exec options.

Troubleshooting

IntelliJ failed to update mirrord binary

If you're seeing a mirrord notification pop-up that says something along the lines of:

failed to update the mirrord binary: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Something is wrong with the local certificate that IntelliJ is trying to use. You can read more about this on the page.

You can fix this issue by navigating to the IntelliJ IDE dir (change it to match where your IntelliJ IDE is installed) in the WSL terminal:

And issuing the following command:

IntelliJ mirrord does not launch and settings Select Active does not work

If you don't see the mirrord UI for selecting the active configuration when clicking Select Active, this usually means that the mirrord plugin has been installed only on one side (host or client). This issue might also prevent mirrord from running inside WSL.

To fix it, make sure that the mirrord plugin is installed on both the host and the client (WSL) IDE. Open the host IDE (or client IDE), navigate to the Plugins section, and install mirrord.

If your application consumes messages from a queue service, you should choose a configuration that matches your intention:

1. Running your application with mirrord without any special configuration will result in your local application competing with the remote target (and potentially other mirrord runs by teammates) for queue messages.

2. Running your application with copy_target + scale_down will result in the deployed application not consuming any messages, and your local application being the exclusive consumer of queue messages.

3. If you want to control which messages will be consumed by the deployed application, and which ones will reach your local application, set up queue splitting for the relevant target, and define a messages filter in the mirrord configuration. Messages that match the filter will reach your local application, and messages that do not, will reach either the deployed application, or another teammate's local application, if they match their filter.

NOTE: So far queue splitting is available for Amazon SQS and Kafka. Pretty soon we'll support RabbitMQ as well.

How It Works

SQS Splitting

When an SQS splitting session starts, the operator changes the target workload to consume messages from a different, temporary queue created by the operator. The operator also creates a temporary queue that the local application reads from.

So if we have a consumer app reading messages from a queue:

After a mirrord SQS splitting session starts, the setup will change to this:

The operator will consume messages from the original queue, and try to match their attributes with filter defined by the user in the mirrord configuration file (read more in the last section). A message that matches the filter will be sent to the queue consumed by the local application. Other messages will be sent to the queue consumed by the remote application.

And as soon as a second mirrord SQS splitting session starts, the operator will create another temporary queue for the new local app:

The users' filters will be matched in the order of the start of their sessions. If filters defined by two users both match a message, the message will go to whichever user started their session first.

After a mirrord session ends, the operator will delete the temporary queue that was created for that session. When all sessions that split a certain queue end, the mirrord Operator will wait for the deployed application to consume the remaining messages in its temporary queue, and then delete that temporary queue as well, and change the deployed application to consume messages back from the original queue.

Kafka Splitting

When a Kafka splitting session starts, the operator changes the target workload to consume messages from a different, temporary topic created by the operator in the same Kafka cluster. The operator also creates a temporary topic that the local application reads from.

So if we have a consumer app reading messages from a topic:

After a mirrord Kafka splitting session starts, the setup will change to this:

The operator will consume messages from the original topic (using the same consumer group id as the target workload), and try to match their headers with filter defined by the user in the mirrord configuration file (read more in the last section). A message that matches the filter will be sent to the topic consumed by the local application. Other messages will be sent to the topic consumed by the remote application.

And as soon as a second mirrord Kafka splitting session starts, the operator will create another temporary queue for the new local app:

The users' filters will be matched in the order of the start of their sessions. If filters defined by two users both match a message, the message will go to whichever user started their session first.

After a mirrord session ends, the operator will delete the temporary topic that was created for that session. When all sessions that split a certain topic end, the mirrord Operator will change the deployed application to consume messages back from the original topic and delete the temporary topic as well.

Getting Started with SQS Splitting

Enabling SQS Splitting in Your Cluster

In order to use the SQS splitting feature, some extra values need be provided during the installation of the mirrord Operator.

First of all, the SQS splitting feature needs to be enabled:

• When installing with the mirrord-operator Helm chart it is enabled by setting the operator.sqsSplitting value to true.

• When installing via the mirrord operator setup command, set the --sqs-splitting flag.

When SQS splitting is enabled during installation, some additional resources are created, and the SQS component of the mirrord Operator is started.

Additionally, the operator needs to be able to do some operations on SQS queues in your account. For that, an IAM role with an appropriate policy has to be assigned to the operator's service account. Please follow AWS's documentation on how to do that.

Some of the permissions are needed for your actual queues that you would like to split, and some permissions are only needed for the temporary queues the mirrord Operator creates and later deletes. Here is an overview:

Here we provide a short explanation for each required permission.

• sqs:GetQueueUrl: the operator finds queue names to split in the provided source, and then it fetches the URL from SQS in order to make all other API calls.

• sqs:GetQueueAttributes: the operator gives all temporary queues the same attributes as their corresponding original queue, so it needs permission to get the original queue's attributes. It also reads the attributes of temporary queues it created, in order to check how many messages they have approximately.

• sqs:ListQueueTags: the operator queries your queue's tags, in order to give all temporary queues that are created for that queue the same tags.

• sqs:ReceiveMessage: the mirrord Operator will read messages from queues you want to split.

• sqs:DeleteMessage: after reading a message and forwarding it to a temporary queue, the operator deletes it.

• sqs:CreateQueue: the mirrord Operator will create temporary queues in your SQS account.

• sqs:TagQueue: all the queues mirrord creates will be tagged with all the tags of their respective original queues, plus any tags that are configured for them in the MirrordWorkloadQueueRegistry in which they are declared.

• sqs:SendMessage: mirrord will send the messages it reads from an original queue to the temporary queue of the client whose filter matches it, or to the temporary queue the deployed application reads from.

• sqs:DeleteQueue: when a user session is done, mirrord will delete the temporary queue it created for that session. After all sessions that split a certain queue end, also the temporary queue that is for the deployed application is deleted.

This is an example for a policy that gives the operator's roles the minimal permissions it needs to split a queue called ClientUploads:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sqs:GetQueueUrl",
                "sqs:GetQueueAttributes",
                "sqs:ListQueueTags",
                "sqs:ReceiveMessage",
                "sqs:DeleteMessage"
            ],
            "Resource": [
                "arn:aws:sqs:eu-north-1:314159265359:ClientUploads"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "sqs:CreateQueue",
                "sqs:TagQueue",
                "sqs:SendMessage",
                "sqs:GetQueueAttributes",
                "sqs:DeleteQueue"
            ],
            "Resource": "arn:aws:sqs:eu-north-1:314159265359:mirrord-*"
        }
    ]
}

• The first statement gives the role the permissions it needs for your original queues.

  Instead of specifying the queues you would like to be able to split in the first statement, you could alternatively make that statement apply for all resources in the account, and limit the queues it applies to using conditions instead of resource names. For example, you could add a condition that makes the statement only apply to queues with the tag splittable=true or env=dev etc. and set those tags for all queues you would like to allow the operator to split.

• The second statement in the example gives the role the permissions it needs for the temporary queues. Since all the temporary queues created by mirrord are created with the name prefix mirrord-, that statement in the example is limited to resources with that prefix in their name.

  If you would like to limit the second statement with conditions instead of (only) with the resource name, you can set a condition that requires a tag, and in the MirrordWorkloadQueueRegistry resource you can specify for each queue tags that mirrord will set for temporary queues that it creates for that original queue.

If the queue messages are encrypted, the operator's IAM role should also have the following permissions:

• kms:Encrypt

• kms:Decrypt

• kms:GenerateDataKey

The ARN of the IAM role has to be passed when installing the operator.

• When installing with Helm, the ARN is passed via the sa.roleArn value

• When installing via the mirrord operator setup command, use the --aws-role-arn flag.

Permissions for Target Workloads

In order to be targeted with SQS queue splitting, a workload has to be able to read from queues that are created by mirrord.

Any temporary queues created by mirrord are created with the same policy as the original queues they are splitting (with the single change of the queue name in the policy), so if a queue has a policy that allows the target workload to call ReceiveMessage on it, that is enough.

However, if the workload gets its access to the queue by an IAM policy (and not an SQS policy, see SQS docs) that grants access to that specific queue by its exact name, you would have to add a policy that would allow that workload to also read from new temporary queues created by mirrord on the run.

Creating a Queue Registry

On operator installation, a new CustomResources type was created on your cluster: MirrordWorkloadQueueRegistry. Users with permissions to get CRDs, can verify its existence with kubectl get crd mirrordworkloadqueueregistries.queues.mirrord.metalbear.co. After an SQS-enabled operator is installed, and before you can start splitting queues, a resource of that type must be created for the target you want to run against, in the target's namespace.

Below we have an example for such a resource, for a meme app that consumes messages from two queues:

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordWorkloadQueueRegistry
metadata:
  name: meme-app-q-registry
spec:
  queues:
    meme-queue:
      queueType: SQS
      nameSource:
        envVar: INCOMING_MEME_QUEUE_NAME
      tags:
        tool: mirrord
    ad-queue:
      queueType: SQS
      nameSource:
        envVar: AD_QUEUE_NAME
      tags:
        tool: mirrord
  consumer:
    name: meme-app
    container: main
    workloadType: Deployment

• spec.queues holds queues that should be split when running mirrord with this target. It is a mapping from a queue ID to the details of the queue.

  • The queue ID is chosen by you, and will be used by every teammate who wishes to filter messages from this queue. You can choose any string for that, it does not have to be the same as the name of the queue. In the example above the first queue has the queue id meme-queue and the second one ad-queue.

  • nameSource tells mirrord where the app finds the name of this queue.

    • Currently envVar is the only supported source for the queue name. The value of envVar is the name of the environment variable the app reads the queue name from. That environment variable could be one that has a value directly in the spec, or it could get its value from a ConfigMap via valueFrom or envFrom. It is crucial that both the local and the deployed app use the queue name they find in that environment variable. mirrord changes the value of that environment variable in order to make the application read from a temporary queue it creates.

  • tags is an optional field where you can specify queue tags that should be added to all temporary queues mirrord creates for splitting this queue.

• spec.consumer is the workload that consumes these queues. The queues specified above will be split whenever that workload is targeted.

  • container is optional, when set - this queue registry only applies to runs that target that container.

Getting Started with Kafka Splitting

Enabling Kafka Splitting in Your Cluster

In order to use the Kafka splitting feature, some extra values need be provided during the installation of the mirrord Operator.

First of all, the Kafka splitting feature needs to be enabled:

• When installing with the mirrord-operator Helm chart it is enabled by setting the operator.kafkaSplitting value to true.

• When installing via the mirrord operator setup command, set the --kafka-splitting flag.

When Kafka splitting is enabled during installation, some additional resources are created, and the Kafka component of the mirrord Operator is started.

Configuring Kafka Splitting with Custom Resources

On operator installation, new CustomResources types were created on your cluster: MirrordKafkaTopicsConsumer and MirrordKafkaClientConfig. Users with permissions to get CRDs, can verify their existence with kubectl get crd mirrordkafkatopicsconsumers.queues.mirrord.metalbear.co and kubectl get crd mirrordkafkaclientconfigs.queues.mirrord.metalbear.co.

After a Kafka-enabled operator is installed, and before you can start splitting queues, resources of these types must be created.

1. MirrordKafkaTopicsConsumer is a resource that must be created in the same namespace as the target workload. It describes Kafka topics that this workload consumes and contains instructions for the mirrord Operator on how to execture splitting. Each MirrordKafkaTopicsConsumer is linked to a single workload that can be targeted with a Kafka splitting session.

2. MirrordKafkaClientConfig is a resource that must be created in the namespace where mirrord operator is installed. It contains properties that the operator will use when creating a Kafka client used for all Kafka operations during the split. This resource is referenced by MirrordKafkaTopicsConsumer.

MirrordKafkaTopicsConsumer

Below we have an example for MirrordKafkaTopicsConsumer resource, for a meme app that consumes messages from a Kafka topic:

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordKafkaTopicsConsumer
metadata:
  name: meme-app-topics-consumer
spec:
  consumerApiVersion: apps/v1
  consumerKind: Deployment
  consumerName: meme-app
  topics:
  - id: views-topic
    clientConfig: base-config
    groupIdSources:
    - directEnvVar:
        container: consumer
        variable: KAFKA_GROUP_ID
    nameSources:
    - directEnvVar:
        container: consumer
        variable: KAFKA_TOPIC_NAME

• spec.topics is a list of topics that can be split when running mirrord with this target.

  • The topic ID is chosen by you, and will be used by every teammate who wishes to filter messages from this topic. You can choose any string for that, it does not have to be the same as the name of the queue. In the example above the topic has id views-topic.

  • clientConfig is the name of the MirrordKafkaClientConfig resource living in the mirrord Operator's namespace that will be used when interacting with the Kafka cluster.

  • groupIdSources holds a list of all occurences of Kafka consumer group id in the workload's pod spec. mirrord Operator will use this group id when consuming messages from the topic.

    Currently the only supported source type is an environment variable with value defined directly in the pod spec.

  • nameSources holds a list of all occurences of topic name in the workload's pod spec. mirrord Operator will use this name when consuming messages. It is crucial that both the local and deployed app take topic name from these sources, as mirrord Operator will use them to inject the names of temporary topics.

    Currently the only supported source type is an environment variable with value defined directly in the pod spec.

MirrordKafkaClientConfig

Below we have an example for MirrordKafkaClientConfig resource:

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordKafkaClientConfig
metadata:
  name: base-config
  namespace: mirrord
spec:
  properties:
  - name: bootstrap.servers
    value: kafka.default.svc.cluster.local:9092

When used by the mirrord Operator for Kafka splitting, the example below will be resolved to following .properties file:

bootstrap.servers=kafka.default.svc.cluster.local:9092

This file will be used when creating a Kafka client for managing temporary topics, consuming messages from the original topic and producing messages to the temporary topics. Full list of available properties can be found here.

NOTE: group.id property will always be overwritten by mirrord Operator when resolving the .properties file.

MirrordKafkaClientConfig resource supports property inheritance via spec.parent field. When resolving a resource X that has parent Y:

1. Y is resolved into a .properties file.

2. For each property defined in X:

   • If value is provided, it overrides any previous value of that property

   • If value is not provided (null), that property is removed

Below we have an example of two MirrordKafkaClientConfigs with inheritance relation:

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordKafkaClientConfig
metadata:
  name: base-config
  namespace: mirrord
spec:
  properties:
  - name: bootstrap.servers
    value: kafka.default.svc.cluster.local:9092
  - name: message.send.max.retries
    value: 4

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordKafkaClientConfig
metadata:
  name: with-client-id
  namespace: mirrord
spec:
  parent: base-config
  properties:
  - name: client.id
    value: mirrord-operator
  - name: message.send.max.retries
    value: null

When used by the mirrord Operator for Kafka splitting, the with-client-id below will be resolved to following .properties file:

bootstrap.servers=kafka.default.svc.cluster.local:9092
client.id=mirrord-operator

MirrordKafkaClientConfig also supports setting properties from a Kubernetes Secret with the spec.loadFromSecret field. The value for loadFromSecret is given in the form: <secret-namespace>/<secret-name>.

Each key-value entry defined in secret's data will be included in the resulting .properties file. Property inheritance from the parent still occurs, and within each MirrordKafkaClientConfig properties loaded from the secret are overwritten by those in properties.

This means the priority of setting properties (from highest to lowest) is like so:

• childProperty

• childSecret

• parentProperty

• parentSecret

Below is an example for a MirrordKafkaClientConfig resource that references a secret:

apiVersion: queues.mirrord.metalbear.co/v1alpha
kind: MirrordKafkaClientConfig
metadata:
  name: base-config
  namespace: mirrord
spec:
  loadFromSecret: mirrord/my-secret
  properties: []

NOTE: By default, the operator will only have access to secrets in its own namespace (mirrord by default).

Customizing mirrord created Kafka Topic Names

NOTE: Available since chart version 1.27 and operator version 3.114.0

To serve Kafka splitting sessions, mirrord Operator creates temporary topics in the Kafka cluster. The default format for their names is as follows:

• mirrord-tmp-1234567890-fallback-topic-original-topic - for the fallback topic (unfiltered messages, consumed by the deployed workload).

• mirrord-tmp-9183298231-original-topic - for the user topics (filtered messages, consumed by local applications running with mirrord).

Note that the random digits will be unique for each temporary topic created by the mirrord Operator.

You can adjust the format of the created topic names to suit your needs (RBAC, Security, Policies, etc.), using the OPERATOR_KAFKA_SPLITTING_TOPIC_FORMAT environment variable of the mirrord Operator, or operator.kafkaSplittingTopicFormat helm chart value. The default value is:

mirrord-tmp-{{RANDOM}}{{FALLBACK}}{{ORIGINAL_TOPIC}}

The provided format must contain the three variables: {{RANDOM}}, {{FALLBACK}} and {{ORIGINAL_TOPIC}}.

• {{RANDOM}} will resolve to random digits.

• {{FALLBACK}} will resolve either to -fallback- or - literal.

• {{ORIGINAL_TOPIC}} will resolve to the name of the original topic that is being split.

Setting a Filter for a mirrord Run

Once everything else is set, you can start using message filters in your mirrord configuration file. Below is an example for what such a configuration might look like:

{
    "operator": true,
    "target": "deployment/meme-app/main",
    "feature": {
        "split_queues": {
            "meme-queue": {
                "queue_type": "SQS",
                "message_filter": {
                    "author": "^me$",
                    "level": "^(beginner|intermediate)$"
                }
            },
            "ad-queue": {
                "queue_type": "SQS",
                "message_filter": {}
            },
            "views-topic": {
                "queue_type": "Kafka",
                "message_filter": {
                    "author": "^me$",
                    "source": "^my-session-"
                }
            }
        }
    }
}

• feature.split_queues is the configuration field you need to specify in order to filter queue messages. Directly under it, we have a mapping from a queue or topic ID to a queue filter definition.

  • Queue or topic ID is the ID that was set in the SQS queue registry resource or Kafka topics consumer resource.

  • message_filter is a mapping from message attribute (SQS) or header (Kafka) names to message attribute or header value regexes. Your local application will only see queue messages that have all of the specified message attributes or headers.

    Empty message_filter is treated as a match-none directive.

In the example above, the local application:

• Will receive a subset of messages from SQS queue with ID meme-queue. All received messages will have an attribute author with the value me, AND an attribute level with value either beginner or intermediate.

• Will receive a subset of messages from Kafka topic with ID views-topic. All received messages will have an attribute author with the value me, AND an attribute source with value starting with my-session- (e.g my-session-844cb78789-2fmsw).

• Will receive no messages from SQS queue with id ad-queue.

Once all users stop filtering a queue (i.e. end their mirrord sessions), the temporary queues (SQS) and topics (Kafka) that mirrord operator created will be deleted.