Some common issues and workarounds.
There are currently two known cases where mirrord cannot load into the application's process:
Statically linked binaries. Since mirrord uses the dynamic linker to load into the application's process, it cannot load if the binary is statically linked. Support for statically linked binaries is planned for the long term, but for now you would have to make sure your binaries are dynamically linked in order to run them with mirrord. With Go programs, for example, it is as simple as adding import "C" to your program code. If you don't want to add an import to your Go program, you can alternatively build a dynamically linked binary using go build -ldflags='-linkmode external'. In VSCode, this can be done by adding "buildFlags": "-ldflags='-linkmode external'" to your launch.json.
On Linux, append -ldflags="-s=false" to instruct go run not to omit the symbol table and debug information required by mirrord.
If you are running mirrord on MacOS and the executable you are running is protected by SIP (the application you are developing wouldn't be, but the binary that is used to execute it, e.g. bash for a bash script, might be protected), mirrord might have trouble loading into it (mirrord can generally bypass SIP, but there are still some unhandled edge cases). If that is the case, you could try copying the binary you're trying to run to an unprotected directory (e.g. anywhere in your home directory), changing the IDE run configuration or the CLI to use the copy instead of the original binary, and trying again. If it still doesn't work, also remove the signature from the copy with:
sudo codesign --remove-signature ./<your-binary>
Please let us know if you're having trouble with SIP by opening an issue on GitHub or talking to us on Discord.
Another reason that mirrord might seem not to work is if your remote pod has more than one container. mirrord works at the level of the container, not the whole pod. If your pod runs multiple containers, you need to make sure mirrord targets the correct one by by specifying it explicitly in the target configuration. Note that we filter out the proxy containers added by popular service meshes automatically.
This can be caused when Go resolves DNS without going through libc. Build your Go binary with the following environment variable: GODEBUG=netdns=cgo
When executing a task Turbo strips most of the existing process environment, including internal mirrord variables required during libc call interception setup. There are two alternative ways to solve this problem:
Explicitly tell Turbo to pass mirrord environment to the task. To do this, merge the snippet below into your turbo.json. You should be able to run the task like mirrord exec turbo dev.
{
"globalPassThroughEnv": ["MIRRORD_*", "LD_PRELOAD", "DYLD_INSERT_LIBRARIES"]
}Invoke mirrord inside the Turbo task command line itself.
This could happen because the local process is listening on a different port than the remote target. You can either change the local process to listen on the same port as the remote target (don't worry about the port being used locally by other processes), or use the port_mapping configuration to map the local port to a remote port.
This can happen in some clusters using a service mesh when stealing incoming traffic. You can use this configuration to fix it:
{"agent": {"flush_connections": false}}mirrord has a list of path patterns that are read locally by default regardless of the configured fs mode. You can override this behavior in the configuration.
Here you can find all the pre-defined exceptions:
Paths that match the patterns defined here are read locally by default.
Paths that match the patterns defined here are read remotely by default when the mode is localwithoverrides.
Paths that match the patterns defined here under the running user's home directory will be failed to be found by default when the mode is not local.
In order to override that settings for a path or a pattern, add it to the appropriate set:
feature.fs.read_only if you want read operations to that path to happen remotely, but write operations to happen locally.
feature.fs.read_write if you want read and write operations to that path to happen remotely.
feature.fs.local if you want read and write operations to that path to happen locally.
feature.fs.not_found if you want the application to "think" that file does not exist.
If you've set feature.fs.mode to local, try changing it to localwithoverrides.
When the local mode is set, all files will be opened locally. This might prevent your process from resolving cluster-internal domain names correctly, because it can no longer read Kubelet-generated configuration files like /etc/resolv.conf. With localwithoverrides, such files are read from the remote pod instead.
If an agent pod's status is Running, it means mirrord is probably still running locally as well. Once you terminate the local process, the agent pod's status should change to Completed.
On clusters with Kubernetes version v1.23 or higher, agent pods are automatically cleaned up immediately (or after a configurable TTL). If your cluster is v1.23 or higher and mirrord agent pods are not being cleaned up automatically, please open an issue on GitHub. As a temporary solution for cleaning up completed agent pods manually, you can run:
kubectl delete jobs --selector=app=mirrord --field-selector=status.successful=1If your cluster is running on Bottlerocket or has SELinux enabled, please try enabling the privileged flag in the agent configuration:
{
"agent": {
"privileged": true
}
}mirrord operator status fails with 503 Service Unavailable on GKEIf private networking is enabled, it is likely due to firewall rules blocking the mirrord operator's API service from the API server. To fix this, add a firewall rule that allows your cluster's master nodes to access TCP port 443 in your cluster's pods. Please refer to the GCP docs for information.
When running processes locally versus in a container within Kubernetes, some languages handle certificate validation differently. For instance, a Go application on macOS will use the macOS Keychain for certificate validation, whereas the same application in a container will use different API calls. This discrepancy can lead to unexpected certificate validation errors when using tools like mirrord.
A specific issue with Go can be found here, where Go encounters certificate validation errors due to certain AWS services serving certificates that are deemed invalid by the macOS Keychain, but not by Go’s certificate validation in other environments.
To work around this issue (on macOS), you can use the following mirrord configuration:
{
"experimental": {"trust_any_certificate": true}
}This configuration would make any certificate trusted for the process.
Other alternatives are to either disable certificate validation in your application or import the problematic certificate (or its root CA) into your macOS Keychain. For guidance on how to do this, refer to this Apple support article.
When running the agent as an ephemeral container, the agent shares the network stack with the target pod. This means that incoming connections to the agent are handled by the service mesh, which might drop it for various reasons (lack of TLS, not HTTP, etc.) To work around that, set the agent.port to be static using agent.port in values.yaml when installing the operator, then add a port exclusion for the agent port in your service mesh's configuration. For example, if you use Istio and have set the agent port to 5000, you can add the following annotation for exclusion:
traffic.sidecar.istio.io/excludeInboundPorts: '50000'mirrord-layer dynamic library with Carbon Black installedWhen running mirrord, you might see an error indicating that it failed to load its required dynamic library, for example:
dyld[68792]: terminating because inserted dylib '/tmp/11832501046814586937-libmirrord_layer.dylib' could not be loaded: tried: '/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/System/Volumes/Preboot/Cryptexes/OS/tmp/11832501046814586937-libmirrord_layer.dylib' (no such file), '/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/private/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib'), '/System/Volumes/Preboot/Cryptexes/OS/private/tmp/11832501046814586937-libmirrord_layer.dylib' (no such file), '/private/tmp/11832501046814586937-libmirrord_layer.dylib' (code signing blocked mmap() of '/private/tmp/11832501046814586937-libmirrord_layer.dylib')This error can occur on systems with Carbon Black Endpoint Detection and Response (EDR) software installed. Carbon Black enforces strict controls over code execution, including blocking attempts to load dynamic libraries (shared objects) that are unsigned or located in potentially untrusted or ephemeral directories like /tmp.
mirrord, by design, extracts a temporary dynamic library at runtime — the mirrord-layer — which is written to a randomized path in /tmp (for example, /tmp/11832501046814586937-libmirrord_layer.dylib). This library is then dynamically injected into your local application so mirrord can intercept and redirect its network traffic transparently to a remote Kubernetes environment.
Carbon Black typically does not trust binaries or dynamic libraries residing in ephemeral directories like /tmp. As a result, Carbon Black treats the loading attempt as potentially suspicious and blocks it, which prevents the library from being injected and disrupts mirrord’s operation.
To resolve this problem, you can create an explicit exclusion policy in Carbon Black to permit loading of the mirrord-layer dynamic library. Your policy should permit loading dynamic libraries from a specific predictable directory, such as /private/tmp/mirrord/**.