Bridged Networking or User mode? Replace ebtables & iptables with iptables-nft to allow bridged networking. #61

Open
opened 2025-12-28 23:22:52 +01:00 by adam · 14 comments
Owner

Originally created by @sickcodes on GitHub (Jul 25, 2020).

Here are the two types of networking


# usermode
-netdev user,id=net0,hostfwd=tcp::10022-:22,
-device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:09:49:17

# bridged
-netdev tap,id=net0,ifname=tap0,script=no,downscript=no
-device vmxnet3,netdev=net0,id=net0,mac=52:54:00:AB:F8:B7

user mode

The image has the same IP as your Docker IP ip n to see neighbors
QEMU cannot be reached unless you add ports in the QEMU args during "docker run"
Can't expose Docker ports on a running container (AFAIK), you have to start a new image and move the installation around.
You also have to edit the Launch.sh or add in arguments

bridged mode

The image will run on the container's own internal network, like 10.0.2.15 or something or 192.168.122.11x

After that, you can ssh OR docker exec into the Docker and then ssh into the QEMU image.

which is better?

Realistically, they both appear exactly the same.
Bridge mode would let people forward all connections to the bridge IP.

# usermode
# you have to add
docker run ... -p 50922:10022 -e EXTRA='-netdev user,hostfwd=tcp::10022-:22'

# bridge-mode
docker run ... -p 50922:10022 -ip p.ubl.ic.ip 

If anyone wants to test it out:

sudo tee -a /etc/sysctl.conf <<< 'net.ipv4.ip_forward=1'
sudo sysctl -p
sudo pacman -S iptables-nft

sudo systemctl enable --now libvirtd.service
sudo systemctl enable --now virtlogd.service

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up promisc on 

sudo virsh net-autostart default
sudo virsh net-start default

sudo ip link set dev virbr0 up

sudo ip link set dev tap0 master virbr0


# use the  networking from above in the QEMU args


The actual bug is ebtalbes + dnsmasq doesn't work virsh networking.
You can test it out:

sudo pacman -S libvirtd
virsh net-info --domain default
# works

Bugged:

sudo pacman -S ebtables dnsmasq

virsh net-info --domain default
# fails 

Fix:

sudo pacman -S iptables-nft

virsh net-info --domain default
# works

It's a strange bug, I though related to polkit, however, I found it incompatible with ebtables.

Possibly just with Arch, that the default network doesn't get installed.

Easy fix: replace iptables and ebtables with iptables-nft. If not, virsh hangs, sudo virsh hangs.

On the other hand, QEMU and Docker networking is really, really easy to debug. If bridged network is added, you'd have to start the bridge every time, I also think you need --net host

Originally created by @sickcodes on GitHub (Jul 25, 2020). Here are the two types of networking ``` # usermode -netdev user,id=net0,hostfwd=tcp::10022-:22, -device e1000-82545em,netdev=net0,id=net0,mac=52:54:00:09:49:17 # bridged -netdev tap,id=net0,ifname=tap0,script=no,downscript=no -device vmxnet3,netdev=net0,id=net0,mac=52:54:00:AB:F8:B7 ``` ## user mode The image has the same IP as your Docker IP `ip n` to see neighbors QEMU cannot be reached unless you add ports in the QEMU args during "docker run" Can't expose Docker ports on a running container (AFAIK), you have to start a new image and move the installation around. You also have to edit the Launch.sh or add in arguments ## bridged mode The image will run on the container's own internal network, like 10.0.2.15 or something or 192.168.122.11x After that, you can ssh OR docker exec into the Docker and then ssh into the QEMU image. ### which is better? Realistically, they both appear exactly the same. **Bridge mode would let people forward all connections to the bridge IP.** ``` # usermode # you have to add docker run ... -p 50922:10022 -e EXTRA='-netdev user,hostfwd=tcp::10022-:22' # bridge-mode docker run ... -p 50922:10022 -ip p.ubl.ic.ip ``` If anyone wants to test it out: ``` sudo tee -a /etc/sysctl.conf <<< 'net.ipv4.ip_forward=1' sudo sysctl -p sudo pacman -S iptables-nft sudo systemctl enable --now libvirtd.service sudo systemctl enable --now virtlogd.service sudo ip tuntap add dev tap0 mode tap sudo ip link set tap0 up promisc on sudo virsh net-autostart default sudo virsh net-start default sudo ip link set dev virbr0 up sudo ip link set dev tap0 master virbr0 # use the networking from above in the QEMU args ``` The actual bug is ebtalbes + dnsmasq doesn't work virsh networking. You can test it out: ``` sudo pacman -S libvirtd virsh net-info --domain default # works ``` Bugged: ``` sudo pacman -S ebtables dnsmasq virsh net-info --domain default # fails ``` Fix: ``` sudo pacman -S iptables-nft virsh net-info --domain default # works ``` It's a strange bug, I though related to polkit, however, I found it incompatible with ebtables. Possibly just with Arch, that the default network doesn't get installed. Easy fix: replace iptables and ebtables with iptables-nft. If not, virsh hangs, sudo virsh hangs. On the other hand, QEMU and Docker networking is really, really easy to debug. If bridged network is added, you'd have to start the bridge every time, I also think you need `--net host `
adam added the enhancementdocumentation labels 2025-12-28 23:22:52 +01:00
Author
Owner

@Julioevm commented on GitHub (Aug 4, 2020):

I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff.

@Julioevm commented on GitHub (Aug 4, 2020): I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff.
Author
Owner

@sickcodes commented on GitHub (Aug 4, 2020):

I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff.

If you use the two -net command strings above in the xml file while using virt-manager you can switch between bridged and userMode networking. Haven't done bridged in the dockerfile but it wouldn't be much more than whats already here.

In summary:

Use virt-manager without docker
Load the xml file from OSX-KVM

Try the networking commands above

Otherwise you can just add more ports to the QEMU command using the -e EXTRA env variable and also forwarding it again as a normal -p 111:2222 to the docker command.

@sickcodes commented on GitHub (Aug 4, 2020): > I was having issues with other machines communicating with a program running inside the macos docker, this might have something to do with it. If I get some free time ill try some of this stuff. If you use the two -net command strings above in the xml file while using virt-manager you can switch between bridged and userMode networking. Haven't done bridged in the dockerfile but it wouldn't be much more than whats already here. In summary: Use virt-manager without docker Load the xml file from OSX-KVM Try the networking commands above Otherwise you can just add more ports to the QEMU command using the -e EXTRA env variable and also forwarding it again as a normal -p 111:2222 to the docker command.
Author
Owner

@sickcodes commented on GitHub (Jan 8, 2021):

Replaced iptables with iptables-nft

8d6d0359b1

@sickcodes commented on GitHub (Jan 8, 2021): Replaced iptables with iptables-nft https://github.com/sickcodes/Docker-OSX/commit/8d6d0359b1fcebd368cf9d21e6955bacadb03526
Author
Owner

@sickcodes commented on GitHub (Feb 8, 2021):

e1000-82545em is preventing iMessage from working on Catalina and above

https://forums.unraid.net/topic/84288-catalina-network-bridging-weirdness/?do=findComment&comment=781612

However, a fix is available here: https://github.com/chris1111/AppleIntelE1000e

@sickcodes commented on GitHub (Feb 8, 2021): ~~e1000-82545em is preventing iMessage from working on Catalina and above~~ ~~https://forums.unraid.net/topic/84288-catalina-network-bridging-weirdness/?do=findComment&comment=781612~~ ~~However, a fix is available here: https://github.com/chris1111/AppleIntelE1000e~~
Author
Owner

@mikob commented on GitHub (Mar 2, 2021):

Couldn't get this working. Trying to have a port shared so that the host can access a server running on the mac on port 8080.

-e EXTRA='-netdev user,hostfwd=tcp::8080-:8080'
This produces: qemu-system-x86_64: Parameter 'id' is missing

-e EXTRA='-netdev user,id=net0,hostfwd=tcp::8080-:8080'
This produces: Duplicate ID 'net0' for netdev

-p 8080:8080 -e EXTRA='-netdev user,id=net1,hostfwd=tcp::8080-:8080'
(qemu) qemu-system-x86_64: warning: netdev net1 has no peer
let's the container run... but can't access the port on the host :/

@mikob commented on GitHub (Mar 2, 2021): Couldn't get this working. Trying to have a port shared so that the host can access a server running on the mac on port 8080. `-e EXTRA='-netdev user,hostfwd=tcp::8080-:8080'` This produces: qemu-system-x86_64: Parameter 'id' is missing `-e EXTRA='-netdev user,id=net0,hostfwd=tcp::8080-:8080'` This produces: Duplicate ID 'net0' for netdev `-p 8080:8080 -e EXTRA='-netdev user,id=net1,hostfwd=tcp::8080-:8080'` (qemu) qemu-system-x86_64: warning: netdev net1 has no peer let's the container run... but can't access the port on the host :/
Author
Owner

@sickcodes commented on GitHub (Mar 2, 2021):

I switched master to vmxnet3 in https://github.com/sickcodes/Docker-OSX/pull/160 as it did not affect iMessage and it massively increases speed.

@sickcodes commented on GitHub (Mar 2, 2021): I switched master to `vmxnet3` in https://github.com/sickcodes/Docker-OSX/pull/160 as it did not affect iMessage and it massively increases speed.
Author
Owner

@mikob commented on GitHub (Mar 2, 2021):

@sickcodes Just unsure how to get port forwarding working. I suppose worst case scenario I can just forward via an ssh tunnel.

@mikob commented on GitHub (Mar 2, 2021): @sickcodes Just unsure how to get port forwarding working. I suppose worst case scenario I can just forward via an ssh tunnel.
Author
Owner

@sickcodes commented on GitHub (Mar 2, 2021):

Maybe we can add another ${ADDITIONAL_PORTS} at the end of the line that has 5900 in it too for any other networking arguments for that interface

@sickcodes commented on GitHub (Mar 2, 2021): Maybe we can add another ${ADDITIONAL_PORTS} at the end of the line that has 5900 in it too for any other networking arguments for that interface
Author
Owner

@sickcodes commented on GitHub (Mar 2, 2021):

At the end of this line: https://github.com/sickcodes/Docker-OSX/blob/master/Dockerfile#L228

I'll add it in shortly

@sickcodes commented on GitHub (Mar 2, 2021): At the end of this line: https://github.com/sickcodes/Docker-OSX/blob/master/Dockerfile#L228 I'll add it in shortly
Author
Owner

@sickcodes commented on GitHub (Mar 2, 2021):

@mikob Added https://github.com/sickcodes/Docker-OSX/pull/162 let me test and then merge

@sickcodes commented on GitHub (Mar 2, 2021): @mikob Added https://github.com/sickcodes/Docker-OSX/pull/162 let me test and then merge
Author
Owner

@mikob commented on GitHub (Mar 3, 2021):

@sickcodes just tested it out, it works! Thanks for your really speedy response!

@mikob commented on GitHub (Mar 3, 2021): @sickcodes just tested it out, it works! Thanks for your really speedy response!
Author
Owner

@sickcodes commented on GitHub (Mar 3, 2021):

@mikob No problem!

Here is a worked example for anyone else reading, as written here: https://github.com/sickcodes/Docker-OSX/pull/162#issuecomment-789490227

On the host

docker run -it \
    --device /dev/kvm \
    -p 50922:10022 \
    -e ADDITIONAL_PORTS='hostfwd=tcp::10023-:80,' \
    -p 10023:10023 \
    sickcodes/docker-osx:auto

Inside the container:

/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

brew install nginx
sudo sed -i -e 's/8080/80/' /usr/local/etc/nginx/nginx.conf
# sudo nginx -s stop
sudo nginx

nginx should appear on the host at port 10023

@sickcodes commented on GitHub (Mar 3, 2021): @mikob No problem! Here is a worked example for anyone else reading, as written here: https://github.com/sickcodes/Docker-OSX/pull/162#issuecomment-789490227 On the host ```bash docker run -it \ --device /dev/kvm \ -p 50922:10022 \ -e ADDITIONAL_PORTS='hostfwd=tcp::10023-:80,' \ -p 10023:10023 \ sickcodes/docker-osx:auto ``` Inside the container: ```bash /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" brew install nginx sudo sed -i -e 's/8080/80/' /usr/local/etc/nginx/nginx.conf # sudo nginx -s stop sudo nginx ``` nginx should appear on the host at port 10023
Author
Owner

@mikob commented on GitHub (Mar 3, 2021):

P.S. you don't need to install nginx for a test server, you can just do eg. python3 -m http.server 8080 to spin up a static test server that serves the CWD and it ships with python by default, which ships with osx by default.

@mikob commented on GitHub (Mar 3, 2021): P.S. you don't need to install nginx for a test server, you can just do eg. `python3 -m http.server 8080` to spin up a static test server that serves the CWD and it ships with python by default, which ships with osx by default.
Author
Owner

@TimVanDyke commented on GitHub (Aug 15, 2023):

I am unable to run:

sudo tee -a /etc/sysctl.conf <<< 'net.ipv4.ip_forward=1'
sudo sysctl -p
sudo pacman -S iptables-nft

sudo systemctl enable --now libvirtd.service
sudo systemctl enable --now virtlogd.service

sudo ip tuntap add dev tap0 mode tap
sudo ip link set tap0 up promisc on 

sudo virsh net-autostart default
sudo virsh net-start default

sudo ip link set dev virbr0 up

sudo ip link set dev tap0 master virbr0


# use the  networking from above in the QEMU args


without getting several errors. However, I believe this is the solution to the problems I'm having. Any help would be appreciated.

Output:

Package (1)        Old Version  New Version  Net Change

core/iptables-nft  1:1.8.9-1    1:1.8.9-1      0.00 MiB

Total Installed Size:  5.64 MiB
Net Upgrade Size:      0.00 MiB

:: Proceed with installation? [Y/n] Y
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
:: Processing package changes...
reinstalling iptables-nft...
:: Running post-transaction hooks...
(1/2) Reloading system manager configuration...
  Skipped: Current root is not booted.
(2/2) Arming ConditionNeedsUpdate...
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
System has not been booted with systemd as init system (PID 1). Can't operate.
Failed to connect to bus: Host is down
Authorization not available. Check if polkit service is running or see debug message for more information.
error: failed to connect to the hypervisor
error: Operation not supported: Cannot use direct socket mode if no URI is set

Authorization not available. Check if polkit service is running or see debug message for more information.
error: failed to connect to the hypervisor
error: Operation not supported: Cannot use direct socket mode if no URI is set

Cannot find device "virbr0"
Error: argument "virbr0" is wrong: Device does not exist

Here is my docker compose:

services:
    macos:
        container_name: 'MacOS'
        privileged: true
        network_mode: br0
        devices:
            - /dev/kvm
            - /dev/snd
            - /dev/null
        
        environment:
            - RAM=6
            - NETWORKING=vmxnet3
            - 'USERNAME=user'
            - 'PASSWORD=pass'
            - 'DISPLAY=${DISPLAY:-:0.0}'
        image: 'docker-osx-vnc:latest'
        
        #network_mode: host
@TimVanDyke commented on GitHub (Aug 15, 2023): I am unable to run: ``` sudo tee -a /etc/sysctl.conf <<< 'net.ipv4.ip_forward=1' sudo sysctl -p sudo pacman -S iptables-nft sudo systemctl enable --now libvirtd.service sudo systemctl enable --now virtlogd.service sudo ip tuntap add dev tap0 mode tap sudo ip link set tap0 up promisc on sudo virsh net-autostart default sudo virsh net-start default sudo ip link set dev virbr0 up sudo ip link set dev tap0 master virbr0 # use the networking from above in the QEMU args ``` without getting several errors. However, I believe this is the solution to the problems I'm having. Any help would be appreciated. Output: ``` Package (1) Old Version New Version Net Change core/iptables-nft 1:1.8.9-1 1:1.8.9-1 0.00 MiB Total Installed Size: 5.64 MiB Net Upgrade Size: 0.00 MiB :: Proceed with installation? [Y/n] Y checking keyring... checking package integrity... loading package files... checking for file conflicts... :: Processing package changes... reinstalling iptables-nft... :: Running post-transaction hooks... (1/2) Reloading system manager configuration... Skipped: Current root is not booted. (2/2) Arming ConditionNeedsUpdate... System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down System has not been booted with systemd as init system (PID 1). Can't operate. Failed to connect to bus: Host is down Authorization not available. Check if polkit service is running or see debug message for more information. error: failed to connect to the hypervisor error: Operation not supported: Cannot use direct socket mode if no URI is set Authorization not available. Check if polkit service is running or see debug message for more information. error: failed to connect to the hypervisor error: Operation not supported: Cannot use direct socket mode if no URI is set Cannot find device "virbr0" Error: argument "virbr0" is wrong: Device does not exist ``` Here is my docker compose: ``` services: macos: container_name: 'MacOS' privileged: true network_mode: br0 devices: - /dev/kvm - /dev/snd - /dev/null environment: - RAM=6 - NETWORKING=vmxnet3 - 'USERNAME=user' - 'PASSWORD=pass' - 'DISPLAY=${DISPLAY:-:0.0}' image: 'docker-osx-vnc:latest' #network_mode: host ```
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: starred/Docker-OSX#61