Docker softether local bridge cascade（site to site）

[775118134]

Requirements

There are 8 LAN servers in the cloud server cluster CA and 4 LAN servers in the cloud server cluster CB; deploy SE in server A in CA and deploy SE in server B in CB, connect the SE of servers A and B through cascade connection, and after the window client is connected through VPN, it can access all servers in the cloud server cluster CA and cloud server cluster CB through the intranet IP, and can access the external network. The displayed IP is the window client IP.

Current status:

Server A

Server
Version: CentOS Linux release 7.9.2009 (Core)

[root@ao mht-softether]# cat /proc/version
Linux version 4.19.0-9.el7.ucloud.x86_64 (root@2549ed846496) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)) #1 SMP Mon Sep 28 10:29:09 UTC 2020
[root@ao mht-softether]# more /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

External network: AAA.AAA.AAA.AAA
Internal network: 10.9.94.119

Docker Container
Server: alpine 3.16.7

/usr/vpnserver # cat /proc/version
Linux version 4.19.0-9.el7.ucloud.x86_64 (root@2549ed846496) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-16) (GCC)) #1 SMP Mon Sep 28 10:29:09 UTC 2020
/usr/vpnserver # more /etc/alpine-release
3.16.7

eth0: 172.20.90.2
tap_soft:192.168.31.1

/usr/vpnserver # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:14:5A:02
inet addr:172.20.90.2 Bcast:172.20.90.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:122162 errors:0 dropped:0 overruns:0 frame:0
TX packets:118276 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:31195291 (29.7 MiB) TX bytes:18085125 (17.2 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:73455 errors:0 dropped:0 overruns:0 frame:0
TX packets:73455 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:13364945 (12.7 MiB) TX bytes:13364945 (12.7 MiB)

tap_soft Link encap:Ethernet HWaddr 5E:AD:A2:57:89:B5
inet addr:192.168.31.1 Bcast:192.168.31.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:1167 errors:0 dropped:0 overruns:0 frame:0
TX packets:18 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:269744 (263.4 KiB) TX bytes:4210 (4.1 KiB)

/usr/vpnserver # ip route show
default via 172.20.90.1 dev eth0
172.20.90.0/24 dev eth0 scope link src 172.20.90.2
192.168.31.0/24 dev tap_soft scope link src 192.168.31.1

/usr/vpnserver # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.20.90.1 0.0.0.0 UG 0 0 0 eth0
172.20.90.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.31.0 0.0.0.0 255.255.255.0 U 0 0 0 tap_soft

dnsmasq

/usr/vpnserver # more /etc/dnsmasq.conf
local-service

port=0

bind-interfaces

interface=tap_soft

dhcp-authoritative

dhcp-no-override

dhcp-range=tap_soft,192.168.31.10,192.168.31.49,12h

dhcp-option=tap_soft,3

dhcp-option=tap_soft,121,192.168.0.0/24,192.168.33.1,10.9.94.0/24,192.168.31.1,10.10.12.0/24,192.168.31.1
dhcp-option=tap_soft,249,192.168.0.0/24,192.168.33.1,10.9.94.0/24,192.168.31.1,10.10.12.0/24,192.168.31.1

conf-dir=/etc/dnsmasq.d/,*.conf

iptables

iptables -t nat -A POSTROUTING -s 192.168.31.0/24 -j SNAT --to-source 172.20.90.2

iptables -t nat -L -n -v --line-numbers

/usr/vpnserver # iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 578 packets, 141K bytes)
num pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 24 packets, 5370 bytes)
num pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 58268 packets, 4022K bytes)
num pkts bytes target prot opt in out source destination
1 26424 1824K DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11

Chain POSTROUTING (policy ACCEPT 77679 packets, 5361K bytes)
num pkts bytes target prot opt in out source destination
1 5 1314 SNAT all -- * * 192.168.31.0/24 0.0.0.0/0 to:172.20.90.2

Chain DOCKER_OUTPUT (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 DNAT [unsupported revision]
2 26424 1824K DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 DNAT [unsupported revision]

Chain DOCKER_POSTROUTING (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:33581 to::53
2 0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:43082 to::53

Server B

Server
Version: CentOS Linux release 7.9.2009 (Core)

[root@iZbp1a8vtwsdkuj7v7o2mfZ mht-softether]# cat /proc/version
Linux version 3.10.0-1160.114.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Mar 20 15:54:52 UTC 2024
[root@iZbp1a8vtwsdkuj7v7o2mfZ mht-softether]# more /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

External network: BBB.BBB.BBB.BBB
Internal network: 192.168.0.36

Docker Container
Server: alpine 3.16.7

/usr/vpnserver # cat /proc/version
Linux version 3.10.0-1160.114.2.el7.x86_64 (mockbuild@kbuilder.bsys.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-44) (GCC) ) #1 SMP Wed Mar 20 15:54:52 UTC 2024
/usr/vpnserver # more /etc/alpine-release
3.16.7

eth0: 172.20.41.2
tap_soft:192.168.33.1

/usr/vpnserver # ifconfig
eth0 Link encap:Ethernet HWaddr 02:42:AC:14:29:02
inet addr:172.20.41.2 Bcast:172.20.41.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:70887 errors:0 dropped:0 overruns:0 frame:0
TX packets:76048 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:31445471 (29.9 MiB) TX bytes:21146620 (20.1 MiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:65536 Metric:1
RX packets:93026 errors:0 dropped:0 overruns:0 frame:0
TX packets:93026 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:10269030 (9.7 MiB) TX bytes:10269030 (9.7 MiB)

tap_soft Link encap:Ethernet HWaddr 5E:B2:3E:75:BD:52
inet addr:192.168.33.1 Bcast:192.168.33.255 Mask:255.255.255.0
inet6 addr: fe80::5cb2:3eff:fe75:bd52/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:4986 errors:0 dropped:0 overruns:0 frame:0
TX packets:14710 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:902357 (881.2 KiB) TX bytes:12243453 (11.6 MiB)

/usr/vpnserver # ip route show
default via 172.20.41.1 dev eth0
172.20.41.0/24 dev eth0 scope link src 172.20.41.2
192.168.33.0/24 dev tap_soft scope link src 192.168.33.1

/usr/vpnserver # route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.20.41.1 0.0.0.0 UG 0 0 0 eth0
172.20.41.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.33.0 0.0.0.0 255.255.255.0 U 0 0 0 tap_soft

dnsmasq

/usr/vpnserver # more /etc/dnsmasq.conf
local-service

port=0

bind-interfaces

interface=tap_soft

dhcp-authoritative

dhcp-no-override

dhcp-range=tap_soft,192.168.33.50,192.168.33.90,12h

dhcp-option=tap_soft,3

dhcp-option=tap_soft,121,192.168.0.0/24,192.168.33.1,10.9.94.0/24,192.168.31.1,10.10.12.0/24,192.168.31.1
dhcp-option=tap_soft,249,192.168.0.0/24,192.168.33.1,10.9.94.0/24,192.168.31.1,10.10.12.0/24,192.168.31.1

conf-dir=/etc/dnsmasq.d/,*.conf

iptables

iptables -t nat -A POSTROUTING -s 192.168.33.0/24 -j SNAT --to-source 172.20.41.2

iptables -t nat -L -n -v --line-numbers

/usr/vpnserver # iptables -t nat -L -n -v --line-numbers
Chain PREROUTING (policy ACCEPT 685 packets, 148K bytes)
num pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 43 packets, 3806 bytes)
num pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 35084 packets, 2422K bytes)
num pkts bytes target prot opt in out source destination
1 42910 2962K DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11

Chain POSTROUTING (policy ACCEPT 70142 packets, 4841K bytes)
num pkts bytes target prot opt in out source destination
1 21 1784 SNAT all -- * * 192.168.33.0/24 0.0.0.0/0 to:172.20.41.2

Chain DOCKER_OUTPUT (1 references)
num pkts bytes target prot opt in out source destination
1 0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:41921
2 42910 2962K DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:52169

Chain DOCKER_POSTROUTING (0 references)
num pkts bytes target prot opt in out source destination
1 0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:41921 to::53
2 0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:52169 to::53

Cascade

A is the client connection entrance. Set up cascade B's SE in A's SE. Cluster CC, cluster CD, etc. may be added in series to connect to B's SE, such as A->B->C->D...

Client: window SoftEther VPN Client

Problem description:
Case 1: If using SecureNAT's vNAT and vDHCP, the requirements can be met, but the access speed is too slow;
Case 2: Using softether+dnsmasq+iptables, the client can access the CA intranet by connecting to A's SE, and the client can access the CB intranet by connecting to B's SE;
Case 3: Using softether+dnsmasq+iptables+cascade (site-to-site), the client connects to A's SE, sometimes responding to A's network segment (192.168.31.XXX), sometimes responding to B's network segment (192.168.33.XXX), when responding to A's network segment, it can only access the CA intranet, and when responding to B's network segment, it can only access the CB intranet. What is needed is that it should respond to A's network segment, and can access all intranets of CA and CB at the same time.

Please help me.

[solo]

Case 3: Using softether+dnsmasq+iptables+cascade (site-to-site), the client connects to A's SE, sometimes responding to A's network segment (192.168.31.XXX), sometimes responding to B's network segment (192.168.33.XXX)

With two DHCP servers on L2 network, no wonder you get random IP assignments.

[775118134]

With two DHCP servers on L2 network, no wonder you get random IP assignments.

Thank you for your reply.
I am not familiar with the network. How should I modify it?
Should I close dnsmasq in server B and delete iptables and point dhcp-option=tap_soft,249 and dhcp-option=tap_soft,121 of dnsmasq in server A to 192.168.31.1 of server A? 192.168.0.0/24,192.168.31.1,10.9.94.0/24,192.168.31.1,10.10.12.0/24,192.168.31.1

According to the above modification, the IP segments obtained are all set in server A and are normal. The cascade connection server B in server A is also online (established). The client can only ping the LAN in server cluster CA, but not the LAN in server cluster CB.

[solo]

Close dnsmasq B, don't delete iptables, and correct the routes.

[775118134]

Close dnsmasq B, don't delete iptables, and correct the routes.

Could you tell me the specific command to execute? I don't understand routing, thank you very much

[775118134]

Can an expert help me take a look?