Routing Instead of Bridging

[JimG]

I'm new to this forum, but I've been using SoftEther for along time and I've seen a recurring problem I can't identify.

First some explanation of the Network Architecture:
- Imagine 3 locations where you want to have a fully routed network between Site 1, Site 2 and Site 3.
- Site 1 connects to Site 2 and Site 3 connects to Site 2 (basic hub and spoke WAN, Site 2 is the hub, Sites 1 and 3 are the spokes).
- Windows 2012 R2 Servers runs at 3 locations, each server hosting a VM running the same OS which is running Softether.
- All VMs running Softether have all Advance Network setting turned off (IPSec Offload, Tran/Rec Offload, etc) both on the VM config and within each Virtual Network Adapter.
- Due to broadcasts on the local LANs, it's very undesirable to bridge any of these LAN segments together (one site has 10 Gb Ethernet, so it's easy to crush the Internet connection with broadcasts)
- Each site has a unique IP address range (10.1.1.0/24, 10.1.2.0/24, 10.1.3.0/24) (as is often the case)
- At each site, the Internet firewall is a VM running PFSense with the .1 address in the given subnet.
- All devices at each site use the Firewall (.1 address) as the default gateway.
- At each site the Softether server's IP address is .3
- At each site, in each Softether server, a local hub is created and connected via the Layer 3 Switch to the local LAN at the .2 address.
- At each site, the local firewall routes the 2 non-local subnets to the .2 address
- To connect Site 1, at Site 1, a hub is created called Site1ToSite2 and at Site 2 another hub is created called Site2ToSite1. A cascade connection is created between these two hubs to connect them.
- At Site 1, the Layer 3 Switch is configured to connect to hub Site1toSite2 using IP Address 172.16.1.2/24, while at Site 2 the Layer 3 Switch is configured to connect to hub Site2toSite1 using IP 172.16.1.1/24
- Similarly Site 3 is connected to Site 2 with Site 2 hub Site2toSite3 and Site hub Site3ToSite2.
- Site2toSite3 is connected to 172.16.2.1/24 and Site3ToSite2 is connected to 172.16.2.2
- In the layer 3 switch, routes are created at each site to route to the other subnets via the gateway in the hub on the other site.
- So at Site 1, the routes for 10.1.2.0/24 and 10.1.3.0/24 use the gateway at 172.16.1.1 (the IP on the hub cascaded from site 1 to Site 2.
- At Site 3, the routes for 10.1.1.0/24 and 10.1.2.0/24 use the gateway at 172.16.2.1 (again, the IP on the hub cascaded from site 3 to site 2.
- At Site 2, the route for 10.1.2.0/24 uses the gateway at 172.16.1.2 (the IP on the Site 2 side of that cascade), AND a second route - for 10.1.3.0/24, the gateway is 172.16.3.2 (the IP on the Site 3 side of that cascade).

So the problem is, this generally works for a time and then something goes crazy in the software and the VPN WAN connection to one of the sites comes and goes. Let's assume the connection between Site 1 and Site 2 is having this problem, Packet captures on the Site 2 Softether server show huge numbers of packets from the Site 1 Softether server on Port 40000 or 40001 whenever the WAN link is down (no pings will pass). The WAN connection cycles every 7 to 15 seconds. The CPU load on both servers is huge (4 processors, 90% loaded) and the smallest bandwidth pipe is 100% loaded.

My only solution so far has been to rebuild the Softether server at the remote site (Site 1 in this example) and that doesn't always work.

Does anyone have any suggestions?

Does anyone know what these packet on port 40000 are about?

Any help would be appreciated.

JimG

[thisjun]

I think there is loop. Please check logs.