High CPU Usage

Post your questions about SoftEther VPN software here. Please answer questions if you can afford.
Post Reply
JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

High CPU Usage

Post by JellyVPN » Fri Jan 06, 2023 11:26 pm

Hi there, CPU usage is too high for my servers
why is that happening?
how can I fix this?
50 users at the same time use more than 5Ghz CPU in 4 or 6 cores
is it a bug or something?
is there any workaround for this issue?
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Sat Jan 07, 2023 7:04 am

SecureNAT?
Precautions relating to Performance

By possessing an internal virtual TCP/IP stack, SecureNAT performs the highly advanced process of reassembling the TCP/IP stream packetized once by the TCP/IP stack and further TCP/IP packetizing via the operating system. The overhead resulting from these processes is large, such that throughput via the virtual NAT is considerably decreased when compared to physical maximum throughput, even when using a computer with sufficiently high speed. That is why virtual NAT should not be used for performance-centric applications. As previously stated, virtual NAT is a function which can be used as an alternative when the local bridge function cannot be used for security or technical reasons. Where high-speed methods such as local bridging are available, those methods should be used.

vpnfail
Posts: 14
Joined: Mon Jan 02, 2023 2:11 pm
Contact:

Re: High CPU Usage

Post by vpnfail » Sat Jan 07, 2023 8:09 am

Unfortunately high CPU is an issue we also encountered doing tests, especially for some VPN protocols such as what you're using. Was CPU usage half when the number of active users was half from what it is now, or did the CPU usage increase unrelated to the increase in users activity?
Free VPN that works. For V2Ray, WireGuard and OpenVPN - https://vpn.fail/
Real-time free proxy list. SOCKS, HTTP & V2RAY - https://vpn.fail/free-proxy

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Sat Jan 07, 2023 9:08 pm

Dear @solo, yes I'm using SecureNAt, can you explain me or do you have any documentation for running a local bridge using physical nat?
I can add ethernet to my VM, I tested but when I try to use the local bridge it's getting an error, Virtual Nat is working fine but still has too high usage for the CPU
so tell me how I can use a local bridge between two ethernets instead virtual one, I want to have maximum usage here
thanks a lot
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Sun Jan 08, 2023 1:17 am

Hello JellyVPN, from our past conversations I assume that you still use Linux for SE server. If so, a conversion from SecureNAT to local bridge with dnsmasq' DHCP and iptables' NAT is rather simple - we have discussed it recently here and here.

Is the VM on a VPS or LAN PC? What error did you get after adding ethernet?

Please add an ethernet and post from the VM, as code, the output of:

Code: Select all

ifconfig
vpncmd localhost:port /server /password:*** /cmd BridgeDeviceList
//replace: *** with SE admin password

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Mon Jan 09, 2023 9:40 pm

I fixed the issue for the local bridge, but still, CPU usage is too high
my servers are VM on ESXi 8
I have Centos 7, Centos 8, Windows Server 2019, 2022, and Ubuntu 22.10
Issues are
1. automatically disconnect users after 1-2 minutes.
2. High CPU usage even small users are connected
3. after some users are connected DHCP won't give IP to new users
thanks for your help Dear @solo
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Mon Jan 09, 2023 11:05 pm

Would you be able to compare Windows vs Linux servers?

On a Windows Server edition you can replace SecureNAT with native DHCP server and RRAS' NAT.
On a Windows non-server edition you could try "Open DHCP Server" and something like https://www.nat32.com/ for NAT.

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Fri Jan 13, 2023 10:33 pm

I checked on Windows and Linux with built-in Securenat, and both of them have High Cpu usage and High Memory usage
100% CPU and 100% RAM
I searched a lot and find it
https://github.com/SoftEtherVPN/SoftEth ... ssues/1616
it seems this huge issue is still not solved after years
for 3rd party DHCP it seems not working as expected and it's a very good idea unless Softether becomes more flexible with 3rd party apps
The best solution is to fix Softether Securenat usage
Best Regards
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Fri Jan 13, 2023 11:44 pm

Thank you for Windows vs Linux server tests!

While we're waiting for SecureNAT fix, can you re-configure the setup as follows?
- disable SecureNAT
- enable local bridge
- offload DHCP+NAT to another PC or a router

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Sat Jan 14, 2023 11:03 am

Can you explain how is possible to offload secure nat?
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Sat Jan 14, 2023 11:32 am

JellyVPN wrote:
Sat Jan 14, 2023 11:03 am
Can you explain how is possible to offload secure nat?
Sure...
Linux: dnsmasq' DHCP and iptables' NAT
Windows: native DHCP server and RRAS' NAT
Router: basic built-in function

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Sat Jan 21, 2023 11:09 pm

Thank you Dear @solo
I didn't check at windows due high usage CPU for windows itself
I'm trying to use Linux but not a successful scenario
1. I did a local bridge with a Virtual Tap adaptor (Softether VPN Server)
2. I installed dnsmasq and iptables in Ubuntu 22.10 (config as well with ipv4 forward active and tested)
but not working, I'm sure something is missing here
can you tell me steps until I can figure it how can I solve it
P.S: I installed ocserv on the same server and working very well without any issues by dnsmasq
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Sun Jan 22, 2023 12:59 am

Hello JellyVPN, this Softether on VPS Using Local Bridge guide is exactly what you ask for.

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Sun Jan 22, 2023 8:10 pm

Thanks the guide is very great
But I face a problem and couldn't resolve the issue, even with a lot of searching on Google
==========
Softether start-up script belongs to Centos, I have a script for Ubuntu 22.10 for Softether Startup but I don't know how can I use virtual adaptor for the bridge to this script

Code: Select all

[Unit]
Description=SoftEther VPN server 
After=network-online.target 
After=dbus.service

[Service]
Type=forking 
ExecStart=/opt/softether/vpnserver start 
ExecReload=/bin/kill -HUP $MAINPID 

[Install]
WantedBy=multi-user.target
==========
I added /etc/init.d/vpnserver based on the guide and only changed the IP Address based on my needs, but still can't use it
also in this folder, there is not file available
LOCK=/var/lock/subsys/vpnserver
==========
tap_soft will not give IPv4 to users, just IPv6
I did all the guide step by step, added to Firewall, dnsmasq, and more
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Sun Jan 22, 2023 10:27 pm


JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Mon Jan 23, 2023 12:15 am

my problem isn't startup, Local Bridge not working!!!
https://blog.lincoln.hk/blog/2013/05/17 ... al-bridge/
I did all steps correctly, still when user connect doesn't get IPv4
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Mon Jan 23, 2023 2:35 am

But the soft-tap bridge is working?
This is the only bridge you need.

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Mon Jan 23, 2023 6:30 am

No the problem is tap_soft installed, script for startup is active, but still users can't get IPv4
Something missed or has issue
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

solo
Posts: 1228
Joined: Sun Feb 14, 2021 10:31 am

Re: High CPU Usage

Post by solo » Mon Jan 23, 2023 7:26 am

To clarify, we're not creating a "Local Bridge" in your VPS context. You use only a soft tap to SE bridge. Typical gotchas of this Linux setup are: missing IP forwarding and restrictive firewall. Review these topics on a very similar dnsmasq/iptables application:
https://www.vpnusers.com/viewtopic.php? ... 926#p97433
https://www.vpnusers.com/viewtopic.php?f=7&t=67987

shakibamoshiri
Posts: 285
Joined: Wed Dec 28, 2022 9:10 pm

Re: High CPU Usage

Post by shakibamoshiri » Tue Jan 24, 2023 7:34 pm

JellyVPN wrote:
Mon Jan 23, 2023 6:30 am
No the problem is tap_soft installed, script for startup is active, but still users can't get IPv4
Something missed or has issue
Using Local Bridge and dnsmasq are not hard. You can follow below steps to check the issue stage

1. save your current iptables rule in order to restore it later

Code: Select all

iptables-save > your-file.v4
2. flush everything

Code: Select all

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -t security -F
iptables -t security -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
3. Enable SecureNat and test a user connectivity
if it passed next

4. Disable SecureNat and crate a soft bridge with SE server (check is has been created)

5. Manually assign IP to local bridge created in step 4

Code: Select all

ip addr add 10.11.12.1/24 brd + dev tap_tap
tap_tap is a soft bridge created
check the IP has been assigned to tap_tap (e.g ip -br a show tap_tap)

6. configure dnsmasq, then restart it and check the status

Code: Select all

interface=tap_tap
dhcp-range=10.11.12.10,10.11.12.250,12h
dhcp-option=3,10.11.12.1
dhcp-option=6,8.8.8.8
if it has port 53 conflict, in dnsmasq.conf file find port and change it to e.g. 5353 and restart it again

Code: Select all

port = 5353
7. check the same user can connect or not

NOTE
If the client/user Internet is too slow/weak they may face ERR_TIMEOUT because SE server DHCP is disabled and dnsmasq is near 3 or more times slower for IP assignment. I have tested with SSTP:
- SecureNAt IP assignments takes 1 or 2 seconds
- dnsmasq IP assignment takes 3 to 10 seconds or fails

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Wed Jan 25, 2023 4:03 am

Thank you Dear @solo
Thank you Dear @shakibamoshiri
problem is solved
But Dear shakibamoshiri I have a concern about CPU usage, with Softether's DHCP CPU usage is hiking
I didn't check with dnsmasq at least for many users, Softether's DHCP after almost 120-200 users won't give IP and server CPU and RAM usages hiking to 100% without dropping even 1 second
after a lot of checks, I got this issue DHCP using a lot of CPU
now I'm concerned about dnsmasq is better or Softether's own NAT
===========================
#Issue 2:
The new Issue I'm facing is with SSTP clients after connect automatically disconnects after a few seconds, I have no clue why it's happening
===================================
#Issue 3:
some users when trying to connect to servers after disconnecting get the below error while trying to reconnect (SSTP Android and iOS)
SSL Connect Error: BROKEN_PIPE
===================================
#Issue 3:
as I understand Dear shakibamoshiri you are providing VPN in Iran, while we have several countries and several servers some users can't connect to some of them
sometimes Irancell, sometimes MCI, and so more
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
I'm confused, all of them use the same config but issue still exists
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

shakibamoshiri
Posts: 285
Joined: Wed Dec 28, 2022 9:10 pm

Re: High CPU Usage

Post by shakibamoshiri » Wed Jan 25, 2023 8:42 pm

JellyVPN wrote:
Wed Jan 25, 2023 4:03 am
Thank you Dear @solo
Thank you Dear @shakibamoshiri
problem is solved
But Dear shakibamoshiri I have a concern about CPU usage, with Softether's DHCP CPU usage is hiking
I didn't check with dnsmasq at least for many users, Softether's DHCP after almost 120-200 users won't give IP and server CPU and RAM usages hiking to 100% without dropping even 1 second
after a lot of checks, I got this issue DHCP using a lot of CPU
now I'm concerned about dnsmasq is better or Softether's own NAT
===========================
#Issue 2:
The new Issue I'm facing is with SSTP clients after connect automatically disconnects after a few seconds, I have no clue why it's happening
===================================
#Issue 3:
some users when trying to connect to servers after disconnecting get the below error while trying to reconnect (SSTP Android and iOS)
SSL Connect Error: BROKEN_PIPE
===================================
#Issue 3:
as I understand Dear shakibamoshiri you are providing VPN in Iran, while we have several countries and several servers some users can't connect to some of them
sometimes Irancell, sometimes MCI, and so more
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
I'm confused, all of them use the same config but issue still exists
Personally I prefer using SE Secure NAT it is match faster in terms of DHCP IP allocation and assignment BUT we know NAT is a kind of high CPU consumption process and it is better to delegate this to Linux Kernel which is highly optimized. Since we give this process to OS, IP allocation and assignment will be slower BUT less pressure will be on SE server and respectively on CPU.

dnsmasq
I did not tested it with high number of users but according to others , it seems to be a better choice for large scale use cases

#Issue 2:
if they can connect successfully and disconnected after a while mostly could be their ISP issue or like Iran deliberately done by ISPs. In this regard we cannot expect a long stable connectivity.

#Issue 3:
I never git this (BROKEN_PIPE) with SSTP, which client you use?
This error is common with SSH ing to a server and again mostly cased by ISP

#Issue 4:
as I understand Dear shakibamoshiri you are providing VPN in Iran,
disclaimer
I setup VPN for companies and mostly I used OpenCoonect but got interested in SE as well recently
WE DO NOT SELL VPNS
sometimes Irancell, sometimes MCI, and so more
All of them of terrible. none of them are good but we have use them. they are pretty unstable and expensive
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
Yes this is true
I explained it here
https://www.vpnusers.com/viewtopic.php? ... 011#p97757
I'm confused, all of them use the same config but issue still exists
Stable VPN connections need
1. stable server
2. stable network
3. working protocols

Number 2 and 3 are hard to find in Iran :)

JellyVPN
Posts: 44
Joined: Sun May 25, 2014 3:37 pm
Contact:

Re: High CPU Usage

Post by JellyVPN » Sat Jan 28, 2023 6:24 pm

Thank you Dear Shakiba for your information
Issue #2 still exists:
customer use VPN Client Pro on Android and sometimes get error (SSL Connect Error: BROKEN_PIPE)
I don't know the reason and can't find a solution yet
any help or clue will be great
#1 Security, Speed, and customer service ;)
JellyVPN - https://jellyvpn.com

shakibamoshiri
Posts: 285
Joined: Wed Dec 28, 2022 9:10 pm

Re: High CPU Usage

Post by shakibamoshiri » Sat Jan 28, 2023 7:58 pm

JellyVPN wrote:
Sat Jan 28, 2023 6:24 pm
Thank you Dear Shakiba for your information
Issue #2 still exists:
customer use VPN Client Pro on Android and sometimes get error (SSL Connect Error: BROKEN_PIPE)
I don't know the reason and can't find a solution yet
any help or clue will be great
To me the issue is the network.
Practically you have these solutions and two are based on double-vpn.

First (double-vpn)
If you can have server in Iran, buy and use it as hop-1 and CC it to your end-hop
pors
- less disconnection
- much more stable
- almost all protocols work
cons
- hiding your identity
- rarely ISPs in Iran give semiofficial bandwidth (1 to 1) and you have keep buying more traffic

Second (double-vpn)
If the "First" one was not possible for you, but a server in Turkey which has the closed route to Iran and make that Turkey's server as hop-1
pros
- less disconnection
- much more stable
- no need to hide
- you may can buy semiofficial bandwidth
cons
- hard to find working protocols

Third (Normal vpn)
just a server in Turkey. As I said Turkey has the closest route to Iran. ping could be near 70ms. which to Germany is near 120 to 150 , to USA more than 200 ms.

Lastly at the moment I am wring this reply, no ISP in Iran has stable network. Even domestic servers somethings cannot ping each other.

Post Reply