Page 1 of 1

High CPU Usage

Posted: Fri Jan 06, 2023 11:26 pm
by JellyVPN
Hi there, CPU usage is too high for my servers
why is that happening?
how can I fix this?
50 users at the same time use more than 5Ghz CPU in 4 or 6 cores
is it a bug or something?
is there any workaround for this issue?

Re: High CPU Usage

Posted: Sat Jan 07, 2023 7:04 am
by solo
SecureNAT?
Precautions relating to Performance

By possessing an internal virtual TCP/IP stack, SecureNAT performs the highly advanced process of reassembling the TCP/IP stream packetized once by the TCP/IP stack and further TCP/IP packetizing via the operating system. The overhead resulting from these processes is large, such that throughput via the virtual NAT is considerably decreased when compared to physical maximum throughput, even when using a computer with sufficiently high speed. That is why virtual NAT should not be used for performance-centric applications. As previously stated, virtual NAT is a function which can be used as an alternative when the local bridge function cannot be used for security or technical reasons. Where high-speed methods such as local bridging are available, those methods should be used.

Re: High CPU Usage

Posted: Sat Jan 07, 2023 8:09 am
by vpnfail
Unfortunately high CPU is an issue we also encountered doing tests, especially for some VPN protocols such as what you're using. Was CPU usage half when the number of active users was half from what it is now, or did the CPU usage increase unrelated to the increase in users activity?

Re: High CPU Usage

Posted: Sat Jan 07, 2023 9:08 pm
by JellyVPN
Dear @solo, yes I'm using SecureNAt, can you explain me or do you have any documentation for running a local bridge using physical nat?
I can add ethernet to my VM, I tested but when I try to use the local bridge it's getting an error, Virtual Nat is working fine but still has too high usage for the CPU
so tell me how I can use a local bridge between two ethernets instead virtual one, I want to have maximum usage here
thanks a lot

Re: High CPU Usage

Posted: Sun Jan 08, 2023 1:17 am
by solo
Hello JellyVPN, from our past conversations I assume that you still use Linux for SE server. If so, a conversion from SecureNAT to local bridge with dnsmasq' DHCP and iptables' NAT is rather simple - we have discussed it recently here and here.

Is the VM on a VPS or LAN PC? What error did you get after adding ethernet?

Please add an ethernet and post from the VM, as code, the output of:

Code: Select all

ifconfig
vpncmd localhost:port /server /password:*** /cmd BridgeDeviceList
//replace: *** with SE admin password

Re: High CPU Usage

Posted: Mon Jan 09, 2023 9:40 pm
by JellyVPN
I fixed the issue for the local bridge, but still, CPU usage is too high
my servers are VM on ESXi 8
I have Centos 7, Centos 8, Windows Server 2019, 2022, and Ubuntu 22.10
Issues are
1. automatically disconnect users after 1-2 minutes.
2. High CPU usage even small users are connected
3. after some users are connected DHCP won't give IP to new users
thanks for your help Dear @solo

Re: High CPU Usage

Posted: Mon Jan 09, 2023 11:05 pm
by solo
Would you be able to compare Windows vs Linux servers?

On a Windows Server edition you can replace SecureNAT with native DHCP server and RRAS' NAT.
On a Windows non-server edition you could try "Open DHCP Server" and something like https://www.nat32.com/ for NAT.

Re: High CPU Usage

Posted: Fri Jan 13, 2023 10:33 pm
by JellyVPN
I checked on Windows and Linux with built-in Securenat, and both of them have High Cpu usage and High Memory usage
100% CPU and 100% RAM
I searched a lot and find it
https://github.com/SoftEtherVPN/SoftEth ... ssues/1616
it seems this huge issue is still not solved after years
for 3rd party DHCP it seems not working as expected and it's a very good idea unless Softether becomes more flexible with 3rd party apps
The best solution is to fix Softether Securenat usage
Best Regards

Re: High CPU Usage

Posted: Fri Jan 13, 2023 11:44 pm
by solo
Thank you for Windows vs Linux server tests!

While we're waiting for SecureNAT fix, can you re-configure the setup as follows?
- disable SecureNAT
- enable local bridge
- offload DHCP+NAT to another PC or a router

Re: High CPU Usage

Posted: Sat Jan 14, 2023 11:03 am
by JellyVPN
Can you explain how is possible to offload secure nat?

Re: High CPU Usage

Posted: Sat Jan 14, 2023 11:32 am
by solo
JellyVPN wrote:
Sat Jan 14, 2023 11:03 am
Can you explain how is possible to offload secure nat?
Sure...
Linux: dnsmasq' DHCP and iptables' NAT
Windows: native DHCP server and RRAS' NAT
Router: basic built-in function

Re: High CPU Usage

Posted: Sat Jan 21, 2023 11:09 pm
by JellyVPN
Thank you Dear @solo
I didn't check at windows due high usage CPU for windows itself
I'm trying to use Linux but not a successful scenario
1. I did a local bridge with a Virtual Tap adaptor (Softether VPN Server)
2. I installed dnsmasq and iptables in Ubuntu 22.10 (config as well with ipv4 forward active and tested)
but not working, I'm sure something is missing here
can you tell me steps until I can figure it how can I solve it
P.S: I installed ocserv on the same server and working very well without any issues by dnsmasq

Re: High CPU Usage

Posted: Sun Jan 22, 2023 12:59 am
by solo
Hello JellyVPN, this Softether on VPS Using Local Bridge guide is exactly what you ask for.

Re: High CPU Usage

Posted: Sun Jan 22, 2023 8:10 pm
by JellyVPN
Thanks the guide is very great
But I face a problem and couldn't resolve the issue, even with a lot of searching on Google
==========
Softether start-up script belongs to Centos, I have a script for Ubuntu 22.10 for Softether Startup but I don't know how can I use virtual adaptor for the bridge to this script

Code: Select all

[Unit]
Description=SoftEther VPN server 
After=network-online.target 
After=dbus.service

[Service]
Type=forking 
ExecStart=/opt/softether/vpnserver start 
ExecReload=/bin/kill -HUP $MAINPID 

[Install]
WantedBy=multi-user.target
==========
I added /etc/init.d/vpnserver based on the guide and only changed the IP Address based on my needs, but still can't use it
also in this folder, there is not file available
LOCK=/var/lock/subsys/vpnserver
==========
tap_soft will not give IPv4 to users, just IPv6
I did all the guide step by step, added to Firewall, dnsmasq, and more

Re: High CPU Usage

Posted: Sun Jan 22, 2023 10:27 pm
by solo

Re: High CPU Usage

Posted: Mon Jan 23, 2023 12:15 am
by JellyVPN
my problem isn't startup, Local Bridge not working!!!
https://blog.lincoln.hk/blog/2013/05/17 ... al-bridge/
I did all steps correctly, still when user connect doesn't get IPv4

Re: High CPU Usage

Posted: Mon Jan 23, 2023 2:35 am
by solo
But the soft-tap bridge is working?
This is the only bridge you need.

Re: High CPU Usage

Posted: Mon Jan 23, 2023 6:30 am
by JellyVPN
No the problem is tap_soft installed, script for startup is active, but still users can't get IPv4
Something missed or has issue

Re: High CPU Usage

Posted: Mon Jan 23, 2023 7:26 am
by solo
To clarify, we're not creating a "Local Bridge" in your VPS context. You use only a soft tap to SE bridge. Typical gotchas of this Linux setup are: missing IP forwarding and restrictive firewall. Review these topics on a very similar dnsmasq/iptables application:
https://www.vpnusers.com/viewtopic.php? ... 926#p97433
https://www.vpnusers.com/viewtopic.php?f=7&t=67987

Re: High CPU Usage

Posted: Tue Jan 24, 2023 7:34 pm
by shakibamoshiri
JellyVPN wrote:
Mon Jan 23, 2023 6:30 am
No the problem is tap_soft installed, script for startup is active, but still users can't get IPv4
Something missed or has issue
Using Local Bridge and dnsmasq are not hard. You can follow below steps to check the issue stage

1. save your current iptables rule in order to restore it later

Code: Select all

iptables-save > your-file.v4
2. flush everything

Code: Select all

iptables -F
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -t raw -F
iptables -t raw -X
iptables -t security -F
iptables -t security -X
iptables -P INPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -P OUTPUT ACCEPT
3. Enable SecureNat and test a user connectivity
if it passed next

4. Disable SecureNat and crate a soft bridge with SE server (check is has been created)

5. Manually assign IP to local bridge created in step 4

Code: Select all

ip addr add 10.11.12.1/24 brd + dev tap_tap
tap_tap is a soft bridge created
check the IP has been assigned to tap_tap (e.g ip -br a show tap_tap)

6. configure dnsmasq, then restart it and check the status

Code: Select all

interface=tap_tap
dhcp-range=10.11.12.10,10.11.12.250,12h
dhcp-option=3,10.11.12.1
dhcp-option=6,8.8.8.8
if it has port 53 conflict, in dnsmasq.conf file find port and change it to e.g. 5353 and restart it again

Code: Select all

port = 5353
7. check the same user can connect or not

NOTE
If the client/user Internet is too slow/weak they may face ERR_TIMEOUT because SE server DHCP is disabled and dnsmasq is near 3 or more times slower for IP assignment. I have tested with SSTP:
- SecureNAt IP assignments takes 1 or 2 seconds
- dnsmasq IP assignment takes 3 to 10 seconds or fails

Re: High CPU Usage

Posted: Wed Jan 25, 2023 4:03 am
by JellyVPN
Thank you Dear @solo
Thank you Dear @shakibamoshiri
problem is solved
But Dear shakibamoshiri I have a concern about CPU usage, with Softether's DHCP CPU usage is hiking
I didn't check with dnsmasq at least for many users, Softether's DHCP after almost 120-200 users won't give IP and server CPU and RAM usages hiking to 100% without dropping even 1 second
after a lot of checks, I got this issue DHCP using a lot of CPU
now I'm concerned about dnsmasq is better or Softether's own NAT
===========================
#Issue 2:
The new Issue I'm facing is with SSTP clients after connect automatically disconnects after a few seconds, I have no clue why it's happening
===================================
#Issue 3:
some users when trying to connect to servers after disconnecting get the below error while trying to reconnect (SSTP Android and iOS)
SSL Connect Error: BROKEN_PIPE
===================================
#Issue 3:
as I understand Dear shakibamoshiri you are providing VPN in Iran, while we have several countries and several servers some users can't connect to some of them
sometimes Irancell, sometimes MCI, and so more
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
I'm confused, all of them use the same config but issue still exists

Re: High CPU Usage

Posted: Wed Jan 25, 2023 8:42 pm
by shakibamoshiri
JellyVPN wrote:
Wed Jan 25, 2023 4:03 am
Thank you Dear @solo
Thank you Dear @shakibamoshiri
problem is solved
But Dear shakibamoshiri I have a concern about CPU usage, with Softether's DHCP CPU usage is hiking
I didn't check with dnsmasq at least for many users, Softether's DHCP after almost 120-200 users won't give IP and server CPU and RAM usages hiking to 100% without dropping even 1 second
after a lot of checks, I got this issue DHCP using a lot of CPU
now I'm concerned about dnsmasq is better or Softether's own NAT
===========================
#Issue 2:
The new Issue I'm facing is with SSTP clients after connect automatically disconnects after a few seconds, I have no clue why it's happening
===================================
#Issue 3:
some users when trying to connect to servers after disconnecting get the below error while trying to reconnect (SSTP Android and iOS)
SSL Connect Error: BROKEN_PIPE
===================================
#Issue 3:
as I understand Dear shakibamoshiri you are providing VPN in Iran, while we have several countries and several servers some users can't connect to some of them
sometimes Irancell, sometimes MCI, and so more
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
I'm confused, all of them use the same config but issue still exists
Personally I prefer using SE Secure NAT it is match faster in terms of DHCP IP allocation and assignment BUT we know NAT is a kind of high CPU consumption process and it is better to delegate this to Linux Kernel which is highly optimized. Since we give this process to OS, IP allocation and assignment will be slower BUT less pressure will be on SE server and respectively on CPU.

dnsmasq
I did not tested it with high number of users but according to others , it seems to be a better choice for large scale use cases

#Issue 2:
if they can connect successfully and disconnected after a while mostly could be their ISP issue or like Iran deliberately done by ISPs. In this regard we cannot expect a long stable connectivity.

#Issue 3:
I never git this (BROKEN_PIPE) with SSTP, which client you use?
This error is common with SSH ing to a server and again mostly cased by ISP

#Issue 4:
as I understand Dear shakibamoshiri you are providing VPN in Iran,
disclaimer
I setup VPN for companies and mostly I used OpenCoonect but got interested in SE as well recently
WE DO NOT SELL VPNS
sometimes Irancell, sometimes MCI, and so more
All of them of terrible. none of them are good but we have use them. they are pretty unstable and expensive
for example, someone easily connects to the USA and the same person can't connect to France, on the other hand, another one can connect to France and can't to the USA
Yes this is true
I explained it here
https://www.vpnusers.com/viewtopic.php? ... 011#p97757
I'm confused, all of them use the same config but issue still exists
Stable VPN connections need
1. stable server
2. stable network
3. working protocols

Number 2 and 3 are hard to find in Iran :)

Re: High CPU Usage

Posted: Sat Jan 28, 2023 6:24 pm
by JellyVPN
Thank you Dear Shakiba for your information
Issue #2 still exists:
customer use VPN Client Pro on Android and sometimes get error (SSL Connect Error: BROKEN_PIPE)
I don't know the reason and can't find a solution yet
any help or clue will be great

Re: High CPU Usage

Posted: Sat Jan 28, 2023 7:58 pm
by shakibamoshiri
JellyVPN wrote:
Sat Jan 28, 2023 6:24 pm
Thank you Dear Shakiba for your information
Issue #2 still exists:
customer use VPN Client Pro on Android and sometimes get error (SSL Connect Error: BROKEN_PIPE)
I don't know the reason and can't find a solution yet
any help or clue will be great
To me the issue is the network.
Practically you have these solutions and two are based on double-vpn.

First (double-vpn)
If you can have server in Iran, buy and use it as hop-1 and CC it to your end-hop
pors
- less disconnection
- much more stable
- almost all protocols work
cons
- hiding your identity
- rarely ISPs in Iran give semiofficial bandwidth (1 to 1) and you have keep buying more traffic

Second (double-vpn)
If the "First" one was not possible for you, but a server in Turkey which has the closed route to Iran and make that Turkey's server as hop-1
pros
- less disconnection
- much more stable
- no need to hide
- you may can buy semiofficial bandwidth
cons
- hard to find working protocols

Third (Normal vpn)
just a server in Turkey. As I said Turkey has the closest route to Iran. ping could be near 70ms. which to Germany is near 120 to 150 , to USA more than 200 ms.

Lastly at the moment I am wring this reply, no ISP in Iran has stable network. Even domestic servers somethings cannot ping each other.