unsuccessful cascadereconnects spawns crash and securityhole

[treintje2002]

Hi everyone,

I discovered the following issue:
Unsuccessful cascade reconnects will spawn vpnservice crash/restart, and opens up a security gap.
When links stays up (no reconnects needed) or short interuptions (eg reboot), link and service is 100% stable.


We successfully installed softether vpnserver (linux ARM eabi version of 24/4/16) on two clean "pi" devices (one bananapi, and one raspi3) and they can both serve vpn-sessions just fine.

Between the two devices (geographically separated) there is a cascade connection, which connects from "banana" to "raspberry".

The banana has two virtualhubs, one for incoming vpn connections, and one for only the cascade.
Both hubs also have their own bridged tapdevice.

The "raspberry" has four hubs, of which one is for the cascade.

"Banana's" cascaded virtualhub connects to this "raspberry" virtualhub.

This works GREAT, thanks to softhether functionality. Connections get picked up automatically, dhcp addresses are successfully distributed. Also after reboots etc. Real resilient.


Problem arises on BANANA, when for some reason banana cannot "see" raspberry for a FEW HOURS over the cascade.... ("raspberry" has no apparent problem, so far as we can see now)

1)After a few hours of unsuccessful connections retries, the vpnserver service (re)starts itself every few seconds, and
2) if this keeps going on, at some point the vpnserver looses it's config and reverts to default.
During this entire period the vpnservice keeps trying to restart, and is unresponsive (for incoming vpn connections).

If you restart (powerdown/powerup) the "banana" device completely, it starts vpnserver successfully again, but with DEFAULT settings.
From which point on EVERYBODY with a softether servermanager, can connect to your device and choose a new administrator pw.
From then on, he/she has FULL access to your network!!!

Needless to say: This is VERY BAD!! (understatement)
Any use of standard softether ports in this scenario is "security-suicide".
If you for some reason need p443, your are pretty much sc****ed....
(but we DO need it unfortunately)


At the end of this post, I will include snippets of the log that will clearify it some more.

So the problem is twofold:
-The restarts after prolonged unsuccessful reconnects (WHY??).
-And the change to DEFAULT.

The latter may be caused by the automatic config-restore function build in softether which doesn't have enough time to finish due to the continuous restarts.
Might be easily disabled (by "bool DontBackupConfig false" ??? please advise), or circumvented by removing write privileges on the .config file (not really desirable)

But the underlying issue (the first issue, aka the "constant service restarts"), I cannot fix myself. It seems to be a bug.
There is also no way to reduce the frequency of cascade reconnections (greyed out) to reduce the chance that this will happen (this is possible in the softether"client" version)


Please advise on creating a SAFE and STABLE situation in case of prolonged interrupted periods.....

Thank You.


/edit: found in syslog a lot of these:
Oct 19 23:24:05 bananapi kernel: [352255.490810] lowmemorykiller: Killing 'vpnserver' (21982), adj 800,
Oct 19 23:24:05 bananapi kernel: [352255.490818] to free 12304kB on behalf of 'kswapd0' (26) because
Oct 19 23:24:05 bananapi kernel: [352255.490822] cache 44616kB is below limit 65536kB for oom_score_adj 12
Oct 19 23:24:05 bananapi kernel: [352255.490827] Free memory is -38424kB above reserved
Oct 19 23:24:07 bananapi kernel: [352256.712735] lowmemorykiller: Killing 'vpnserver' (22085), adj 800,
Oct 19 23:24:07 bananapi kernel: [352256.712742] to free 12628kB on behalf of 'kswapd0' (26) because
Oct 19 23:24:07 bananapi kernel: [352256.712747] cache 43588kB is below limit 65536kB for oom_score_adj 12
Oct 19 23:24:07 bananapi kernel: [352256.712752] Free memory is -37216kB above reserved

Seems like a MASSIVE vpnserver memory leak while trying to do reconnect (since the device normally only needs about 80MB to run all programs including vpnserver (out of a total 1024MB)).
Kernel is trying to survive, and kills "vpnserver", which accounts for the constant restarts.

So the underlying problem changes from "why restart" to "why memoryleak"?
Softether devellopers???? Anyone??

[attachment=0]vpnlog.txt[/attachment]

[treintje2002]

Is there really nobody concerned about this issue? Or is nobody actually using cascades??
Come on people! In particular the mod's/softether developers. This is a serious matter....
The only response until now was a troll trying to lure people to his/hers infested website.

This is a fantastic piece of sw, and I would like to continue using it, but this is really a showstopper.
Especially sad because it is an issue in a re-connection routine, not in the actual link handling in steady-state (which is excellent).

I'm trying to script a workaround by probing and (when needed) shutting down the cascade via vpncmd. And in that way preventing the unsuccessful (and flawed) reconnects at 10s intervals.
But making this script creates even more security issues on its own, since 1)vpncmd requires plain-text passwords in the command, and 2)probing triggers the (softether) DOS attack modules on the remote site, which then needs to be disabled.... Not exactly what one would like to do if it is required to use port 443.

I myself am not a programmer, but an electronics engineer. Some scripting I can do.
But even if I would have access to the softether sources, this would be way over my head.

So I really need assistance of the softether experts out here.
When resolved, it would mean a real benefit to ALL softether server users out there!!

Thx

[moatazelmasry]

Hi there,

i can understand the frustration. I don't use the cascade functionality myself, so I can't help much. But to address your other points:
- For some reason it doesn't look like this project is actively developed (I hope I'm wrong)
- you might have more luck reporting this issue on SoftEther github page

Good luck,
Moataz

[treintje2002]

Hi Moataz,

Thank You for responding. You are the first serious responder out of 140 viewers. I almost gave up.
(except for an unrelated troll message, which the mods deleted. So they ARE reading, they're just not DOING anything with the issue)
I too hope softethervpn is still under active development, since it is a mighty fine piece of software.
Last release is from April 24, and beta. still a reasonable age, considering the nature of the project.
Although the last two pictures on the bottom of this page* aren't exactly encouraging this hope. :(
(maybe smart to look for an alternative DDNS source too, if they run DDNS from there)

I will take your advise and try Github.

Thank You for taking the trouble to respond. The tip is much appreciated.

Kind Regards.

Richard.


*https://www.softether.org/9-about

[dnobori]

Hello Richard,

I am a developer of SoftEther VPN.
Thank you so much for your report.

I am going to investigate the phenomenon as soon as possible.

Today I have tried to reproduce your reported problem, however, currently I have not succeeded to reproduce it.

I have installed the latest VPN Server on both x64 and arm-eabi environment, and let VPN servers to continue to reconnect infinity. However, I have not observed any memory leaks on every reconnect attempts yet. I have checked this on both the in-process memory leak checker and the valgrind utility.

Therefore, I am making my effort to reproduce the memory-leak problem which you reported.

I read your post in the forum, and you mentioned that you are using Raspberry Pi 3.

Because there are no similar reports of memory-leaks on the reconnect attempts of the cascading function yet for general environment, I suspect that something is wrong on the built-in library of some specific version for Raspberry Pi 3. I believe that I should investigate and make the cause clear where the problem exists.

Since I do not have any available Raspberry Pi 3, today I just placed a purchase order for a Raspberry Pi 3 through Amazon to reproduce your reported environment on my side. A Raspberry Pi 3 will be delivered tomorrow to my home. Then I will continue testing on a Raspberry Pi 3 to try reproducing your reported phenomenon.

On the config-file-corrupt problem, I also suspect that it could be caused by the special condition of the file system. Raspberry Pi 3 should have a SD card-based file storage, and it could have an unexpected race condition on the complicated file I/O and create/delete/rename operation. To investigate this, I also need a real Raspberry Pi 3 in my testing environment. Therefore, I am waiting for Raspberry Pi 3's delivery.

Anyway, I will reply to you about the result of my reproducing test with Raspberry Pi 3.

[treintje2002]

Hi Dnobori,

Thank you for willingness to investigate.

Indeed we use "pi's" to connect the locations. One BananaPI, and one RaspberryPI3
BananaPI connects to PI3 via a cascade. Therefore the behavior was first observed there.

But inverting the cascade (define cascade on pi3 connecting towards BananaPi) exhibits the same behavior, albeit far less severe. Therefore you might have a very good point that there is a relation to the platform.

Anyway, thank you very much for starting investigations, as this is potentially a very serious issue.

If I can do anything to assist (like testing something, etc), let me know.

[dnobori]

Hello Richard,

I purchased a Raspberry Pi 3 and have conducted the memory leak tests with retrying cascade connection attempts for 8 days.
(I do not have a banana pi, and it is unavailable in my country. However, testing in the general ARM system, using Raspberry Pi 3 is an adequate device.)

As the result I found no memory leaks on it.

The vpnserver process have tired +80000 times to retry to connect, however, the memory consumption amount did not change. The initial memory consumption was 38Mbytes. After 8 days later, the process still using 38M bytes.

Therefore, I conclude that this problem does not happen on the usual Linux environment, including ARM system like Raspberry Pi.

I suspect that your ARM system (banana pi) has a library which have a potential problem on the memory leak or some kinds of problem. I cannot try your particular device (banana pi) to reproduce the problem because I do not have a banana pi. Please consider to use the proved and stable Linux ARM systems to run a VPN Server.

Also, there are no other similar reports on this. So I suppose that this is a problem depending on the specific environment as wrote above.

Additionally, for the file corruption problem in the crash timing (e.g. caused by your banana pi) I am going to add a dual-configuration file saving mechanism to realize more securely check mechanism in the next build.

The actual results:
-------
[root@raspi3dev1 ~]# uname -a
Linux raspi3dev1 4.4.21-v7+ #911 SMP Thu Sep 15 14:22:38 BST 2016 armv7l GNU/Linux

2016.11.08 10:28
root 1175 3.0 1.7 38068 16772 ? S<l Nov04 172:09 \_ /root/v1/vpnserver/vpnserver execsvc

2016.11.16 08:45
root 1175 2.9 1.8 38068 17224 ? S<l Nov04 511:21 \_ /root/v1/vpnserver/vpnserver execsvc
-------

Thank you.

[treintje2002]

Hi Dnobori,

Thank You very much for taking the time and trouble of testing this on the Pi3. Greatly appreciated!
Also great that you will strengthen the auto config-backup strategy.
(although a simple 2 file strategy won't work either in case of corruption, that just delays total failure. Something more complex like eg. checksum verification with conditional rotation will be needed here)

I concur that it is almost impossible to test all software combinations with all different libraries available on all different platforms available.
Also concur that it must be a very rare combination of factors leading to the leak. It must be a interaction between "something" in bananian (debian for bananapi) and the cascade routines.
Otherwise leaks on Raspberry should be equal to BananiPi, and they are not.
(But they DO exist on both, even on the platform what you deem "proved and stable Linux ARM systems": the pi3. Clearly in your test you do not have the offending component installed yet....)

But I do not concur with your conclusion on the following:
The Bananapi-Debian (bananian) is as plain-vanilla as it gets, and it's rock stable as far as NOT using Softether cascades. There is no-more online evidence of memory leaks on this platform/OS, as there is online-evidence of memory leaks caused by Softether-cascades. Stop using the cascade will also stop the leak, so cascade is most definitely a key factor here.
Sure Bananapi is not as popular as Raspberry, but RaspberryPi (or ARM in general) is not as common as x86 either. So drop ARM altogether then?? Well, ofcourse not...
Unfortunately, in this forum is/was at least one more report of leaks that I found during my initial search, but NObody bothered to even reply..... Sure very rare occurrence, but nevertheless very sad nobody seemed to care...

As mentioned, I realize that testing everything is impossible. And I don't expect You to do so.
I appreciate your efforts so far on this testing.

That brings me to the next point: I mentioned in a previous post that I was working on a script that block the situation from becoming a problem by stopping the cascade. This seems to work.
But VPNCMD still needs a plaintext admin password?
Is there a way around this?

Or can the remote-server probing be done from within the Softether Vhub BEFORE actual trying to reconnect (so that no extra external command is needed)?

Or something with "signals" (eg SIGUSR) or something to trigger something from outside of the Softether software????

Or .... whatever...?

....Really hate plaintext passwords.... they're so nastily insecure....


Thank You for your efforts....