Feature #2071

Survive connection loss

Added by S. Wrede almost 6 years ago. Updated about 5 years ago.

Status:ResolvedStart date:11/03/2014
Priority:NormalDue date:
Assignee:-% Done:

0%

Category:-
Target version:-

Description

Currently, VDemo is very sensitive to connection loss. If a connection to a node within a setup is down and comes up again, VDemo needs to be restarted completely.

Given more complex setups (such as the one we currently use with about 20 nodes) and long-running processes, this is really annoying. Hence, it would be really great if VDemo could also monitor node status and re-initiate ssh connections / and screen sessions at runtime.


Related issues

Related to vdemo - Feature #2072: Ignore non-reachable Hosts during Startup Closed 11/03/2014

History

#1 Updated by S. Wrede almost 6 years ago

  • Description updated (diff)

#2 Updated by S. Wrede almost 6 years ago

  • Related to Feature #2072: Ignore non-reachable Hosts during Startup added

#3 Updated by R. Haschke almost 6 years ago

There is already a mechanism to detect connection loss and reestablish connections: handle_screen_failure()
However, to work, inotify-tools need to be installed. It notices that the screen session attached to a component or host is gone, because the correpsonding screen file vanishes.

Does that suffice?

#4 Updated by S. Wrede almost 6 years ago

That is also good news. We'll test that in our setup and close this issue afterwards.

Again, my question would be if this is already documented somewhere... ;-)

#5 Updated by R. Haschke almost 6 years ago

Not yet documented.
With inotifywait installed on the startup machine, you can monitor and reestablish master connections.
With inotifywait installed on the remote machine, you can monitor individual components. Currently, their status only turns red automatically. They are not respawned.

#6 Updated by R. Haschke about 5 years ago

  • Status changed from New to Resolved

Both respawning master connections and components works.

Also available in: Atom PDF