Survive connection loss
Currently, VDemo is very sensitive to connection loss. If a connection to a node within a setup is down and comes up again, VDemo needs to be restarted completely.
Given more complex setups (such as the one we currently use with about 20 nodes) and long-running processes, this is really annoying. Hence, it would be really great if VDemo could also monitor node status and re-initiate ssh connections / and screen sessions at runtime.
#3 Updated by R. Haschke almost 6 years ago
There is already a mechanism to detect connection loss and reestablish connections: handle_screen_failure()
However, to work, inotify-tools need to be installed. It notices that the screen session attached to a component or host is gone, because the correpsonding screen file vanishes.
Does that suffice?
#5 Updated by R. Haschke almost 6 years ago
Not yet documented.
With inotifywait installed on the startup machine, you can monitor and reestablish master connections.
With inotifywait installed on the remote machine, you can monitor individual components. Currently, their status only turns red automatically. They are not respawned.