You can send signals to a Teleport daemon to get diagnostic or gracefully shut it down:
|Signal||Teleport Daemon Behavior|
|Dumps diagnostics/debugging information into syslog.|
|Graceful shutdown. The daemon will wait until connections are dropped.|
|Immediate non-graceful shutdown. All existing connections will be dropped.|
|Forks a new Teleport daemon to serve new connections.|
|Forks a new Teleport daemon to serve new connections and initiates the graceful shutdown of the existing process when there are no more clients connected to it.|
In this guide we will try a manual graceful upgrade of a binary and a rollback using signals.
Locate a running teleport daemon PID:
Locate teleport process PIDpidof teleport
Unpack a new binary and replace a binary without stopping a server. Preserve the old binary, just in case if the upgrade goes wrong.
mv /usr/bin/teleport /usr/bin/teleport.bakcp /new/binary/teleport /usr/bin/teleport
Fork a new teleport process:
kill -USR2 $(pidof teleport)
The original teleport process forked a new child process and passed existing file descriptors to the child. You now have two processes handling requests on the same socket:
In our example,
235276 is a PID of the child process, and
235119 is a PID of the parent.
In the logs you will see that the parent process reports that it has forked a new child process, and the child accepts file descriptors from it's parent.
2021-08-19T10:16:51-07:00 [PROC:1] INFO Forked new child process. path:/usr/local/teleport service/signals.go:457 2021-08-19T10:16:51-07:00 [PROC:1] INFO Using file descriptor diag 127.0.0.1:3434 passed by the parent process. service/signals.go:207
Examine the logs and use the system. You have two options:
If the new binary behaves with errors, shut down the child process:
kill -TERM 235276
Do not forget to restore the original binary
mv /usr/bin/teleport.bak /usr/bin/teleport
You can retry the process again later.
If you are upgrading the Teleport using SSH connection using Teleport,
make sure to re-establish a new connection to the new teleport instance and launch shut down from it.
You can always see which node process handles the connection by using
pstree -aps $$
└─pstree,242371 -aps 190718
Shut down the parent process gracefully using
kill -QUIT 235119
The parent process will log graceful shutdown:
2021-08-19T10:32:10-07:00 INFO [PROXY:SER] Shutting down gracefully. service/service.go:2952
In a couple of minutes all existing connections drain off and the parent will exit:
If for some reason, the parent process gets stuck, for example waiting on existing connections to finish, you can shut it down non-gracefully:
kill -TERM 235119
You are all set.