Fork me on GitHub
Teleport

Teleport Admin Manual

This manual covers the installation and configuration of Teleport and the ongoing management of a Teleport cluster. It assumes that the reader has a good understanding of Linux administration.

Definitions

Before diving into configuring and running Teleport, it helps to take a look at the Teleport Architecture and review the key concepts this document will be referring to:

ConceptDescription
NodeSynonym to "server" or "computer", something one can "SSH to". A node must be running the teleport daemon with "node" role/service turned on.
Certificate Authority (CA)A pair of public/private keys Teleport uses to manage access. A CA can sign a public key of a user or node, establishing their cluster membership.
Teleport ClusterA Teleport Auth Service contains two CAs. One is used to sign user keys and the other signs node keys. A collection of nodes connected to the same CA is called a "cluster".
Cluster NameEvery Teleport cluster must have a name. If a name is not supplied via teleport.yaml configuration file, a GUID will be generated. IMPORTANT: renaming a cluster invalidates its keys and all certificates it had created.
Trusted ClusterTeleport Auth Service can allow 3rd party users or nodes to connect if their public keys are signed by a trusted CA. A trusted cluster is a pair of public keys of the trusted CA. It can be configured via teleport.yaml file.

Ports

Teleport services listen on several ports. This table shows the default port numbers.

PortServiceDescription
3022NodeSSH port. This is Teleport's equivalent of port #22 for SSH.
3023ProxySSH port clients connect to. A proxy will forward this connection to port #3022 on the destination node.
3024ProxySSH port used to create "reverse SSH tunnels" from behind-firewall environments into a trusted proxy server.
3025AuthSSH port used by the Auth Service to serve its API to other nodes in a cluster.
3080ProxyHTTPS connection to authenticate tsh users and web users into the cluster. The same connection is used to serve a Web UI.
3026KubernetesHTTPS Kubernetes proxy proxy_service.kube_listen_addr
3027KubernetesKubernetes Service kubernetes_service.listen_addr

Filesystem layout

By default, a Teleport node has the following files present. The location of all of them is configurable.

Full pathPurpose
/etc/teleport.yamlTeleport configuration file (optional).
/usr/local/bin/teleportTeleport daemon binary.
/usr/local/bin/tctlTeleport admin tool. It is only needed for auth servers.
/var/lib/teleportTeleport data directory. Nodes keep their keys and certificates there. Auth servers store the audit log and the cluster keys there, but the audit log storage can be further configured via auth_service section in the config file.

Configuration

You should use a configuration file to configure the teleport daemon. For simple experimentation, you can use command line flags with the teleport start command. Read about all the allowed flags in the CLI Docs or run teleport start --help

Configuration File

Teleport uses the YAML file format for configuration. A sample configuration file is shown below. By default, it is stored in /etc/teleport.yaml, below is an expanded and commented version from teleport configure.

The default path Teleport uses to look for a config file is /etc/teleport.yaml. You can override this path and set it explicitly using the -c or --config flag to teleport start:

sudo teleport start --config=/etc/teleport.yaml

For a complete reference, see our Configuration Reference - teleport.yaml

IMPORTANT
When editing YAML configuration, please pay attention to how your editor handles white space. YAML requires consistent handling of tab characters.
#
# Sample Teleport configuration file
# Creates a single proxy, auth, and node server.
#
# Things to update:
#  1. ca_pin: Obtain the CA pin hash for joining more nodes by running 'tctl status'
#     on the auth server once Teleport is running.
#  2. license-if-using-teleport-enterprise.pem: If you are an Enterprise customer,
#     obtain this from https://dashboard.gravitational.com/web/login
#
teleport:
  # nodename allows to assign an alternative name this node can be reached by.
  # by default it's equal to hostname
  nodename: NODE_NAME
  data_dir: /var/lib/teleport

  # Invitation token used to join a cluster. it is not used on
  # subsequent starts
  auth_token: xxxx-token-xxxx

  # Optional CA pin of the auth server. This enables a more secure way of adding new
  # nodes to a cluster. See "Adding Nodes to the Cluster"
  # (https://goteleport.com/teleport/docs/admin-guide/#adding-nodes-to-the-cluster).
  ca_pin: "sha256:ca-pin-hash-goes-here"

  # Auth Server address and port to connect to. If you configure teleport auth to run in High
  # Availability configuration the address should point to a Load Balancer.
  # If adding a node located behind NAT, use the Proxy URL. e.g.
  #  auth_servers:
  #     - teleport-proxy.example.com:443
  auth_servers:
      - 10.1.0.5:3025


  # Logging configuration. Possible output values to disk via '/var/lib/teleport/teleport.log',
  # 'stdout', 'stderr' and 'syslog'. Possible severity values are INFO, WARN
  # and ERROR (default).
  log:
    output: stderr
    severity: INFO

auth_service:
  enabled: "yes"
  # A cluster name is used as part of a signature in certificates
  # generated by this CA.
  #
  # We strongly recommend explicitly set it to something meaningful as it
  # becomes important when configuring trust between multiple clusters.
  #
  # By default an automatically generated name is used (not recommended)
  #
  # IMPORTANT: if you change cluster_name, it will invalidate all generated
  # certificates and keys (may need to wipe out /var/lib/teleport directory)
  cluster_name: "teleport-aws-us-east-1"

  # IP and the port to bind to. Other Teleport nodes will be connecting to
  # this port (AKA "Auth API" or "Cluster API") to validate client
  # certificates
  listen_addr: 0.0.0.0:3025

  tokens:
  - proxy,node:xxxx-token-xxxx
  # license_file: /path/to/license-if-using-teleport-enterprise.pem

  authentication:
    # Default authentication type. possible values are 'local' and 'github' for OSS
    # and 'oidc', 'saml' and 'false' for Enterprise.
    type: local
    # second_factor can be off, on, optional, otp, or u2f
    second_factor: otp

    # If running the auth service behind a load balancer, it is highly recommended to have the keepalive settings
    # on your Teleport cluster match the value set on your load balancers.
    # The default keep alive interval is 300 seconds.
    keep_alive_interval: 1m
    keep_alive_count_max: 3

ssh_service:
  enabled: "yes"
  labels:
    teleport: static-label-example
  commands:
  - name: hostname
    command: [/usr/bin/hostname]
    period: 1m0s
  - name: arch
    command: [/usr/bin/uname, -p]
    period: 1h0m0s
proxy_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3023
  web_listen_addr: 0.0.0.0:3080
  tunnel_listen_addr: 0.0.0.0:3024

  # Expose a k8s listening port on the proxy if using Kubernetes
  kube_listen_addr: 0.0.0.0:3026

  # The DNS name of the proxy HTTPS endpoint as accessible by cluster users.
  # Defaults to the proxy's hostname if not specified. If running multiple
  # proxies behind a load balancer, this name must point to the load balancer
  # See the "Public Addr" section for more details
  # (https://goteleport.com/teleport/docs/admin-guide/#public-addr).
  public_addr: TELEPORT_PUBLIC_DNS_NAME:3080

  # TLS certificate for the HTTPS connection. Configuring these properly is
  # critical for Teleport security.
  https_keypairs:
    - key_file: /var/lib/teleport/webproxy_key.pem
      cert_file: /var/lib/teleport/webproxy_cert.pem

Public address

Notice that all three Teleport services (proxy, auth, node) have an optional public_addr property. The public address can take an IP or a DNS name. It can also be a list of values:

public_addr: ["proxy-one.example.com", "proxy-two.example.com"]

Specifying a public address for a Teleport service may be useful in the following use cases:

  • You have multiple identical services, like proxies, behind a load balancer.
  • You want Teleport to issue an SSH certificate for the service with the additional principals, e.g.host names.

Authentication

Teleport uses the concept of "authentication connectors" to authenticate users when they execute tsh login command. There are three types of authentication connectors:

Local connector

Local authentication is used to authenticate against a local Teleport user database. This database is managed by tctl users command. Teleport also supports second-factor authentication (2FA) for the local connector. There are several possible values (types) of 2FA:

  • otp is the default. It implements TOTP standard. You can use Google Authenticator or Authy or any other TOTP client.
  • u2f implements the U2F standard for utilizing hardware (USB) keys for the second factor. You can use YubiKeys, SoloKeys or any other hardware token which implements the FIDO U2F standard.
  • on enables both TOTP and U2F, and all local users are required to have at least one 2FA device registered.
  • optional enables both TOTP and U2F but makes it optional for users. Local users that register a 2FA device will be prompted for it during login. This option is useful when you need to gradually enable 2FA usage before switching the value to on.
  • off turns off second-factor authentication.

Here is an example of this setting in the teleport.yaml :

auth_service:
  authentication:
    type: local
    second_factor: off
Note
SSO users can also register 2FA devices, but Teleport will not prompt them for 2FA during login. Login 2FA for SSO users should be handled by the SSO provider.

GitHub OAuth 2.0 connector

This connector implements Github OAuth 2.0 authentication flow. Please refer to GitHub documentation on Creating an OAuth App to learn how to create and register an OAuth app.

Here is an example of this setting in the teleport.yaml :

auth_service:
  authentication:
    type: github

See Github OAuth 2.0 for details on how to configure it.

SAML

This connector type implements SAML authentication. It can be configured against any external identity manager like Okta or Auth0. This feature is only available for Teleport Enterprise.

Here is an example of this setting in the teleport.yaml :

auth_service:
  authentication:
    type: saml

OIDC

Teleport implements OpenID Connect (OIDC) authentication, which is similar to SAML in principle. This feature is only available for Teleport Enterprise.

Here is an example of this setting in the teleport.yaml :

auth_service:
  authentication:
    type: oidc

Audit Log

Teleport logs every SSH event into its audit log. There are two components of the audit log:

  1. SSH Events: Teleport logs events like successful user logins along with the metadata like remote IP address, time, and the session ID.
  2. Recorded Sessions: Every SSH shell session is recorded and can be replayed later. The recording is done by the nodes themselves, by default, but can be configured to be done by the proxy.
  3. Optional: Enhanced Session Recording

Refer to the "Audit Log" chapter in the Teleport Architecture to learn more about how the Audit Log and Session Recording are designed.

Events

Teleport supports multiple storage back-ends for storing the SSH, Application, and Kubernetes events. The section below uses the dir backend as an example. dir backend uses the local filesystem of an auth server using the configurable data_dir directory.

For High Availability configurations, users can refer to our DynamoDB or Firestore chapters for information on how to configure the SSH events and recorded sessions to be stored on network storage. It is even possible to store the audit log in multiple places at the same time - see audit_events_uri setting in the sample configuration file above for how to do that.

Let's examine the Teleport audit log using the dir backend. The event log is stored in data_dir under log directory, usually /var/lib/teleport/log . Each day is represented as a file:

ls -l /var/lib/teleport/log/

total 104

-rw-r----- 1 root root 31638 Jan 22 20:00 2017-01-23.00:00:00.log

-rw-r----- 1 root root 91256 Jan 31 21:00 2017-02-01.00:00:00.log

-rw-r----- 1 root root 15815 Feb 32 22:54 2017-02-03.00:00:00.log

The log files use JSON format. They are human-readable but can also be programmatically parsed. Each line represents an event and has the following format:

{
    // Event type. See below for the list of all possible event types
    "event": "session.start",
    // uid: A unique ID for the event log. Useful for  deduplication.
    "uid": "59cf8d1b-7b36-4894-8e90-9d9713b6b9ef",
    // Teleport user name
    "user": "ekontsevoy",
    // OS login
    "login": "root",
    // Server namespace. This field is reserved for future use.
    "namespace": "default",
    // Unique server ID.
    "server_id": "f84f7386-5e22-45ff-8f7d-b8079742e63f",
    // Server Labels.
    "server_labels": {
      "datacenter": "us-east-1",
      "label-b": "x"
    }
    // Session ID. Can be used to replay the session.
    "sid": "8d3895b6-e9dd-11e6-94de-40167e68e931",
    // Address of the SSH node
    "addr.local": "10.5.l.15:3022",
    // Address of the connecting client (user)
    "addr.remote": "73.223.221.14:42146",
    // Terminal size
    "size": "80:25",
    // Timestamp
    "time": "2017-02-03T06:54:05Z"
}

The possible event types are:

Event TypeDescription
authAuthentication attempt. Adds the following fields: {"success": "false", "error": "access denied"}
session.startStarted an interactive shell session.
session.endAn interactive shell session has ended.
session.joinA new user has joined the existing interactive shell session.
session.leaveA user has left the session.
session.diskA list of files opened during the session. Requires Enhanced Session Recording.
session.networkA list of network connections made during the session. Requires Enhanced Session Recording.
session.commandA list of commands ran during the session. Requires Enhanced Session Recording.
execRemote command has been executed via SSH, like tsh ssh [email protected] ls /. The following fields will be logged: {"command": "ls /", "exitCode": 0, "exitError": ""}
scpRemote file copy has been executed. The following fields will be logged: {"path": "/path/to/file.txt", "len": 32344, "action": "read" }
resizeTerminal has been resized.
user.loginA user logged into web UI or via tsh. The following fields will be logged: {"user": "[email protected]", "method": "local"} .
app.session.startA user accessed an application
app.session.chunkA record of activity during an app session

Recorded sessions

In addition to logging session.start and session.end events, Teleport also records the entire stream of bytes going to/from standard input and standard the output of an SSH session.

Teleport can store the recorded sessions in an AWS S3 bucket or in a local filesystem (including NFS).

The recorded sessions are stored as raw bytes in the sessions directory under log. Each session consists of two files, both are named after the session ID:

  1. .bytes file or .chunks.gz compressed format represents the raw session bytes and is somewhat human-readable, although you are better off using tsh play or the Web UI to replay it.
  2. .log file or .events.gz compressed file contains the copies of the event log entries that are related to this session.
ls /var/lib/teleport/log/sessions/default

-rw-r----- 1 root root 506192 Feb 4 00:46 4c146ec8-eab6-11e6-b1b3-40167e68e931.session.bytes

-rw-r----- 1 root root 44943 Feb 4 00:46 4c146ec8-eab6-11e6-b1b3-40167e68e931.session.log

To replay this session via CLI:

tsh --proxy=proxy play 4c146ec8-eab6-11e6-b1b3-40167e68e931

HTTP CONNECT proxies

Some networks funnel all connections through a proxy server where they can be audited and access control rules are applied. For these scenarios, Teleport supports HTTP CONNECT tunneling.

To use HTTP CONNECT tunneling, simply set either the HTTPS_PROXY or HTTP_PROXY environment variables and when Teleport builds and establishes the reverse tunnel to the main cluster, it will funnel all traffic through the proxy. Specifically, if using the default configuration, Teleport will tunnel ports 3024 (SSH, reverse tunnel) and 3080 (HTTPS, establishing trust) through the proxy.

The value of HTTPS_PROXY or HTTP_PROXY should be in the format scheme://host:port where scheme is either https or http . If the value is host:port , Teleport will prepend http .

It's important to note that for Teleport to use HTTP CONNECT tunneling, the HTTP_PROXY and HTTPS_PROXY environment variables must be set within Teleport's environment. You can also optionally set the NO_PROXY environment variable to avoid use of the proxy when accessing specified hosts/netmasks. When launching Teleport with systemd, this will probably involve adding some lines to your systemd unit file:

[Service]
Environment="HTTP_PROXY=http://proxy.example.com:8080/"
Environment="HTTPS_PROXY=http://proxy.example.com:8080/"
Environment="NO_PROXY=localhost,127.0.0.1,192.168.0.0/16,172.16.0.0/12,10.0.0.0/8"
Note
localhost and 127.0.0.1 are invalid values for the proxy host. If for some reason your proxy runs locally, you'll need to provide some other DNS name or a private IP address for it.

Certificate rotation

Take a look at the Certificates chapter in the architecture document to learn how the certificate rotation works. This section will show you how to implement certificate rotation in practice.

The easiest way to start the rotation is to execute this command on a cluster's auth server:

tctl auth rotate

This will trigger a rotation process for both hosts and users with a grace period of 48 hours.

This can be customized, i.e.

Rotate only user certificates with a grace period of 200 hours:

tctl auth rotate --type=user --grace-period=200h

Rotate only host certificates with a grace period of 8 hours:

tctl auth rotate --type=host --grace-period=8h

The rotation takes time, especially for hosts, because each node in a cluster needs to be notified that a rotation is taking place and request a new certificate for itself before the grace period ends.

Warning
Be careful when choosing a grace period when rotating host certificates. The grace period needs to be long enough for all nodes in a cluster to request a new certificate. If some nodes go offline during the rotation and come back only after the grace period has ended, they will be forced to leave the cluster, i.e. users will no longer be allowed to SSH into them.

To check the status of certificate rotation:

tctl status
CA Pinning Warning
If you are using CA Pinning when adding new nodes, the CA pin will change after the rotation. Make sure you use the new CA pin when adding nodes after rotation.

Storage backends

A Teleport cluster stores different types of data in different locations. By default everything is stored in a local directory at the Auth server. Integration with other storage types is implemented based on the nature of the stored data (size, read/write ratio, mutability, etc.).

Data typeDescriptionSupported storage backends
core cluster stateCluster configuration (e.g. users, roles, auth connectors) and identity (e.g. certificate authorities, registered nodes, trusted clusters).Local directory (SQLite), etcd, AWS DynamoDB, GCP Firestore
audit eventsJSON-encoded events from the audit log (e.g. user logins, RBAC changes)Local directory, AWS DynamoDB, GCP Firestore
session recordingsRaw terminal recordings of interactive user sessionsLocal directory, AWS S3 (and any S3-compatible product), GCP Cloud Storage
teleport instance stateID and credentials of a non-auth teleport instance (e.g. node, proxy)Local directory

The High Availability section describes how to configure non-default storage backends.

High Availability

Tip
Before continuing, please make sure to take a look at the Cluster State section in the Teleport Architecture documentation.

Usually there are two ways to achieve High Availability. You can "outsource" this function to the infrastructure. For example, using a highly available network-based disk volumes (similar to AWS EBS) and by migrating a failed VM to a new host. In this scenario, there's nothing Teleport-specific to be done.

If High Availability cannot be provided by the infrastructure (perhaps you're running Teleport on a bare metal cluster), you can still configure Teleport to run in a highly available fashion.

Auth server High Availability

To run multiple instances of Teleport Auth Server, you must switch to a High Availability secrets back-end first. Also, you must tell each node in a cluster that there is more than one auth server available. There are two ways to do this:

  • Use a load balancer to create a single auth API access point (AP) and specify this AP in auth_servers section of Teleport configuration for all nodes in a cluster. This load balancer should do TCP level forwarding.
  • If a load balancer is not an option, you must specify each instance of an auth server in auth_servers section of Teleport configuration.

IMPORTANT: with multiple instances of the auth servers running, special attention needs to be paid to keeping their configuration identical. Settings like cluster_name , tokens , storage , etc must be the same.

Teleport Proxy High Availability

The Teleport Proxy is stateless which makes running multiple instances trivial. If using the default configuration, configure your load balancer to forward ports 3023 and 3080 to the servers that run the Teleport proxy. If you have configured your proxy to use non-default ports, you will need to configure your load balancer to forward the ports you specified for listen_addr and web_listen_addr in teleport.yaml. The load balancer for web_listen_addr can terminate TLS with your own certificate that is valid for your users, while the remaining ports should do TCP level forwarding, since Teleport will handle its own SSL on top of that with its own certificates.

NOTE
If you terminate TLS with your own certificate at a load balancer you'll need to run Teleport with --insecure-no-tls

If your load balancer supports HTTP health checks, configure it to hit the /readyz diagnostics endpoint on machines running Teleport. This endpoint must be enabled by using the --diag-addr flag to teleport start: teleport start --diag-addr=127.0.0.1:3000 The http://127.0.0.1:3000/readyz endpoint will reply {"status":"ok"} if the Teleport service is running without problems.

NOTE
As the new auth servers get added to the cluster and the old servers get decommissioned, nodes and proxies will refresh the list of available auth servers and store it in their local cache /var/lib/teleport/authservers.json - the values from the cache file will take precedence over the configuration file.

We'll cover how to use etcd, DynamoDB, and Firestore storage back-ends to make Teleport highly available below.

Teleport scalability tweaks

When running Teleport at scale (for example in the case where there are 10,000+ nodes connected to a cluster via node tunneling mode, the following settings should be set on Teleport auth and proxies:

Proxy servers

These settings alter Teleport's default connection limit from 15000 to 65000.

# Teleport Proxy
teleport:
  cache:
    # Use an in-memory cache to speed up the connection of many teleport nodes
    # back to proxy.
    type: in-memory
  # Set up connection limits to prevent throttling of many IoT nodes connecting to proxies
  connection_limits:
    max_connections: 65000
    max_users: 1000

Auth Servers

# Teleport Auth
teleport:
  connection_limits:
    max_connections: 65000
    max_users: 1000

Using etcd

Teleport can use etcd as a storage backend to achieve highly available deployments. You must take steps to protect access to etcd in this configuration because that is where Teleport secrets like keys and user records will be stored.

IMPORTANT
etcd can only currently be used to store Teleport's internal database in a highly-available way. This will allow you to have multiple auth servers in your cluster for an High Availability deployment, but it will not also store Teleport audit events for you in the same way that DynamoDB or Firestore will. etcd is not designed to handle large volumes of time series data like audit events.

To configure Teleport for using etcd as a storage backend:

  • Make sure you are using etcd versions 3.3 or newer.
  • Install etcd and configure peer and client TLS authentication using the etcd security guide.
  • Configure all Teleport Auth servers to use etcd in the "storage" section of the config file as shown below.
  • Deploy several auth servers connected to etcd backend.
  • Deploy several proxy nodes that have auth_servers pointed to the list of auth servers to connect to.
teleport:
  storage:
     type: etcd

     # List of etcd peers to connect to:
     peers: ["https://172.17.0.1:4001", "https://172.17.0.2:4001"]

     # Required path to TLS client certificate and key files to connect to etcd.
     #
     # To create these, follow
     # https://coreos.com/os/docs/latest/generate-self-signed-certificates.html
     # or use the etcd-provided script
     # https://github.com/etcd-io/etcd/tree/master/hack/tls-setup.
     tls_cert_file: /var/lib/teleport/etcd-cert.pem
     tls_key_file: /var/lib/teleport/etcd-key.pem

     # Optional file with trusted CA authority
     # file to authenticate etcd nodes
     #
     # If you used the script above to generate the client TLS certificate,
     # this CA certificate should be one of the other generated files
     tls_ca_file: /var/lib/teleport/etcd-ca.pem

     # Alternative password-based authentication, if not using TLS client
     # certificate.
     #
     # See https://etcd.io/docs/v3.4.0/op-guide/authentication/ for setting
     # up a new user.
     username: username
     password_file: /mnt/secrets/etcd-pass

     # etcd key (location) where teleport will be storing its state under.
     # make sure it ends with a '/'!
     prefix: /teleport/

     # NOT RECOMMENDED: enables insecure etcd mode in which self-signed
     # certificate will be accepted
     insecure: false

     # Optionally sets the limit on the client message size.
     # This is usually used to increase the default which is 2MiB
     # (1.5MiB server's default + gRPC overhead bytes).
     # Make sure this does not exceed the value for the etcd
     # server specified with `--max-request-bytes` (1.5MiB by default).
     # Keep the two values in sync.
     #
     # See https://etcd.io/docs/v3.4.0/dev-guide/limit/ for details
     #
     # This bumps the size to 15MiB as an example:
     etcd_max_client_msg_size_bytes: 15728640

Using Amazon S3

Tip
Before continuing, please make sure to take a look at the Cluster State section in Teleport Architecture documentation.
AWS Authentication
The configuration examples below contain AWS access keys and secret keys. They are optional, they exist for your convenience but we DO NOT RECOMMEND using them in production. If Teleport is running on an AWS instance it will automatically use the instance IAM role. Teleport also will pick up AWS credentials from the ~/.aws folder, just like the AWS CLI tool.

S3 buckets can only be used as storage for the recorded sessions. S3 cannot store the audit log or the cluster state. Below is an example of how to configure a Teleport auth server to store the recorded sessions in an S3 bucket.

teleport:
  storage:
      # The region setting sets the default AWS region for all AWS services
      # Teleport may consume (DynamoDB, S3)
      region: us-east-1

      # Path to S3 bucket to store the recorded sessions in.
      audit_sessions_uri: "s3://Example_TELEPORT_S3_BUCKET/records"

      # Teleport assumes credentials. Using provider chains, assuming IAM role or
      # standard .aws/credentials in the home folder.

The AWS authentication settings above can be omitted if the machine itself is running on an EC2 instance with an IAM role.

These optional GET parameters control how Teleport interacts with an S3 endpoint, including S3-compatible endpoints.

s3://bucket/path?region=us-east-1&endpoint=mys3.example.com&insecure=false&disablesse=false

  • region=us-east-1 - set the Amazon region to use.
  • endpoint=mys3.example.com - connect to a custom S3 endpoint.
  • insecure=true - set to true or false. If true, TLS will be disabled.
  • disablesse=true - set to true or false. If true, S3 server-side encryption will be disabled. If false, aws:kms (Key Management Service) will be used for server-side encryption. Other SSE types are not supported at this time.

Using DynamoDB

Tip
Before continuing, please make sure to take a look at the Cluster State section in Teleport Architecture documentation.

If you are running Teleport on AWS, you can use DynamoDB as a storage back-end to achieve High Availability. DynamoDB backend supports two types of Teleport data:

  • Cluster state
  • Audit log events

DynamoDB cannot store the recorded sessions. You are advised to use AWS S3 for that as shown above. To configure Teleport to use DynamoDB:

  • Make sure you have AWS access key and a secret key that give you access to DynamoDB account. If you're using (as recommended) an IAM role for this, the policy with the necessary permissions is listed below.
  • Configure all Teleport Auth servers to use DynamoDB back-end in the "storage" section of teleport.yaml as shown below.
  • Deploy several auth servers connected to DynamoDB storage back-end.
  • Deploy several proxy nodes.
  • Make sure that all Teleport nodes have auth_servers configuration setting populated with the auth servers.
teleport:
  storage:
    type: dynamodb
    # Region location of dynamodb instance, https://docs.aws.amazon.com/en_pv/general/latest/gr/rande.html#ddb_region
    region: us-east-1

    # Name of the DynamoDB table. If it does not exist, Teleport will create it.
    table_name: Example_TELEPORT_DYNAMO_TABLE_NAME

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in DynamoDB, a copy on a local filesystem, and also output the events to stdout.
    # NOTE: The DynamoDB events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like DynamoDB, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['dynamodb://events_table_name', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in an S3 bucket:
    audit_sessions_uri: s3://Example_TELEPORT_S3_BUCKET/records
  • Replace us-east-1 and Example_TELEPORT_DYNAMO_TABLE_NAME with your own settings. Teleport will create the table automatically.
  • Example_TELEPORT_DYNAMO_TABLE_NAME and events_table_name must be different DynamoDB tables. The schema is different for each. Using the same table name for both will result in errors.
  • The AWS authentication setting above can be omitted if the machine itself is running on an EC2 instance with an IAM role.
  • Audit log settings above are optional. If specified, Teleport will store the audit log in DynamoDB and the session recordings must be stored in an S3 bucket, i.e. both audit_xxx settings must be present. If they are not set, Teleport will default to a local file system for the audit log, i.e. /var/lib/teleport/log on an auth server.
  • If DynamoDB is used for the audit log, the logged events will be stored with a TTL of 1 year. Currently, this TTL is not configurable.
Access to DynamoDB
Make sure that the IAM role assigned to Teleport is configured with sufficient access to DynamoDB. Below is the example of the IAM policy you can use:
{
    "Version": "2012-10-17",
    "Statement": [{
            "Sid": "AllAPIActionsOnTeleportAuth",
            "Effect": "Allow",
            "Action": "dynamodb:*",
            "Resource": "arn:aws:dynamodb:eu-west-1:123456789012:table/prod.teleport.auth"
        },
        {
            "Sid": "AllAPIActionsOnTeleportStreams",
            "Effect": "Allow",
            "Action": "dynamodb:*",
            "Resource": "arn:aws:dynamodb:eu-west-1:123456789012:table/prod.teleport.auth/stream/*"
        }
    ]
}

DynamoDB autoscaling

When setting up DynamoDB it's important to set up backup and autoscaling. We make setup simpler by allowing AWS DynamoDB settings to be set automatically during Teleport startup.

DynamoDB Continuous Backups

DynamoDB Autoscaling Options

# ...
teleport:
  storage:
    type: "dynamodb"
    [...]

    # continuous_backups is used to optionally enable continuous backups.
    # default: false
    continuous_backups: [true|false]

    # auto_scaling is used to optionally enable (and define settings for) auto scaling.
    # default: false
    auto_scaling:  [true|false]
    # Minimum/maximum read capacity in units
    read_min_capacity: int
    read_max_capacity: int
    read_target_value: float
    # Minimum/maximum write capacity in units
    write_min_capacity: int
    write_max_capacity: int
    write_target_value: float

To enable these options you will need to update the IAM Role for Teleport.

{
    "Action": [
        "application-autoscaling:PutScalingPolicy",
        "application-autoscaling:RegisterScalableTarget"
    ],
    "Effect": "Allow",
    "Resource": "*"
},
{
    "Action": [
        "iam:CreateServiceLinkedRole"
    ],
    "Condition": {
        "StringEquals": {
            "iam:AWSServiceName": [
                "dynamodb.application-autoscaling.amazonaws.com"
            ]
        }
    },
    "Effect": "Allow",
    "Resource": "*"
}

Using GCS

Tip
Before continuing, please make sure to take a look at the Cluster State section in Teleport Architecture documentation.

Google Cloud Storage (GCS) can only be used as storage for the recorded sessions. GCS cannot store the audit log or the cluster state. Below is an example of how to configure a Teleport auth server to store the recorded sessions in a GCS bucket.

teleport:
  storage:
      # Path to GCS to store the recorded sessions in.
      audit_sessions_uri: "gs://Example_TELEPORT_STORAGE/records"
      credentials_path: /var/lib/teleport/gcs_creds

Using Firestore

Tip
Before continuing, please make sure to take a look at the Cluster State section in Teleport Architecture documentation.

If you are running Teleport on GCP, you can use Firestore as a storage back-end to achieve high availability. Firestore backend supports two types of Teleport data:

  • Cluster state
  • Audit log events

Firestore cannot store the recorded sessions. You are advised to use Google Cloud Storage (GCS) for that as shown above. To configure Teleport to use Firestore:

  • Configure all Teleport Auth servers to use Firestore back-end in the "storage" section of teleport.yaml as shown below.
  • Deploy several auth servers connected to Firestore storage back-end.
  • Deploy several proxy nodes.
  • Make sure that all Teleport nodes have auth_servers configuration setting populated with the auth servers or use a load balancer for the auth servers in high availability mode.
teleport:
  storage:
    type: firestore
    # Project ID https://support.google.com/googleapi/answer/7014113?hl=en
    project_id: Example_GCP_Project_Name

    # Name of the Firestore table. If it does not exist, Teleport won't start
    collection_name: Example_TELEPORT_FIRESTORE_TABLE_NAME

    credentials_path: /var/lib/teleport/gcs_creds

    # This setting configures Teleport to send the audit events to three places:
    # To keep a copy in Firestore, a copy on a local filesystem, and also write the events to stdout.
    # NOTE: The Firestore events table has a different schema to the regular Teleport
    # database table, so attempting to use the same table for both will result in errors.
    # When using highly available storage like Firestore, you should make sure that the list always specifies
    # the High Availability storage method first, as this is what the Teleport web UI uses as its source of events to display.
    audit_events_uri:  ['firestore://Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME', 'file:///var/lib/teleport/audit/events', 'stdout://']

    # This setting configures Teleport to save the recorded sessions in GCP storage:
    audit_sessions_uri: gs://Example_TELEPORT_S3_BUCKET/records
  • Replace Example_GCP_Project_Name and Example_TELEPORT_FIRESTORE_TABLE_NAME with your own settings. Teleport will create the table automatically.
  • Example_TELEPORT_FIRESTORE_TABLE_NAME and Example_TELEPORT_FIRESTORE_EVENTS_TABLE_NAME must be different Firestore tables. The schema is different for each. Using the same table name for both will result in errors.
  • The GCP authentication setting above can be omitted if the machine itself is running on a GCE instance with a Service Account that has access to the Firestore table.
  • Audit log settings above are optional. If specified, Teleport will store the audit log in Firestore and the session recordings must be stored in a GCP bucket, i.e.both audit_xxx settings must be present. If they are not set, Teleport will default to a local file system for the audit log, i.e. /var/lib/teleport/log on an auth server.

Production releases

First of all, avoid running pre-releases (release candidates) in production environments. The Teleport development team uses Semantic Versioning which makes it easy to tell if a specific version is recommended for production use.

Component compatibility

When running multiple binaries of Teleport within a cluster (nodes, proxies, clients, etc), the following rules apply:

Before 5.0.0

  • Only patch versions are always compatible, for example, any 4.0.1 component will work with any 4.0.3 component.
  • Minor versions are always compatible with the previous minor release. This means you must not attempt to upgrade from 4.1.x straight to 4.3.x. You must upgrade to 4.2.x first.
  • Teleport clients tsh for users and tctl for admins may not be compatible with different versions of the teleport service.

After 5.0.0

  • Patch and minor versions are always compatible, for example, any 5.0.1 component will work with any 5.0.3 component and 6.1.0 component will work with any 6.7.0 component.
  • Major versions are always compatible with the previous major release. This means you must not attempt to upgrade from 5.x.x straight to 7.x.x. You must upgrade to 6.x.x first.
  • The above applies to both clients and servers. For example, a 6.x.x proxy is compatible with 5.x.x nodes and 5.x.x tsh. But we don't guarantee that a 7.x.x tsh will work with a 5.x.x proxy.

As an extra precaution, you might want to backup your application before upgrading. We provide more instructions in Backup before upgrading.

Upgrading to Teleport 4.0+

Teleport 4.0+ switched to GRPC and HTTP/2 as an API protocol. The HTTP/2 spec bans two previously recommended ciphers. tls-rsa-with-aes-128-gcm-sha256 & tls-rsa-with-aes-256-gcm-sha384, make sure these are removed from teleport.yaml Visit our community for more details

If upgrading you might want to consider rotating CA to SHA-256 or SHA-512 for RSA SSH certificate signatures. The previous default was SHA-1, which is now considered weak against brute-force attacks. SHA-1 certificate signatures are also no longer accepted by OpenSSH versions 8.2 and above. All new Teleport clusters will default to SHA-512 based signatures. To upgrade an existing cluster, set the following in your teleport.yaml:

teleport:
  ca_signature_algo: "rsa-sha2-512"

After updating to 4.3+ rotate the cluster CA following these docs.

Backup Before Upgrading

As an extra precaution, you might want to backup your application before upgrading. We have more instructions in Backing up Teleport.

Upgrade Sequence

When upgrading a single Teleport cluster:

  1. Upgrade the auth server first. The auth server keeps the cluster state and if there are data format changes introduced in the new version this will perform necessary migrations.
  2. Then, upgrade the proxy servers. The proxy servers are stateless and can be upgraded in any sequence or at the same time.
  3. Finally, upgrade the SSH nodes in any sequence or at the same time.
Warning
If several auth servers are running in High Availability configuration (for example, in AWS auto-scaling group) you have to shrink the group to just one auth server before performing an upgrade. While Teleport will attempt to perform any necessary migrations, we recommend users create a backup of their backend before upgrading the Auth Server, as a precaution. This allows for a safe rollback in case the migration itself fails.

When upgrading multiple clusters:

  1. First, upgrade the main cluster, i.e. the one which other clusters trust.
  2. Upgrade the trusted clusters.

Backing up Teleport

When planning a backup of Teleport, it's important to know what is where and the importance of each component. Teleport's Proxies and Nodes are stateless, and thus only teleport.yaml should be backed up.

The Auth server is Teleport's brains, and depending on the backend should be backed up regularly.

For example, a customer running Teleport on AWS with DynamoDB have these key items of data:

WhatWhere ( Example AWS Customer )
Local Users ( not SSO )DynamoDB
Certificate AuthoritiesDynamoDB
Trusted ClustersDynamoDB
Connectors: SSODynamoDB / File System
RBACDynamoDB / File System
teleport.yamlFile System
teleport.serviceFile System
license.pemFile System
TLS key/certificate( File System / Outside Scope )
Audit logDynamoDB
Session recordingsS3

For this customer, we would recommend using AWS best practices for backing up DynamoDB. If DynamoDB is used for the audit log, logged events have a TTL of 1 year.

BackendRecommended backup strategy
dir ( local filesystem )Backup /var/lib/teleport/storage directory and the output of tctl get all --with-secrets.
DynamoDBFollow AWS Guidelines for Backup & Restore
etcdFollow etcD Guidleines for Disaster Recovery
FirestoreFollow GCP Guidlines for Automated Backups

Teleport resources

Teleport uses YAML resources for roles, trusted clusters, local users, and auth connectors. These could be created via tctl or the UI.

GitOps

If you're running Teleport at scale, your teams need to have an automated way to restore Teleport. At a high level, this is our recommended approach:

  • Persist and backup your backend
  • Share that backend among auth servers
  • Store your configs as discrete files in VCS
  • Have your CI run tctl create -f *.yaml from that git directory

Migrating backends

As of version v4.1, you can now quickly export a collection of resources from Teleport. This feature was designed to help customers migrate from local storage to etcd.

Using tctl get all --with-secrets will retrieve the below items:

  • Users
  • Certificate Authorities
  • Trusted Clusters
  • Connectors:
    • Github
    • SAML [Teleport Enterprise]
    • OIDC [Teleport Enterprise]
    • Roles [Teleport Enterprise]

When migrating backends, you should back up your auth server's data_dir/storage directly.

Example of backing up and restoring a cluster.

Export dynamic configuration state from old cluster

tctl get all --with-secrets > state.yaml

Prepare a new uninitialized backend (make sure to port

any non-default config values from the old config file)

mkdir fresh && cat > fresh.yaml << EOFteleport: data_dir: freshEOF

bootstrap fresh server (kill the old one first!)

sudo teleport start --config fresh.yaml --bootstrap state.yaml

from another terminal, verify state transferred correctly

tctl --config fresh.yaml get all

<your state here>

The --bootstrap flag has no effect, except during backend initialization (performed by auth server on first start), so it is safe for use in supervised/High Availability contexts.

Limitations

  • The --bootstrap flag doesn't re-trigger trusted cluster handshakes, so trusted cluster resources need to be recreated manually.
  • All the same limitations around modifying the config file of an existing cluster also apply to a new cluster being bootstrapped from the state of an old cluster. Of particular note:
    • Changing cluster name will break your CAs (this will be caught and teleport will refuse to start).
    • Some user authentication mechanisms (e.g. u2f) require that the public endpoint of the web ui remains the same (this can't be caught by teleport, be careful!).
  • Any node whose invite token is defined statically (in the config file of the auth server) will be able to join automatically, but nodes that were added dynamically will need to be re-invited

License file

Commercial Teleport subscriptions require a valid license. The license file can be downloaded from the Teleport Customer Portal.

The Teleport license file contains a X.509 certificate and the corresponding private key in PEM format. Place the downloaded file on Auth servers and set the license_file configuration parameter of your teleport.yaml to point to the file location:

auth_service:
    license_file: /var/lib/teleport/license.pem

The license_file path can be either absolute or relative to the configured data_dir. If the license file path is not set, Teleport will look for the license.pem file in the configured data_dir.

NOTE
Only Auth servers require the license. Proxies and Nodes that do not also have Auth role enabled do not need the license.
Have a suggestion or can’t find something?
IMPROVE THE DOCS