Lesson 16 intermediate

SSH Keys and Secure Remote Access

How SSH proves who the server is and who you are, how to create and protect ed25519 keys, use ssh-agent and ~/.ssh/config with a bastion, copy files, harden sshd without locking yourself out and give CI a restricted deploy key.

SSH is the front door of every Linux server you will manage. You log in with it, Ansible and CI pipelines deploy through it, git push often rides on it, and rsync and scp copy files over it. Because it is the front door, it is also what attackers knock on all day: an internet-facing server sees password guesses within minutes of booting.

In this lesson you learn what actually happens when you type ssh, create a key pair the right way, install it on a server, make daily access comfortable with ssh-agent and ~/.ssh/config, and then lock the door: no passwords, no root logins, only the right group. You also give Orbit’s CI pipeline a key that can deploy and do nothing else. You need the orbit-lab VM from lesson 1, reached from your own computer over the network.

How an SSH connection works

Every connection goes through two checks, in this order:

  1. The server proves who it is. It signs the key exchange with its host key (/etc/ssh/ssh_host_ed25519_key). Your client compares the public half with the copy saved in ~/.ssh/known_hosts.
  2. Both sides agree on encryption keys, so everything after this point is private.
  3. You prove who you are. The client signs a challenge with your private key; the server checks the signature against the public keys in your ~/.ssh/authorized_keys. The private key never leaves your machine: think of the public key as a padlock you can hand out freely and the private key as the only key that opens it.
FileLives onContainsSecret?
/etc/ssh/ssh_host_*_keyServerThe server’s private host keysYes, root only
~/.ssh/known_hostsClientHost keys of servers you trustNo, but its integrity matters
~/.ssh/id_ed25519ClientYour private keyYes, never share it
~/.ssh/id_ed25519.pubClient, copied to serversYour public keyNo
~/.ssh/authorized_keysServer, in your homePublic keys allowed to log in as youNo, but only you may write it

The first-connection prompt

The first time you connect, your client has no saved host key and asks:

$ ssh leo@orbit-web-01
The authenticity of host 'orbit-web-01 (203.0.113.21)' can't be established.
ED25519 key fingerprint is SHA256:6vY0kq1mX0QhH3d9tWc8bqE2fJx7Rz4uLp5nK1sT9aA.
This key is not known by any other names.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

Typing yes without checking is exactly what a man-in-the-middle attack needs. Compare the fingerprint with one obtained through a different channel: the cloud console’s boot log, your provisioning tool’s output, or a colleague who runs this on the server:

$ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:6vY0kq1mX0QhH3d9tWc8bqE2fJx7Rz4uLp5nK1sT9aA root@orbit-web-01 (ED25519)

-l prints the fingerprint and -f names the key file. You can also paste the expected fingerprint at the prompt instead of yes. Once saved, the key is checked on every connection, and a mismatch stops you with “REMOTE HOST IDENTIFICATION HAS CHANGED”. Find out why (a rebuilt server? a new IP?) before removing the old entry with ssh-keygen -R orbit-web-01.

Create a key pair

$ ssh-keygen -t ed25519 -C "leo@laptop 2026-11" -f ~/.ssh/orbit_ed25519
Generating public/private ed25519 key pair.
Enter passphrase for "/home/leo/.ssh/orbit_ed25519" (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/leo/.ssh/orbit_ed25519
Your public key has been saved in /home/leo/.ssh/orbit_ed25519.pub
The key fingerprint is:
SHA256:Qm3t8bV1cZ0yX4wR7nE2kD9sH5fJ6aL1pU8oI3gT2xY leo@laptop 2026-11
  • -t ed25519 picks the algorithm: short keys, fast, and the right default in 2026.
  • -C sets a comment. Put the person, the device and the date in it; future you will know which line in authorized_keys to delete.
  • -f chooses the file name. Without it you get ~/.ssh/id_ed25519, which ssh tries automatically.
  • Use a passphrase. It encrypts the private key on disk, so a stolen laptop backup does not mean a stolen server. ssh-agent (below) means you type it once per session, not per login. ssh refuses a private key that others can read, so keep it at 600; the .pub file can be 644.
Key typeUse it?
ed25519Yes, the default for people and automation
ed25519-sk, ecdsa-skYes, with a FIDO2 hardware key (see below)
rsa (3072 bits or more)Only for old systems that do not support ed25519
ecdsaWorks, but there is no reason to choose it over ed25519
dsaNo: removed completely from OpenSSH 10

Install the public key on a server

If you can still log in with a password, ssh-copy-id appends your public key to the remote authorized_keys and fixes the permissions:

$ ssh-copy-id -i ~/.ssh/orbit_ed25519.pub leo@orbit-web-01
Number of key(s) added: 1
$ ssh -i ~/.ssh/orbit_ed25519 leo@orbit-web-01

Without ssh-copy-id (it is missing on Windows) or without password login, add the line yourself through another session, a console or your provisioning tool, and get the permissions right:

$ install -d -m 700 ~/.ssh
$ cat orbit_ed25519.pub >> ~/.ssh/authorized_keys
$ chmod 600 ~/.ssh/authorized_keys
$ ls -ld ~ ~/.ssh ~/.ssh/authorized_keys
drwxr-x--- 5 leo leo 4096 Nov  3 10:02 /home/leo
drwx------ 2 leo leo 4096 Nov  3 10:02 /home/leo/.ssh
-rw------- 1 leo leo   99 Nov  3 10:02 /home/leo/.ssh/authorized_keys

install -d -m 700 creates the directory with the right mode in one step. The server’s StrictModes check refuses keys if your home, ~/.ssh or authorized_keys is writable by anyone else, and the only symptom on the client is “Permission denied (publickey)”. The real reason is in the server’s log: journalctl -u ssh shows “Authentication refused: bad ownership or modes”. When you create the files as root for another user, also chown -R leo:leo /home/leo/.ssh.

Reach the lab VM over SSH

multipass shell uses its own key, so to practise real SSH, find the VM’s address and install your key once:

$ multipass info orbit-lab
Name:           orbit-lab
State:          Running
IPv4:           192.168.64.5
Release:        Ubuntu 26.04 LTS
$ multipass transfer ~/.ssh/orbit_ed25519.pub orbit-lab:/home/ubuntu/
$ multipass exec orbit-lab -- bash -c 'cat ~/orbit_ed25519.pub >> ~/.ssh/authorized_keys'
$ multipass exec orbit-lab -- ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 SHA256:Hk2v8M1nQ0rX5tY7uW3zA9bC4dE6fG8hJ1kL2mN3oP4 root@orbit-lab (ED25519)
$ ssh -i ~/.ssh/orbit_ed25519 ubuntu@192.168.64.5

Your IP will differ (on Windows with Hyper-V it is often 172.x.x.x). The ssh-keygen -lf line is your trusted channel: compare it with the fingerprint in the first-connection prompt. The Windows 11 OpenSSH client works the same way in PowerShell, with keys in C:\Users\<you>\.ssh.

ssh-agent: type the passphrase once

ssh-agent holds decrypted keys in memory and signs on your behalf. Most desktops start one for you; on a server or in a bare terminal start it yourself:

$ eval "$(ssh-agent -s)"
Agent pid 2817
$ ssh-add -t 8h ~/.ssh/orbit_ed25519
Enter passphrase for /home/leo/.ssh/orbit_ed25519:
Identity added: /home/leo/.ssh/orbit_ed25519 (leo@laptop 2026-11)
Lifetime set to 28800 seconds
$ ssh-add -l
256 SHA256:Qm3t8bV1cZ0yX4wR7nE2kD9sH5fJ6aL1pU8oI3gT2xY leo@laptop 2026-11 (ED25519)

-s prints shell commands that eval turns into the SSH_AUTH_SOCK variable (lesson 14), -t 8h forgets the key after a working day, ssh-add -l lists loaded keys and ssh-add -D removes them all. On Windows, enable the “OpenSSH Authentication Agent” service once; on macOS the agent integrates with the Keychain.

Warning: Agent forwarding (ssh -A, ForwardAgent yes) lets the remote server use your agent while you are connected, and so does anyone with root on that server. To hop through a bastion, use ProxyJump instead: the inner connection is end-to-end from your laptop and your keys never touch the bastion.

~/.ssh/config: aliases instead of long commands

Typing ssh -i ~/.ssh/orbit_ed25519 -J leo@bastion.orbit.example.com leo@10.0.1.21 gets old fast. Describe hosts once in ~/.ssh/config (mode 600):

Host bastion
    HostName bastion.orbit.example.com
    User leo

Host orbit-web-01
    HostName 10.0.1.21
    ProxyJump bastion

Host orbit-lab
    HostName 192.168.64.5
    User ubuntu

Host *
    User leo
    IdentityFile ~/.ssh/orbit_ed25519
    IdentitiesOnly yes
    AddKeysToAgent yes
    ServerAliveInterval 60

Now ssh orbit-web-01 connects to the bastion, then on to the private address. For each option the first value found wins, so put specific hosts first and Host * defaults last. IdentitiesOnly yes stops ssh from offering every key in the agent (servers count each one as a failed attempt), AddKeysToAgent yes loads the key on first use and ServerAliveInterval 60 keeps idle sessions through NAT and firewalls. Every tool that uses ssh, including git, scp, rsync and Ansible, reads this file too. ssh -G orbit-web-01 prints the final settings for a host.

Copying files: scp and rsync

scprsync over SSH
Good forOne file, quicklyDirectories, repeated syncs, large transfers
Re-runCopies everything againSends only what changed
Resume and deleteNo--partial, --delete
Examplescp backup.tar.gz orbit-web-01:/tmp/rsync -avz --delete public/ orbit-web-01:/srv/orbit/public/

In the rsync example, -a keeps permissions, times and links, -v lists files, -z compresses, and --delete removes files on the target that no longer exist locally. The trailing slash on public/ means “the contents of”. Add -n (dry run) the first time you use --delete. Since OpenSSH 9.0, scp uses the SFTP protocol underneath, which fixed old quoting surprises; it accepts the same host aliases as ssh.

Harden the SSH server

Never edit the long /etc/ssh/sshd_config directly. It includes every file in /etc/ssh/sshd_config.d/*.conf, and for most options the first value read wins, so a file with a low number overrides everything after it, including drop-ins that cloud images add (such as 50-cloud-init.conf or 60-cloudimg-settings.conf).

First make sure the people who need access are in an allowed group, including yourself:

# On orbit-web-01; on the lab VM, add ubuntu instead
$ sudo groupadd ssh-users
$ sudo usermod -aG ssh-users leo
$ sudo usermod -aG ssh-users maya
$ sudo usermod -aG ssh-users deploy

Then create /etc/ssh/sshd_config.d/10-orbit-hardening.conf:

PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin no
AllowGroups ssh-users
MaxAuthTries 3
X11Forwarding no

The first two lines disable every password-style login, PermitRootLogin no forces people to log in as themselves and use sudo (so logs show who did what), AllowGroups rejects any account outside the group even with a valid key, and MaxAuthTries 3 cuts brute-force attempts short. Now apply it safely:

$ sudo sshd -t
$ sudo sshd -T | grep -E '^(passwordauthentication|permitrootlogin|allowgroups)'
passwordauthentication no
permitrootlogin no
allowgroups ssh-users
$ sudo systemctl reload ssh

sshd -t checks the syntax and prints nothing when all is well; sshd -T prints the effective configuration, which proves your values won. Before reloading, keep your current session open and, after reloading, open a second terminal and log in again. Existing sessions survive a reload, so if the new login fails you still have a way in to fix it. On a cloud VM, also know where the provider’s serial console is.

Tip: Changing the port from 22 cuts log noise but is not security. Keys, no passwords, a firewall (lesson 17) and a cloud security group that only admits your office or VPN are.

Red Hat and Alpine

On RHEL, Rocky and Alma the service is sshd (sudo systemctl reload sshd), the drop-in directory works the same, and SELinux must know about a non-standard port (sudo semanage port -a -t ssh_port_t -p tcp 2222). If keys are refused after you copied .ssh in by hand, restorecon -Rv ~/.ssh fixes the SELinux labels. Alpine installs the server with apk add openssh and manages it with rc-service sshd reload under OpenRC.

Rotate and remove keys

Keys do not expire on their own, so rotation is a habit: create a new key, add its public half next to the old one, test a login with ssh -i newkey, then delete the old line. List what a server accepts, with comments, using the fingerprint view:

$ ssh-keygen -lf ~/.ssh/authorized_keys
256 SHA256:Qm3t8bV1cZ0yX4wR7nE2kD9sH5fJ6aL1pU8oI3gT2xY leo@laptop 2026-11 (ED25519)
256 SHA256:Za9pL4kR2vT6yB0nC8xW1mQ5eH3jU7sF2dG4aK6oI9b leo@old-laptop 2024-03 (ED25519)

When someone leaves the team, removing their line on every server is the step that gets forgotten. Manage authorized_keys with configuration management, or move to certificates (below), where access simply expires.

Deploy keys for CI

A pipeline that deploys Orbit should never use a person’s key. Give it the deploy user and a key of its own, generated without a passphrase, stored only in the CI system’s protected secret variables, and restricted on the server. In /home/deploy/.ssh/authorized_keys:

restrict,from="10.0.5.0/24",command="/usr/local/bin/orbit-deploy" ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK3...q8 ci-deploy orbit 2026-11
  • restrict turns off port, agent and X11 forwarding and the terminal.
  • from= accepts this key only from the runners’ network.
  • command= runs orbit-deploy whatever the client asks for; the script can read the request in $SSH_ORIGINAL_COMMAND and accept only, say, a release tag. For file copies, the rrsync script shipped with rsync limits the key to one directory.

Do not confuse this with a deploy key on a Git host, which is a public key with read access to a single repository. Both follow the same idea: one key, one job, nothing more. The CI/CD Tutorial shows how pipelines store such secrets.

Beyond static keys

  • FIDO2 hardware keys. ssh-keygen -t ed25519-sk -O resident -O verify-required keeps the private key inside a security key; logging in needs the device, a touch and its PIN. Malware on your laptop cannot copy it.
  • SSH certificates. A team CA signs a user’s public key with a short validity (ssh-keygen -s user_ca -I leo -n leo -V +8h key.pub), and servers trust the CA with one TrustedUserCAKeys line instead of hundreds of authorized_keys entries. Tools such as step-ca, HashiCorp Vault or OpenBao automate it.
  • Cloud session managers. AWS Systems Manager Session Manager, Google Cloud IAP and Azure Bastion give shell access through the cloud’s identity system and audit logs, often with port 22 closed entirely.

OpenSSH 10 in 2026

Ubuntu 26.04 and other current distributions ship OpenSSH 10 (check with ssh -V). Two changes matter: DSA keys are gone entirely, and the default key exchange is the post-quantum hybrid mlkem768x25519-sha256, which protects today’s traffic against being recorded and decrypted by a future quantum computer. ssh -v orbit-lab shows kex: algorithm: mlkem768x25519-sha256 in its debug output, and recent clients warn when a server cannot negotiate a post-quantum exchange, a good hint that the server needs an upgrade.

Practice lab

Leo is joining the Orbit on-call rotation and needs comfortable, secure access to the lab server.

  1. On your own computer, create ~/.ssh/orbit_ed25519 with a passphrase and a meaningful comment.
  2. Install the public key on orbit-lab, verify the host fingerprint and log in with the key over the VM’s IP.
  3. Add an orbit-lab entry to ~/.ssh/config and load the key into the agent, so ssh orbit-lab asks for nothing.
  4. On the VM, create the ssh-users group with ubuntu in it, add the hardening drop-in, test it and reload. Prove from a second terminal that you can still log in and that ssh -o PubkeyAuthentication=no orbit-lab is refused.
Show a solution

Run ssh-keygen -t ed25519 -C "leo@laptop 2026-11" -f ~/.ssh/orbit_ed25519, then find the IP with multipass info orbit-lab, copy the key with multipass transfer and append it with multipass exec orbit-lab -- bash -c 'cat ~/orbit_ed25519.pub >> ~/.ssh/authorized_keys'; compare multipass exec orbit-lab -- ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub with the prompt of ssh -i ~/.ssh/orbit_ed25519 ubuntu@<ip> before typing yes. Add Host orbit-lab with HostName <ip>, User ubuntu, IdentityFile ~/.ssh/orbit_ed25519 and AddKeysToAgent yes, then ssh-add ~/.ssh/orbit_ed25519 once. On the VM, sudo groupadd ssh-users and sudo usermod -aG ssh-users ubuntu, write /etc/ssh/sshd_config.d/10-orbit-hardening.conf, run sudo sshd -t and sudo sshd -T | grep allowgroups, then sudo systemctl reload ssh with your session still open. A fresh ssh orbit-lab works, and the forced password attempt ends with “Permission denied (publickey)”.

Exercise

Leo copied his public key into /home/leo/.ssh/authorized_keys on orbit-web-02 as root, but ssh orbit-web-02 still says “Permission denied (publickey)”. The server log shows “Authentication refused: bad ownership or modes for directory /home/leo/.ssh”. What fixes it?

  1. Set PasswordAuthentication yes and log in with a password instead
  2. Generate a new RSA 4096 key, because ed25519 is not supported
  3. Make leo the owner of ~/.ssh and its files, with mode 700 on the directory and 600 on authorized_keys
  4. Add the key to /root/.ssh/authorized_keys as well
Show answer

Answer: C. StrictModes rejects keys when the home directory, ~/.ssh or authorized_keys is owned by someone else or writable by others. chown -R leo:leo /home/leo/.ssh plus chmod 700 and 600 fixes it; the client error alone never tells you this, the server log does.

Exercise

You are about to disable password logins on orbit-web-01, and you are connected over SSH right now. Which sequence is safe?

  1. Edit sshd_config, restart ssh, log out and log back in to test
  2. Add a drop-in, run sudo sshd -t, reload ssh while keeping this session open, then test a new login from a second terminal
  3. Reboot the server so the new configuration is loaded cleanly
  4. Disable the firewall first so nothing can block the new configuration
Show answer

Answer: B. sshd -t catches syntax errors before they can stop the daemon, and the open session survives a reload, so if the new login fails you can still repair the configuration. Logging out first or rebooting removes your safety net.

Exercise

Orbit’s GitLab pipeline must run a deploy script on orbit-web-01 and orbit-web-02. What is the best way to give it SSH access?

  1. Store Leo’s personal private key in a CI variable, since he already has access
  2. Allow root login with a key so the script can do anything it needs
  3. Re-enable password authentication for the deploy user and keep the password in the pipeline
  4. A dedicated key for the deploy user, kept in a protected CI secret and limited in authorized_keys with restrict, from= and command=
Show answer

Answer: D. A pipeline key should belong to a dedicated account and do one job. restrict removes forwarding and terminals, from= limits where it can be used and command= pins what it can run, so a leaked key is far less useful.

Key takeaways

  • SSH checks the server first (host key against known_hosts), then you (your signature against authorized_keys); verify fingerprints out of band.
  • Use ssh-keygen -t ed25519 with a passphrase and a dated comment; keep ~/.ssh at 700 and private keys and authorized_keys at 600.
  • ssh-agent and ~/.ssh/config make secure access convenient; hop through bastions with ProxyJump, not agent forwarding.
  • Harden sshd with a low-numbered drop-in, test with sshd -t and sshd -T, and keep a session open while you reload.
  • Give automation its own restricted key, rotate keys, and consider FIDO2 keys, certificates or session managers as the fleet grows.

FAQ

Which SSH key type should I use in 2026?

Use ed25519 for everything new, or ed25519-sk if you have a FIDO2 security key. RSA with at least 3072 bits is only for old systems that cannot handle ed25519. DSA keys no longer work with OpenSSH 10, and ecdsa offers nothing ed25519 does not.

What does “REMOTE HOST IDENTIFICATION HAS CHANGED” mean?

The server presented a different host key from the one saved in your known_hosts. Usually the server was rebuilt or the IP now belongs to another machine, but it is also exactly what an interception looks like. Confirm the new fingerprint through a trusted channel, then remove the old entry with ssh-keygen -R hostname and connect again.

Can I copy my private key to a second laptop?

You can, but it is better to create one key per device. Then a lost laptop means removing one line from authorized_keys, not replacing your key everywhere, and the comment tells you which device each line belongs to.

Is SSH agent forwarding safe?

Only on servers you fully trust. While you are connected, anyone with root on the remote server can use your forwarded agent to log in elsewhere as you. For reaching private servers through a bastion, ProxyJump does the job without exposing your agent.

How do I log in to a server with a key file?

Run ssh -i ~/.ssh/keyfile user@host, and make sure the file has mode 600 or ssh ignores it. For hosts you use often, put HostName, User and IdentityFile under a Host alias in ~/.ssh/config, so ssh alias is enough.

Should I disable password authentication on every server?

Yes, once key access works for everyone who needs it and you have a recovery path such as a cloud console. Password logins are what bots attack all day, and keys with a passphrase or a hardware token are far stronger. Disable root login at the same time and restrict access with AllowGroups.

↑ ↓ navigate Enter open Esc close