Managing SSH PEM Keys on Windows: Lessons Learned from a CRLF Nightmare

The Setup

I needed a reliable way to start an SSH tunnel at boot, so here is what I did:

• Wrapped the ssh -R command in a PowerShell script (tunnel_service.ps1).
• Saved the user’s private key as "myapp_key.pem" in %APPDATA%\MyApp.
• Installed it as a Windows service via NSSM, with the service referencing powershell.exe -ExecutionPolicy Bypass -File tunnel_service.ps1.
• Logged both stdout and stderr to tunnel_output.log so that I could debug startup failures in a headless environment.

Everything seemed fine until I tried to start the service:

In PowerShell:

$nssmPath install MyAppService 'powershell.exe' "-ExecutionPolicy Bypass -File `"tunnel_service.ps1`""

PowerShell reported “Service installed successfully.” But when NSSM launched it, I got:

C:\> sc start MyAppService
SERVICE_NAME: MyAppService
  STATE              : 1  STOPPED
  WIN32_EXIT_CODE    : 1077

Suddenly, my tunnel process was crashing immediately, and I had no obvious error messages appearing in tunnel_output.log.

The Mystery: No Error, Just Exit

When you wrap SSH in NSSM, NSSM monitors the child process. If ssh.exe exits right away (exit code 255 on parse error, for example), NSSM marks the service STOPPED. Since I wasn’t getting any explanation, I ran this command:

ssh -vvv -i myapp_key.pem user@host

Finally, I got:

debug1: key_parse_private_pem: PEM_read_PrivateKey failed
debug1: Could not load private key 'myapp_key.pem': invalid format

An “invalid format” on a key I’d generated myself? That didn’t make sense until I dug into the raw bytes.

Diagnosing Line Endings

In Windows PowerShell 5.1, Format-Hex does not allow setting the byte count. But it defaults to 16 bytes per line. Here is the command for dumping the file:

[System.IO.File]::ReadAllBytes('myapp_key.pem') | Format-Hex

I was expecting that every line would end in 0D 0A (CRLF), but I was getting0D. In other words, the file had “Classic Mac” line endings (CR-only), leftover from some editor, or perhaps the user uploaded it from macOS without conversion.

Also, this was much clearer when checking the difference between the two files using dedicated tools.

The OpenSSH's PEM parser never saw the line breaks it needed to locate and decode the Base64 block. Why is that? Because the PEM parser on Windows only recognizes LF or CRLF as valid line markers.The result? A single giant line in memory, an immediate parse failure, and a stopped service.

Converting Line Endings Without Extra Newlines

My first instinct was to pipe through Set-Content:

(Get-Content myapp_key.pem) |
  Set-Content myapp_key_fixed.pem -Encoding Ascii

But that action added an extra blank line at the end of the file, so that's another format violation. PowerShell’s built-in cmdlets append a trailing newline, so the fix had to go lower-level:

1. Read lines (this strips the original EOL markers):

$lines = Get-Content -Path 'myapp_key.pem' -Encoding Ascii

2. Join them with CRLF (no trailing CRLF):

$content = [string]::Join("`r`n", $lines)

3. Write exactly what we want in ASCII only, with no extra newline:

[System.IO.File]::WriteAllText(
    'myapp_key_fixed.pem',
    $content,
    [System.Text.Encoding]::ASCII
)

Now the .pem file contains exactly:

-----BEGIN OPENSSH PRIVATE KEY-----\r\n
(Base64…)\r\n
…\r\n
-----END OPENSSH PRIVATE KEY-----    ← no trailing \r\n

SSH loaded it without error, and the service stood running.

Avoiding the User-Paste Problem

I realized an even simpler safeguard: to stop asking users to paste key content and instead, have them upload or point to a local .pem file. When you accept raw text input, you risk:

• Invisible whitespace (leading/trailing spaces)
• Extra blank lines
• Wrong encoding (UTF-8 BOMs or other code pages)

By taking a file path (e.g., via a GUI file picker or a -KeyPath parameter), you let the OS deliver a pristine byte-for-byte file, so there is no chance of stray characters.

Param(
  [Parameter(Mandatory)]
  [string]$KeyPath
)
# Validate existence
if (-not (Test-Path $KeyPath)) {
  Throw "Key file not found: $KeyPath"
}
# Copy byte-perfect:
Copy-Item -Path $KeyPath -Destination $fixedKeyPath -Force

No conversion needed, no text-processing hacks.