Get-Content -Encoding UTF8 fixed four of my log files and broke two others. I wrote the same string 13 ways to find out which is which.
I had two log files sitting in the same folder. One was written by my own script. One was written by a node process my script had launched. Get-Content read mine perfectly and returned garbage for node's. Adding -Encoding UTF8 fixed node's and broke mine.
So I wrote the same string with every writer I could think of, and read each file back both ways. The string is 12 characters of Japanese — it is the phrase a lot of tools print for "file not found", which is exactly the kind of line you cannot afford to lose.
Host: Windows 11, ja-JP, ACP=932, OEMCP=932, Windows PowerShell 5.1.26100.9168. [Console]::OutputEncoding = 932 (shift_jis), $OutputEncoding = 20127 (us-ascii).
writer first bytes bare -Enc UTF8
------------------------------- -------------- ---------- ----------
Out-File (default) FF FE D5 30 OK OK
Out-File -Encoding utf8 EF BB BF E3 OK OK
Out-File -Encoding ascii 3F 3F 3F 3F MOJIBAKE MOJIBAKE
Set-Content (default) 83 74 83 40 OK MOJIBAKE
Set-Content -Encoding UTF8 EF BB BF E3 OK OK
Add-Content (default) 83 74 83 40 OK MOJIBAKE
Tee-Object -FilePath FF FE D5 30 OK OK
> redirection FF FE D5 30 OK OK
IO.File WriteAllText (UTF8) E3 83 95 E3 MOJIBAKE OK
IO.File WriteAllBytes (UTF8) E3 83 95 E3 MOJIBAKE OK
python via cmd.exe > E3 83 95 E3 MOJIBAKE OK
node via cmd.exe > E3 83 95 E3 MOJIBAKE OK
node captured by PS, Out-File EF BB BF E7 MOJIBAKE MOJIBAKE
Three groups.
1. BOM present, text intact — 5 rows. Both reads work. Get-Content sniffs FF FE or EF BB BF and uses it. The read parameter is irrelevant. Note that Out-File, Tee-Object and > all default to UTF-16LE here, which is why they are in this group by accident rather than by anyone's intent.
2. No BOM — 6 rows. Exactly one read is correct, and which one flips depending on the writer.
Set-Content and Add-Content without -Encoding write the machine ANSI code page — 83 74 is CP932, not UTF-8 — so the bare read is right and -Encoding UTF8 is wrong. Everything that put real UTF-8 on disk without a BOM is the exact reverse. With no BOM, Get-Content falls back to ANSI, and that fallback is correct precisely when the writer also used ANSI.
This is the part I did not expect: "just add -Encoding UTF8" is not a safe default. Across these 13 files it corrects 4 and corrupts 2. There is no single read parameter that is right for all of them. If you have a folder holding both your own logs and a build tool's logs, no one setting reads both.
3. Damage that happened before the file existed — 2 rows. No read parameter can fix these.
Out-File -Encoding ascii wrote 3F 3F 3F 3F, which is literally ????. The characters were destroyed at write time.
The last row is the one worth your time. I let PowerShell capture node's stdout into a variable and re-write it with Out-File -Encoding utf8:
node via cmd.exe > 36 bytes 12 chars E3 83 95 E3 82 A1 E3 82 A4 E3 83 AB
U+30D5 U+30A1 U+30A4 U+30EB U+304C U+898B U+3064 U+304B
node captured by PS 65 bytes 20 chars EF BB BF E7 B9 9D E8 BC 94 E3 81 83
U+7E5D U+8F14 U+3043 U+7E67 U+FF64 U+7E5D U+FF6B U+7E3A
What PowerShell actually wrote into that file, all 20 characters of it:
繝輔ぃ繧､繝ｫ縺瑚ｦ九▽縺九ｊ縺ｾ縺帙ｓ
That second file carries a valid UTF-8 BOM and is well-formed UTF-8. It is also wrong. [Console]::OutputEncoding is 932 on this host, so PowerShell decoded node's UTF-8 bytes as CP932, got 20 different characters out of 12, and then faithfully encoded those as UTF-8 with a BOM. The file went from 36 bytes to 65. Nothing threw, nothing warned.
It is also the only row where the two reads agree with each other and are both wrong. Everywhere else, when one read returns garbage the other returns clean text, so there is a way to notice. Here there is no second opinion.
A BOM tells you how the file is encoded. It tells you nothing about whether the text in it is correct.
Minimal repro (numbers below are from the 932 host; on a Latin-1 ANSI code page the first pair behaves differently, because CP1252 cannot represent these characters at all):
$s = [char]0x30D5 + [char]0x30A1
$d = $env:TEMP
Set-Content -Path "$d\ansi.log" -Value $s
[IO.File]::WriteAllBytes("$d\utf8.log", [Text.Encoding]::UTF8.GetBytes($s))
(Get-Content "$d\ansi.log" -Raw).TrimEnd() -eq $s # True
(Get-Content "$d\ansi.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # False
(Get-Content "$d\utf8.log" -Raw).TrimEnd() -eq $s # False
(Get-Content "$d\utf8.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # True
Same cmdlet, same parameter, opposite answers, two files in one directory.
What I changed in my own scripts
- Reading a log a native child process wrote (redirected by
cmd.exe, so nothing decoded it on the way in): always pass-Encoding UTF8. That file holds the program's own bytes and will not have a BOM. - Reading a file PowerShell itself wrote: leave
Get-Contentbare. The BOM is there and handles it. Adding-Encoding UTF8here is what broke rows 4 and 6. - Do not capture a native process's stdout into a variable when the output can be non-ASCII. Redirect it to a file and read the file. That decode is governed by
[Console]::OutputEncoding, which was 932 here; I have not tested whether setting it to UTF-8 up front avoids the problem, so I am not claiming that it does. Out-File -Encoding asciion non-ASCII text is silent data loss, not a display issue.
Measured on one locale. If you are on a non-Latin ANSI code page I would be curious whether rows 4 and 6 come out the same for you — that is the pair that makes the usual advice backfire.