🔤 Data Cleaning · 7 min read

CSV File Encoding: UTF-8, UTF-16 and Windows-1252 Explained

A CSV file's encoding is the map between the bytes in the file and the characters you see on screen. UTF-8 is the default you should produce; Windows-1252 is the legacy encoding that causes most of the mangled text you will meet. You do not have to guess which one a file uses: read its opening bytes for a marker called a BOM, check it in an editor that displays the encoding, or read it in Python with an explicit encoding argument. Once you know the pair — what it is and what it was read as — the conversion is a single save.

What "encoding" means for a CSV file

A file is bytes. A character is a code point in the Unicode standard. An encoding is the rule that turns one into the other. UTF-8 writes most characters as a single byte and anything outside plain ASCII as two to four bytes. Windows-1252 writes one byte per character from a much smaller set, which is why a character it cannot represent is lost outright rather than stored differently.

The CSV format itself says very little about this. RFC 4180 registers the text/csv media type with an optional charset parameter, and notes that common usage of CSV is US-ASCII but that other character sets defined by IANA for the text tree may be used with it. In other words, a CSV is simply a text file, and the encoding is whatever the program that wrote it decided.

The encodings a CSV actually arrives in

The marker column comes from the Unicode FAQ: a BOM is the character U+FEFF at the start of a data stream, and the bytes that appear on disk are whatever that character converts into in the chosen encoding form — EF BB BF for UTF-8, FF FE for UTF-16 little-endian, FE FF for UTF-16 big-endian, and 00 00 FE FF or FF FE 00 00 for the two UTF-32 forms.

How to check a CSV file's encoding

1. Read the first bytes

A BOM, if present, is the fastest answer because it is a signature rather than an inference. From a terminal:

head -c 3 data.csv | xxd
# efbbbf   -> UTF-8 with BOM
# fffe     -> UTF-16 little-endian
# feff     -> UTF-16 big-endian

No marker all means the file has to be inferred from its content. That is where the wrong guess comes from.

2. Ask a tool that reports it

file -i data.csv prints the detected charset. In an editor, Visual Studio Code and Notepad++ both show the encoding in a status bar; the same control lets you reopen the file with a different one to see whether the text becomes readable, which is the quickest way to confirm a theory without changing the file on disk.

3. Read it in Python

Two statements settle it. The first shows whether a BOM is present; the second forces an encoding and fails loudly if the guess is wrong:

with open("data.csv", "rb") as f:
    print(f.read(4))
# b'\xef\xbb\xbf'  -> UTF-8 with BOM
# b'\xff\xfe'       -> UTF-16 little-endian
# no BOM            -> try utf-8, then cp1252

import csv

with open("data.csv", newline="", encoding="utf-8") as f:
    print(next(csv.reader(f)))

A UnicodeDecodeError from encoding="utf-8" is useful information rather than a dead end: it means the file is not UTF-8, and cp1252 or utf-16 is usually the answer. The wider Python case, with pandas and chunked reads, is in how to open a CSV in Python.

The symptom tells you the pair

When you cannot inspect the file — a screenshot, a report from a colleague, a file already on a server — the shape of the mangled text names both encodings at once.

None of these are corruption. The bytes are unchanged; the reader used the wrong rule. That distinction matters, because if the file was saved in an encoding that could not represent the characters in the first place, the characters are gone and no conversion brings them back.

If the job is to get the text readable in a spreadsheet, the step-by-step repair is a separate guide: fix garbled CSV in Excel covers the import dialog, the BOM, and the re-encode options.

Which encoding to choose when you export

• UTF-8, no BOM — for anything read by software: database loads, API imports, scripts, data platforms. This is the default worth standardising on.
• UTF-8 with BOM — for a file that has to survive a double-click in Excel on a machine you do not control, because the marker saves the reader from guessing.
• UTF-16 — only when a target system demands it. It doubles the size of the file for text that is mostly ASCII and gives no benefit in return.
• Windows-1252 — expect to receive it from older Windows tools; do not create new files in it. Any character outside its set is replaced, and the replacement is permanent.

When the destination is Excel, the safest habit is to set the file origin in the import dialog rather than accepting the default, so the encoding is a decision instead of a guess. Microsoft documents the row and column ceiling for that import and export at 1,048,576 rows by 16,384 columns, which is worth knowing before you blame the encoding for a truncated file.

Converting a CSV file to UTF-8

Every method is the same two steps: read with the encoding the file is in, write with the encoding you want. The first half is the part that is easy to get wrong, and it is why you check before you convert.

import csv

with open("in.csv", newline="", encoding="cp1252") as f:
    rows = list(csv.reader(f))

with open("out.csv", "w", newline="", encoding="utf-8") as f:
    csv.writer(f).writerows(rows)

# add a BOM instead of plain utf-8 if the file must
# open correctly on a double-click in Excel

• In an editor: open the file, then use the encoding control to reopen it with the correct one, and save with UTF-8 (or UTF-8 with BOM).
• In Google Sheets: import the file so Google decodes it, then download as a CSV, which comes back as UTF-8.
• On the command line: iconv -f cp1252 -t utf-8 in.csv writes the converted stream, and adding -f utf-8 -t utf-8 lets you check whether a file is already valid UTF-8 before touching it.

Converting changes the encoding, not the structure. If a converted file arrives back with the columns fused into one, the problem was never the text — that is a delimiter question, and CSV opens in one column covers it. The CSV analyzer reads UTF-8 files directly, which is a quick way to confirm a converted file before you hand it to anything else.

Frequently asked questions

Sources for this article: the Unicode FAQ "UTF-8, UTF-16, UTF-32 & BOM" for the BOM character U+FEFF and the byte sequences EF BB BF, FF FE, FE FF and the two UTF-32 forms; RFC 4180 for the text/csv media type, its optional charset parameter and the CRLF line rule; and Microsoft Support "Import or export text (.txt or .csv) files" for the 1,048,576 by 16,384 import and export ceiling. The commands shown are from the Python, GNU file and iconv documentation. All were checked on the day this page was published.