🐍 File Operations · 8 min read

Pandas read_csv and Too Many Columns: Why Columns Vanish or the Read Fails

If the columns exist in the file but not on screen, pandas is truncating the printout, not dropping data: run len(df.columns), then pd.set_option("display.max_columns", None). If instead you get ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 5, some rows carry more fields than the header, and the fix belongs at read time: usecols, names, or on_bad_lines. Those two symptoms share a search query and nothing else.

The third case is the file is genuinely wide and the read works but the process grows. All three are below with the output we got on pandas 2.3.3 and Python 3.13.5, so you can match your situation to one of them instead of guessing.

Which of the three problems do you have?

Problem 1: the columns are there, the printout is not

pandas documentation for display.max_columnssays: "If max_cols is exceeded, switch to truncate view. Depending on large_repr, objects are either centrally truncated or printed as a summary view. 'None' value means unlimited." On a default install the option reads 0, which means pandas works out how many columns fit the display and shows that many.

Here is a 60-column CSV read and printed in a script with display.width set to 80:

>>> wide = pd.read_csv("nc_wide_60.csv")
>>> wide.shape
(3, 60)
>>> print(wide.head())
   c00  c01  c02  c03  c04  c05  c06  ...  c53  c54  c55  c56  c57  c58  c59
0    0    1    2    3    4    5    6  ...   53   54   55   56   57   58   59
1  100  101  102  103  104  105  106  ...  153  154  155  156  157  158  159
2  200  201  202  203  204  205  206  ...  253  254  255  256  257  258  259

[3 rows x 60 columns]

Read the footer before you touch the data: [3 rows x 60 columns] is pandas telling you the frame is complete and the middle was elided for the terminal. The same frame with the option set to None rendered every column, 1,207 characters across four lines:

pd.set_option("display.max_columns", None)
print(wide.head())
# c00 c01 c02 ... c59, no ellipsis, 4 lines, 1207 chars

The documentation adds a warning worth repeating: printing a large number of columns "could cause your rendering environment (the browser, etc.) to crash", so reach for None on a 60-column frame, not on a 3,000-column one. For code that has to see every column, df.columns.tolist() or df.head().to_csv() is safer than a printout.

Problem 2: ParserError, "Expected 3 fields in line 3, saw 5"

This one is a real failure and the message is precise. Our test file had the header id,name,amount, a clean first row, then a third line with two extra values:

id,name,amount
1,ada,10
2,bo,20,extra,more
3,cy,30

And pandas stopped the whole read:

>>> pd.read_csv("nc_extra_fields.csv")
ParserError: Error tokenizing data. C error: Expected 3 fields in line 3, saw 5

The default is deliberate. The read_csv documentation for on_bad_linesdescribes a bad line as "a line with too many fields" and lists error as the default, with warn to keep a warning and skip the line, and skip to drop it silently. Four remedies, in the order we would try them:

• Fix the file. A field that contains the delimiter but is not quoted is the most common cause. The rules are in RFC 4180 section 2, and the symptom on the other side is the one described in CSV not importing properly.
• Select what you need. pd.read_csv(path, usecols=[0, 1, 2]) returned a 3 by 3 frame from the same ragged file. The documentation for usecols accepts positional indices or labels, and element order is ignored.
• Drop the bad rows. pd.read_csv(path, on_bad_lines="skip") returned 2 rows of 3 columns, which is honest about the loss: the third line is gone. Use warn first so you can see how many rows that costs.
• Impose your own shape. pd.read_csv(path, header=None, names=[...5 names...]) parsed cleanly but returned 4 rows, not 3: with header=None the real header row becomes data. Add skiprows=1 if the file does have a header.

Do not reach for low_memory=False here. It changes how pandas chunks the file while guessing dtypes; it does not reconcile field counts, and it will happily read a ragged file into wrong columns.

Problem 3: the file really is wide

A wide file reads fine and then costs memory in three places: the frame, the parser buffer, and whatever intermediate objects the read creates. We measured a 500-row by 3,000-column file, 6,962,391 bytes on disk:

shape: (500, 3000)
df.memory_usage(deep=True).sum(): 11.4 MB
process peak RSS delta during read: 28.1 MB   (resource.getrusage, pandas 2.3.3)

The frame is 11.4 MB and the read cost about 28 MB of peak memory, roughly two and a half times the frame. Two habits follow. Load only the columns you will use with usecols, and pin types with dtype so pandas does not build an object column full of Python strings. If the file is wider than a few thousand columns, the limiting factor is no longer pandas: the CSV column limit measured on this machine covers where stream parsers cope and where memory becomes the wall, and what a large file does to a hosted AI tool is the same arithmetic with an upload ceiling on top.

Count the columns before you read them

Two cheap checks answer "how many columns does this file actually have" before a DataFrame exists, which separates a display problem from a data problem in one command:

import csv, pandas as pd

with open("data.csv", newline="", encoding="utf-8-sig") as f:
    header = next(csv.reader(f))
print(len(header), header[:5])

# or, without reading any rows:
print(pd.read_csv("data.csv", nrows=0).columns.size)

If the header has 60 fields and the frame has 60 columns, the printout was the only thing hiding them. If the header has 60 and the read fails, the file is ragged. A field count per line, done with the same csv.reader loop, shows exactly which lines carry extra fields, which is the number the ParserError gives you and the one worth fixing at the source. For a first-pass look at types and widths without code, the CSV analyzerreports the parsed column count and each column's inferred type; the code-level basics are in opening a CSV in Python, and the leading-zero trap that bites after a successful read is in keeping leading zeros.

Frequently Asked Questions

Sources

• pandas 2.3.3 user guide, Options and settings: the display.max_columns entry, including the truncate-view behaviour and the warning about rendering large numbers of columns. Retrieved 2026-10-23.
• pandas 2.3.3 API reference, pandas.read_csv: the usecols parameter (positional indices or labels, order ignored) and on_bad_lines ("a line with too many fields"; default error; warn, skip). Retrieved 2026-10-23.
• RFC 4180, Common Format and MIME Type for CSV Files, section 2: quoting rules for fields that contain commas, quotes or line breaks. Retrieved 2026-10-23.
• Local measurement, 2026-10-23, Python 3.13.5 with pandas 2.3.3 on a 4 vCPU Linux host: a 60-column file of 894 bytes, a three-line ragged file, and a 500 by 3,000 file of 6,962,391 bytes. Script and output stored in seo_geo_ops/nc_measure_20261023/.