Skip to content

Fix JSON loader schema inference for files starting with a UTF-8 BOM (#8241) - #8243

Merged
lhoestq merged 1 commit into
huggingface:mainfrom
archievi:fix/json-utf8-bom-schema-8241
Jun 9, 2026
Merged

Fix JSON loader schema inference for files starting with a UTF-8 BOM (#8241)#8243
lhoestq merged 1 commit into
huggingface:mainfrom
archievi:fix/json-utf8-bom-schema-8241

Conversation

@archievi

@archievi archievi commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

Fixes #8241.

What's the bug

When a JSONL file starts with a UTF-8 BOM (\xef\xbb\xbf), load_dataset("json", ...) infers a different schema than the same file without the BOM.

The JSON builder's first-batch "mixed-struct-types" pre-scan parses each line with ujson to decide whether to preserve heterogeneous nested dicts as a Json type:

try:
    examples = [ujson_loads(line) for line in batch.splitlines()]
except ValueError:
    pass
else:
    json_field_paths += find_mixed_struct_types_field_paths(examples)

ujson rejects the leading BOM with ValueError, the broad except ValueError swallows it, and find_mixed_struct_types_field_paths is never called. PyArrow's read_json tolerates the BOM and instead unifies the schema, materializing spurious nulls for fields absent from individual records. So the user-visible content silently changes depending on whether the file happens to start with a BOM.

Fix

Strip a leading UTF-8 BOM from the first batch (guarded to the start of the file) so the pre-scan and PyArrow operate on the same bytes.

Test

Added a regression test: a BOM-prefixed variant of the existing jsonl_file_with_lists_of_dicts_of_varying_keys fixture, asserting it produces the same schema (EXPECTED_LISTS_OF_DICTS_WITH_VARYING_KEYS) as the non-BOM file. It fails on main and passes with this change.

ruff check, ruff format, and pytest tests/packaged_modules/test_json.py::test_json_generate_tables all pass locally.

…uggingface#8241)

A leading UTF-8 BOM made the first-batch mixed-struct pre-scan (ujson) raise
ValueError, which was silently swallowed, so the schema-unification path ran
instead and materialized spurious nulls. PyArrow tolerates the BOM, so the
inferred schema depended on whether the file started with one. Strip the BOM so
the pre-scan and PyArrow see the same bytes. Adds a regression test.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm !

@lhoestq
lhoestq merged commit 35a7f29 into huggingface:main Jun 9, 2026
10 of 14 checks passed
Elsword016 pushed a commit to Elsword016/datasets that referenced this pull request Aug 6, 2026
…uggingface#8241) (huggingface#8243)

A leading UTF-8 BOM made the first-batch mixed-struct pre-scan (ujson) raise
ValueError, which was silently swallowed, so the schema-unification path ran
instead and materialized spurious nulls. PyArrow tolerates the BOM, so the
inferred schema depended on whether the file started with one. Strip the BOM so
the pre-scan and PyArrow see the same bytes. Adds a regression test.

Co-authored-by: archievi <13202986+archievi@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

load_dataset("json", ...) infers a different schema when the JSONL file starts with a UTF-8 BOM

3 participants