load_dataset("json", ...) infers a different schema when the JSONL file starts with a UTF-8 BOM #8241

Describe the bug

When a JSONL file starts with a UTF-8 BOM (\xef\xbb\xbf), datasets.load_dataset("json", data_files=...) infers a different schema than when the same file is loaded without the BOM. Specifically, the loader's "mixed-struct-types" pre-scan — which normally promotes columns with heterogeneous nested keys to a Json extension type so each dict is preserved as-is — is silently skipped, and PyArrow's normal schema unification kicks in instead. This materializes null values for fields that were absent in the source data, changing the user-visible content of records.

The root cause is in datasets/packaged_modules/json/json.py (the JSON builder). On the first batch, the loader parses each line with ujson_loads to detect mixed struct schemas:

try:
    examples = [ujson_loads(line) for line in batch.splitlines()]
except ValueError:
    # the file is likely not JSON Lines and may contain one single multi-line JSON object
    pass
else:
    json_field_paths += find_mixed_struct_types_field_paths(examples)

ujson rejects the leading BOM with ValueError('Expected object or value'), the broad except ValueError: pass swallows it, and find_mixed_struct_types_field_paths is never called. PyArrow's paj.read_json, on the other hand, tolerates the BOM and goes on to parse the file successfully — but with a unified list<struct<...>> schema where every element must conform to the same struct, producing spurious nulls for absent fields.

This is a silent correctness issue: no warning is logged, and downstream code that checks key in record (or that is sensitive to extra None values, e.g. validators or API payloads) will behave differently depending on whether the file happens to start with a BOM.

Steps to reproduce the bug

Two minimal JSONL files with identical content — except bom.jsonl begins with a UTF-8 BOM.
nobom.jsonl

{"id": 1, "tags": [{"name": "a"}, {"name": "b", "weight": 0.5}]}
{"id": 2, "tags": [{"name": "c"}]}
{"id": 3, "tags": [{"name": "d", "weight": 0.9}]}

bom.jsonl is byte-for-byte identical except for the leading 3 bytes:

$ xxd bom.jsonl   | head -1
00000000: efbb bf7b 2269 6422 3a20 312c 2022 7461  ...{"id": 1, "ta
$ xxd nobom.jsonl | head -1
00000000: 7b22 6964 223a 2031 2c20 2274 6167 7322  {"id": 1, "tags"

Repro script:

import pathlib
import datasets

HERE = pathlib.Path(__file__).parent

for name in ("nobom", "bom"):
    ds = datasets.load_dataset(
        "json",
        data_files={"train": str(HERE / f"{name}.jsonl")},
    )["train"]
    print(f"=== {name} ===")
    print("schema  :", ds.features)
    print("row 0   :", ds[0])
    print("tags[0] :", ds[0]["tags"][0])
    print()

Output:

=== nobom ===
schema  : {'id': Value('int64'), 'tags': List(Json(decode=True))}
row 0   : {'id': 1, 'tags': [{'name': 'a'}, {'name': 'b', 'weight': 0.5}]}
tags[0] : {'name': 'a'}

=== bom ===
schema  : {'id': Value('int64'), 'tags': List({'name': Value('string'), 'weight': Value('float64')})}
row 0   : {'id': 1, 'tags': [{'name': 'a', 'weight': None}, {'name': 'b', 'weight': 0.5}]}
tags[0] : {'name': 'a', 'weight': None}

Note that the BOM version injects 'weight': None into the first tag, even though the source line contains {"name": "a"} with no weight key.

repro.zip

Expected behavior

A leading UTF-8 BOM should not change the inferred schema or the content of loaded records. Either:

1. Strip the BOM (or otherwise normalize the input) before running the mixed-struct-types pre-scan, so the same code path is taken regardless of BOM, or
2. At minimum, log a warning when the pre-scan is skipped due to a ValueError, so silently divergent schema inference doesn't go unnoticed.

In both files above, the expected result is the nobom output: tags inferred as List(Json(decode=True)) and each dict preserved exactly as written.

Environment info

- `datasets` version: 4.8.5
- Platform: Linux-6.6.87.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
- Python version: 3.12.3
- `huggingface_hub` version: 1.12.2
- PyArrow version: 21.0.0
- Pandas version: 2.3.3
- `fsspec` version: 2026.2.0