Skip to content

Fix silent data loss in push_to_hub when num_proc > num_shards - #8044

Merged
lhoestq merged 2 commits into
huggingface:mainfrom
HaukurPall:fix-push-to-hub-data-loss-num-proc
Mar 9, 2026
Merged

Fix silent data loss in push_to_hub when num_proc > num_shards#8044
lhoestq merged 2 commits into
huggingface:mainfrom
HaukurPall:fix-push-to-hub-data-loss-num-proc

Conversation

@HaukurPall

Copy link
Copy Markdown
Contributor

Summary

When num_proc exceeds the number of output shards in Dataset.push_to_hub, workers
without assigned output shards silently drop their data. The metadata still reports the
correct dataset length (using len(self[split])), masking the data loss.

This fix caps num_jobs at num_shards in _push_parquet_shards_to_hub, matching
what the streaming path already does.

Reproduction

  1. Have a dataset with a small number of shards (e.g. 2)
  2. Call push_to_hub with num_proc > num_shards (e.g. 6)
  3. Only a fraction of the samples are actually written to the hub

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm !

Comment thread src/datasets/arrow_dataset.py Outdated
@lhoestq
lhoestq merged commit 6ef54e7 into huggingface:main Mar 9, 2026
@HaukurPall
HaukurPall deleted the fix-push-to-hub-data-loss-num-proc branch March 11, 2026 12:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants