# PBY public-domain Hebrew lines

Text from Project Ben-Yehuda (https://benyehuda.org), prepared by Project Ben-Yehuda volunteers.

Credit: [Project Ben-Yehuda volunteers](https://benyehuda.org).

- Source: `projectbenyehuda/public_domain_dump` at `5e4277fead7b565f32cc4b352abd3565023a77d8` (VERSION `2026-03`), 26455 catalogue rows.
- Rights computed for `as_of` 2026. `pd_israel` and `pd_us` are this build's reading of the law, not legal advice; see `pby_rights_index.v1.jsonl.gz`.
- Outputs are CC0-1.0. Source text: Public domain (public_domain_dump LICENSE); PBY requests credit, which this card gives.
- Release: `htext-cc0` (CC0-1.0); source licences: `Public Domain` (25267 works). Text quality: `transcription`.
- Builder: `htext` 0.1.0.

## Files

| File | Variant | Subset | Lines | Works | Bytes | sha256 |
| --- | --- | --- | ---: | ---: | ---: | --- |
| `pby_lines_il_plain_v3.txt` | il | plain | 12278848 | 16409 | 558952843 | `f033b0aaf76c84dfe9b6ff61dabbbc59d45c0080ada226f9a7180f9a0215bef0` |
| `pby_lines_il_niqqud_v3.txt` | il | niqqud | 1680290 | 15738 | 119885597 | `d436f87dd54be6a71070f2b1bcaa33dc8d575963d5de6bc8a968ad9c7591083d` |
| `pby_lines_il_us_plain_v3.txt` | il_us | plain | 407456 | 564 | 18956168 | `8de0454dfd135ff2987db8964e30e397f9d295f1335b995560e57764a2d961f0` |
| `pby_lines_il_us_niqqud_v3.txt` | il_us | niqqud | 18613 | 325 | 1325641 | `162e6fbd83762b0921d13375ca17726ca7335ebe9009554547f1ff8c345ec5b8` |

`il`: every line is from a work that is public domain in Israel. `il_us`: also public domain in the US.
`niqqud` lines are pointed (density at or above the threshold); `plain` lines carry no points.

## Length profile

Characters per line; niqqud lines are measured without their points.

| File | Max | Median | 31-55 | 36-47 | Passes |
| --- | ---: | ---: | ---: | ---: | --- |
| `pby_lines_il_plain_v3.txt` | 55 | 23.0 | 20.7% | 11.7% | all |
| `pby_lines_il_niqqud_v3.txt` | 55 | 21.0 | 17.6% | 10.2% | fails: share_31_55 |
| `pby_lines_il_us_plain_v3.txt` | 55 | 23.0 | 22.6% | 14.0% | all |
| `pby_lines_il_us_niqqud_v3.txt` | 55 | 23 | 17.2% | 5.3% | fails: share_31_55, share_36_47 |

Targets (S10b): at most 55 characters; 18%-35% of lines at 31-55; at least 10% at 36-47; median 18-30.

## Rights

| `pd_israel_basis` | Works |
| --- | ---: |
| `all_creators_died_by:1955` | 25601 |
| `anonymous_unresolved` | 164 |
| `creator_death_year_unknown` | 475 |
| `creator_in_term` | 215 |

| `pd_us_basis` | Works |
| --- | ---: |
| `published_before:1931` | 715 |
| `unknown` | 25740 |

Works with lines in this build: 25267 (`il`), 701 (`il_us`). Every other work has `htext_release: null` in the rights index: it is in no release.

### Creators in term (died after 1955)

Their works are `pd_israel: false` (`creator_in_term`) and contribute no line.

| Creator | Wikidata | Died | Source | Works | Works with lines |
| --- | --- | ---: | --- | ---: | ---: |
| זלמן שניאור | Q2357623 | 1959 | `wikidata` | 33 | 0 |
| נתן גורן | Q4144071 | 1956 | `wikidata` | 25 | 0 |
| חנניה ריכמן | Q12407568 | 1982 | `wikidata` | 18 | 0 |
| שמשון מלצר | Q7051598 | 2000 | `wikidata` | 18 | 0 |
| נפתלי הרץ טור־סיני | Q214141 | 1973 | `wikidata` | 16 | 0 |
| אמיל פוירשטיין | Q7043772 | 1993 | `wikidata` | 13 | 0 |
| דב סדן | Q2890635 | 1989 | `wikidata` | 7 | 0 |
| יהושע השל ייבין |  | 1970 | `manual` | 7 | 0 |
| גיטה אבינור | Q64682595 | 1975 | `wikidata` | 5 | 0 |
| יצחק דב ברקוביץ' | Q2909488 | 1967 | `wikidata` | 5 | 0 |
| משה אטר (אטינגר) | Q6900316 | 1990 | `wikidata` | 4 | 0 |
| ר' בנימין | Q12411166 | 1957 | `wikidata` | 4 | 0 |
| אברהם שלמון | Q16129332 | 1970 | `wikidata` | 3 | 0 |
| ישעיהו זנה | Q57418943 | 1960 | `wikidata` | 3 | 0 |
| ישראל ישעיהו | Q983232 | 1979 | `wikidata` | 3 | 0 |
| ס. יזהר | Q528647 | 2006 | `wikidata` | 3 | 0 |
| אהרן שמי | Q6921055 | 2010 | `wikidata` | 2 | 0 |
| דוד בן־גוריון | Q37610 | 1973 | `wikidata` | 2 | 0 |
| יוסף קלוזנר | Q964668 | 1958 | `wikidata` | 2 | 0 |
| יעקב פיכמן | Q1972109 | 1958 | `wikidata` | 2 | 0 |
| מנחם זלמן וולפובסקי | Q7024345 | 1975 | `wikidata` | 2 | 0 |
| מתתיהו לזרסון | Q7244922 | 1961 | `wikidata` | 2 | 0 |
| סמואיל מרשק | Q435584 | 1964 | `wikidata` | 2 | 0 |
| עמיהוד ארבל | Q12410379 | 1987 | `wikidata` | 2 | 0 |
| ק. א. ברתיני | Q4206184 | 1995 | `wikidata` | 2 | 0 |
| ראובן אבינועם | Q12411170 | 1974 | `wikidata` | 2 | 0 |
| שולמית איתן | Q16129642 | 2014 | `wikidata` | 2 | 0 |
| שמואל ישעיהו פנואלי | Q12411484 | 1965 | `wikidata` | 2 | 0 |
| אביגדור המאירי | Q2874331 | 1970 | `wikidata` | 1 | 0 |
| אברהם שלונסקי | Q373167 | 1973 | `wikidata` | 1 | 0 |
| אהרן אמיר | Q2891315 | 2008 | `wikidata` | 1 | 0 |
| אהרן זאב בן ישי | Q30526837 | 1977 | `wikidata` | 1 | 0 |
| אלכסנדר טברדובסקי [2042] | Q362828 | 1971 | `wikidata` | 1 | 0 |
| אנה אחמטובה | Q80440 | 1966 | `wikidata` | 1 | 0 |
| אפרים פרויד | Q108056032 | 2017 | `wikidata` | 1 | 0 |
| אריה רפאלי | Q30527733 | 1977 | `wikidata` | 1 | 0 |
| ב' טראוון | Q55072099 | 1969 | `wikidata` | 1 | 0 |
| בן־ציון ידידיה | Q111889802 | 1956 | `wikidata` | 1 | 0 |
| ברוך קרוא | Q12404704 | 1972 | `wikidata` | 1 | 0 |
| ברכה חבס | Q6910729 | 1968 | `wikidata` | 1 | 0 |
| ה. לייוויק | Q972092 | 1962 | `wikidata` | 1 | 0 |
| זאב אלבלינגר | Q28532615 | 1989 | `wikidata` | 1 | 0 |
| חיים אורלן | Q98400310 | 1997 | `wikidata` | 1 | 0 |
| יהודה ארז | Q12407813 | 1983 | `wikidata` | 1 | 0 |
| יהויכין סטוצ'בסקי | Q214140 | 1982 | `wikidata` | 1 | 0 |
| יוסף יואל ריבלין | Q7035023 | 1971 | `wikidata` | 1 | 0 |
| יוסף יזרעאלי | Q6546666 | 1973 | `wikidata` | 1 | 0 |
| ישראל זמורה | Q4192630 | 1983 | `wikidata` | 1 | 0 |
| מאיר פרבר | Q43196479 | 1993 | `wikidata` | 1 | 0 |
| מנחם קפליוק | Q12409291 | 1988 | `wikidata` | 1 | 0 |
| משה שרת | Q208477 | 1965 | `wikidata` | 1 | 0 |
| ניקולאי זבולוצקי [2029] | Q515872 | 1958 | `wikidata` | 1 | 0 |
| צבי ארד | Q12410810 | 1994 | `wikidata` | 1 | 0 |
| ציליה דראפקין | Q5057993 | 1956 | `wikidata` | 1 | 0 |
| שלמה הרברג | Q51719287 | 1966 | `wikidata` | 1 | 0 |
| שלמה צמח | Q7499621 | 1974 | `wikidata` | 1 | 0 |
| שמואל אביצור | Q7058607 | 1999 | `wikidata` | 1 | 0 |
| שמואל אייזנשטדט | Q19692169 | 1970 | `wikidata` | 1 | 0 |

## Build parameters and checks

- Genres: prose, article, memoir, letters, fables, poetry, reference, drama.
- Niqqud threshold: 0.5. Seed: 15. Max lines per work: None.
- Maqaf allowed: True.
- Content screen: none applied. No words are filtered by default: HeOCR has to read real historical text as written. `--screen-terms` exists for builds that need a screen.
- No-loss wrap check: {"ran": true, "hocrsyngen_version": null, "hocrsyngen_commit": "88c61af4c12cc4a462fc1f57a1c4027b3459366d", "bodies": [{"template_id": "printed_letter", "font_file": "Alef-Regular.ttf", "size": 42, "width": 968, "rows_per_line": null}, {"template_id": "handwritten_note", "font_file": "GveretLevin-Regular.ttf", "size": 46, "width": 968, "rows_per_line": null}, {"template_id": "archive_card", "font_file": "Alef-Regular.ttf", "size": 38, "width": 498, "rows_per_line": 2}, {"template_id": "ledger", "font_file": "Alef-Regular.ttf", "size": 30, "width": 206, "rows_per_line": 10}], "maqaf_glyph_check": true}.
- Length profile source: hocrsyngen texts/hebrew_lines_train_v1.txt.
- Leakage: no line, and no word 3-gram, is shared with these protected texts (pointed and stripped forms):
  - `HeOCR/heocr-restricted-eval` `dbench/synth/text_pool.json (sentences)` @ `e56d79e`, file sha256 `8d28491cd29b`
  - `HeOCR/heocr-restricted-eval` `preregistration/print_scan_PS-v1_2026-11-18/PS-v1_all_text.txt` @ `prereg/print`, file sha256 `56ece1442711`
  - `HeOCR/hocrgen` `docs/evaluation-fixtures/bidi-synthetic-corpus.txt` @ `c8fb74eb5d0a`, file sha256 `b325673476bc`
  - `HeOCR/hocrsyngen` `src/hocrsyngen/data/synthetic/texts/hebrew_lines.txt` @ `88c61af4c12c`, file sha256 `ac7e1dc2c21d`
  - `HeOCR/hocrsyngen` `src/hocrsyngen/data/synthetic/texts/hebrew_lines_eval_v1.txt` @ `88c61af4c12c`, file sha256 `89e52f8b90cd`
  - `HeOCR/hocrsyngen` `src/hocrsyngen/data/synthetic/texts/hebrew_lines_niqqud_eval_v1.txt` @ `88c61af4c12c`, file sha256 `974ffa19251b`
  - `HeOCR/hocrsyngen` `src/hocrsyngen/data/synthetic/texts/hebrew_mixed_lines_eval_v1.txt` @ `88c61af4c12c`, file sha256 `45a81fff1105`
  - `HeOCR/hocrsyngen` `src/hocrsyngen/data/synthetic/texts/hebrew_typographic_lines_eval_v1.txt` @ `88c61af4c12c`, file sha256 `336428359249`
- Eval reservations: works reserved by HeOCR's evaluation are left out whole. set `print_scan_PS-v1`: 4 pby, 20 reshumot units, file sha256 `c8842a90f442`; set `midcentury (test, dev, retired dev units; v0-v2 union)`: 60 plenary, 125 reshumot units, file sha256 `c8842a90f442`; set `P-mod eval and holdout units (v6-v7 union, incl. exposed units)`: 7 knesset_protocol, 138 plenary, 157 reshumot units, file sha256 `c8842a90f442`; set `D-bench synthetic text pool (sets A, B, C and the gate's fresh set)`: 83 reshumot units, file sha256 `c8842a90f442`; set `P-hist audit2 (Hebrew Wikisource proofread pages)`: 14 wikisource units, file sha256 `c8842a90f442`; set `line_verify scored lines (units held out whole)`: 2 benyehuda_nli_aligned, 10 pby, 23 wikisource units, file sha256 `c8842a90f442`; set `h_eval_v1 (handwriting)`: 34 hash units, file sha256 `c8842a90f442`; set `P-hist eval hold-out v1 (27 P-hist books + 3 Rashi-script stratum; with audit2, 40 P-hist books)`: 13 ia, 11 pby, 5 sefaria, 23 wikisource units, file sha256 `c8842a90f442`; set `P-hist eval hold-out v1 top-up (standard square; 8 books + 2 reserves)`: 10 ia, 6 pby units, file sha256 `c8842a90f442`

## Pipeline counts

| Step | Count |
| --- | ---: |
| `segments` | 14847588 |
| `segments_dropped:charset` | 100784 |
| `segments_dropped:duplicate` | 600851 |
| `segments_dropped:fit:archive_card:too_many_rows` | 31705 |
| `segments_dropped:fit:archive_card:word_too_wide` | 171 |
| `segments_dropped:fit:ledger:word_too_wide` | 33628 |
| `segments_dropped:fit:printed_letter:word_too_wide` | 28 |
| `segments_dropped:leak_line` | 192 |
| `segments_dropped:leak_word3` | 40297 |
| `segments_dropped:long_digit_run` | 67 |
| `segments_dropped:medial_final_form` | 2160 |
| `segments_dropped:too_few_letters` | 77751 |
| `segments_dropped:too_long` | 816 |
| `source_lines` | 5469772 |
| `source_lines_dropped:entity` | 8426 |
| `source_lines_dropped:latin` | 53138 |
| `source_lines_dropped:letter_digit` | 27031 |
| `source_lines_dropped:url` | 1 |
| `source_lines_plain_only:word_mismatch` | 137 |
| `source_lines_split:verse_break_mark` | 91069 |
| `works_read` | 25571 |
| `works_skipped:eval_reserved` | 31 |
| `works_skipped:genre` | 2 |
| `works_skipped:not_pd_israel` | 851 |
| `works_with_lines` | 25267 |

## Niqqud density histogram

Segments that had a pointed form, by the share of Hebrew letters carrying a point.

| Density | Segments |
| --- | ---: |
| 0.0-0.1 | 12087462 |
| 0.1-0.2 | 499404 |
| 0.2-0.3 | 197105 |
| 0.3-0.4 | 43912 |
| 0.4-0.5 | 21887 |
| 0.5-0.6 | 218137 |
| 0.6-0.7 | 887246 |
| 0.7-0.8 | 590214 |
| 0.8-0.9 | 107883 |
| 0.9-1.0 | 11247 |
