Datasetpublic

seonglae/wiki_dpr_token

Dataset Card for "wiki_dpr_token" Distribution [ { // Token length '~128': 2625007, '128~256': 18370607, '256~512': 19066, '512~1024': 571, '1024~2048': 47, '2048~4096': 2, '4096~8192': 0, '8192~16384': 0, '16384~32768': 0, '32768~65536': 0, '65536~128000': 0, '128000~': 0, }, { // Text length '~512': 86519, '512~1024': 20927180, '1024~2048': 1557, '2048~4096': 43, '4096~8192': 1… See the full description on the dataset page: https://huggingface.co/datasets/seonglae/wiki_dpr_token.

sourceHugging Faceupdated 3y agoView on Hugging Face

0likes186downloads

Dataset Card

Dataset Card for "wikidprtoken"

Distribution

[
  { // Token length
    '~128': 2625007,
    '128~256': 18370607,
    '256~512': 19066,
    '512~1024': 571,
    '1024~2048': 47,
    '2048~4096': 2,
    '4096~8192': 0,
    '8192~16384': 0,
    '16384~32768': 0,
    '32768~65536': 0,
    '65536~128000': 0,
    '128000~': 0,
  },
  { // Text length
    '~512': 86519,
    '512~1024': 20927180,
    '1024~2048': 1557,
    '2048~4096': 43,
    '4096~8192': 1,
    '8192~16384': 0,
    '16384~32768': 0,
    '32768~65536': 0,
    '65536~': 0,
  },
  { // Token distribution
    '~128': '12.49%',
    '128~256': '87.42%',
    '256~512': '0.09%',
    '512~1024': '0.00%',
    '1024~2048': '0.00%',
    '2048~4096': '0.00%',
    '4096~8192': '0.00%',
    '8192~16384': '0.00%',
    '16384~32768': '0.00%',
    '32768~65536': '0.00%',
    '65536~128000': '0.00%',
    '128000~': '0.00%',
  },
  { // Text distribution
    '~512': '0.41%',
    '512~1024': '99.58%',
    '1024~2048': '0.01%',
    '2048~4096': '0.00%',
    '4096~8192': '0.00%',
    '8192~16384': '0.00%',
    '16384~32768': '0.00%',
    '32768~65536': '0.00%',
    '65536~': '0.00%',
  }
]