Team Ai
Datasetpublic

Brunobkr/llama.cpp_AlgMor24_github

ΩFFFΣLLIa • llama.cpp • AlgMor24 ██████╗ ███████╗███████╗███████╗██╗ ██╗ ██╗ █████╗ ██╔═══██╗██╔════╝██╔════╝██╔════╝██║ ██║ ██║██╔══██╗ ██║ ██║█████╗ █████╗ █████╗ ██║ ██║ ██║███████║ ██║ ██║██╔══╝ ██╔══╝ ██╔══╝ ██║ ██║ ██║██╔══██║ ╚██████╔╝██║ ██║ ███████╗███████╗███████╗██║██║ ██║ ╚═════╝ ╚═╝ ╚═╝ ╚══════╝╚══════╝╚══════╝╚═╝╚═╝ ╚═╝ High-Performance LLM / VLM Inference & Autonomous Agentic Ecosystem… See the full description on the dataset page: https://huggingface.co/datasets/Brunobkr/llama.cpp_AlgMor24_github.

sourceHugging Faceupdated 2mo agoView on Hugging Face
0likes3.1kdownloads
README.md826 linesDownload Raw Back to minipass
1# minipass2 3A _very_ minimal implementation of a [PassThrough4stream](https://nodejs.org/api/stream.html#stream_class_stream_passthrough)5 6[It's very7fast](https://docs.google.com/spreadsheets/d/1K_HR5oh3r80b8WVMWCPPjfuWXUgfkmhlX7FGI6JJ8tY/edit?usp=sharing)8for objects, strings, and buffers.9 10Supports `pipe()`ing (including multi-`pipe()` and backpressure11transmission), buffering data until either a `data` event handler12or `pipe()` is added (so you don't lose the first chunk), and13most other cases where PassThrough is a good idea.14 15There is a `read()` method, but it's much more efficient to16consume data from this stream via `'data'` events or by calling17`pipe()` into some other stream. Calling `read()` requires the18buffer to be flattened in some cases, which requires copying19memory.20 21If you set `objectMode: true` in the options, then whatever is22written will be emitted. Otherwise, it'll do a minimal amount of23Buffer copying to ensure proper Streams semantics when `read(n)`24is called.25 26`objectMode` can only be set at instantiation. Attempting to27write something other than a String or Buffer without having set28`objectMode` in the options will throw an error.29 30This is not a `through` or `through2` stream. It doesn't31transform the data, it just passes it right through. If you want32to transform the data, extend the class, and override the33`write()` method. Once you're done transforming the data however34you want, call `super.write()` with the transform output.35 36For some examples of streams that extend Minipass in various37ways, check out:38 39- [minizlib](http://npm.im/minizlib)40- [fs-minipass](http://npm.im/fs-minipass)41- [tar](http://npm.im/tar)42- [minipass-collect](http://npm.im/minipass-collect)43- [minipass-flush](http://npm.im/minipass-flush)44- [minipass-pipeline](http://npm.im/minipass-pipeline)45- [tap](http://npm.im/tap)46- [tap-parser](http://npm.im/tap-parser)47- [treport](http://npm.im/treport)48- [minipass-fetch](http://npm.im/minipass-fetch)49- [pacote](http://npm.im/pacote)50- [make-fetch-happen](http://npm.im/make-fetch-happen)51- [cacache](http://npm.im/cacache)52- [ssri](http://npm.im/ssri)53- [npm-registry-fetch](http://npm.im/npm-registry-fetch)54- [minipass-json-stream](http://npm.im/minipass-json-stream)55- [minipass-sized](http://npm.im/minipass-sized)56 57## Usage in TypeScript58 59The `Minipass` class takes three type template definitions:60 61- `RType` the type being read, which defaults to `Buffer`. If62  `RType` is `string`, then the constructor _must_ get an options63  object specifying either an `encoding` or `objectMode: true`.64  If it's anything other than `string` or `Buffer`, then it65  _must_ get an options object specifying `objectMode: true`.66- `WType` the type being written. If `RType` is `Buffer` or67  `string`, then this defaults to `ContiguousData` (Buffer,68  string, ArrayBuffer, or ArrayBufferView). Otherwise, it69  defaults to `RType`.70- `Events` type mapping event names to the arguments emitted71  with that event, which extends `Minipass.Events`.72 73To declare types for custom events in subclasses, extend the74third parameter with your own event signatures. For example:75 76```js77import { Minipass } from 'minipass'78 79// a NDJSON stream that emits 'jsonError' when it can't stringify80export interface Events extends Minipass.Events {81  jsonError: [e: Error]82}83 84export class NDJSONStream extends Minipass<string, any, Events> {85  constructor() {86    super({ objectMode: true })87  }88 89  // data is type `any` because that's WType90  write(data, encoding, cb) {91    try {92      const json = JSON.stringify(data)93      return super.write(json + '\n', encoding, cb)94    } catch (er) {95      if (!er instanceof Error) {96        er = Object.assign(new Error('json stringify failed'), {97          cause: er,98        })99      }100      // trying to emit with something OTHER than an error will101      // fail, because we declared the event arguments type.102      this.emit('jsonError', er)103    }104  }105}106 107const s = new NDJSONStream()108s.on('jsonError', e => {109  // here, TS knows that e is an Error110})111```112 113Emitting/handling events that aren't declared in this way is114fine, but the arguments will be typed as `unknown`.115 116## Differences from Node.js Streams117 118There are several things that make Minipass streams different119from (and in some ways superior to) Node.js core streams.120 121Please read these caveats if you are familiar with node-core122streams and intend to use Minipass streams in your programs.123 124You can avoid most of these differences entirely (for a very125small performance penalty) by setting `{async: true}` in the126constructor options.127 128### Timing129 130Minipass streams are designed to support synchronous use-cases.131Thus, data is emitted as soon as it is available, always. It is132buffered until read, but no longer. Another way to look at it is133that Minipass streams are exactly as synchronous as the logic134that writes into them.135 136This can be surprising if your code relies on137`PassThrough.write()` always providing data on the next tick138rather than the current one, or being able to call `resume()` and139not have the entire buffer disappear immediately.140 141However, without this synchronicity guarantee, there would be no142way for Minipass to achieve the speeds it does, or support the143synchronous use cases that it does. Simply put, waiting takes144time.145 146This non-deferring approach makes Minipass streams much easier to147reason about, especially in the context of Promises and other148flow-control mechanisms.149 150Example:151 152```js153// hybrid module, either works154import { Minipass } from 'minipass'155// or:156const { Minipass } = require('minipass')157 158const stream = new Minipass()159stream.on('data', () => console.log('data event'))160console.log('before write')161stream.write('hello')162console.log('after write')163// output:164// before write165// data event166// after write167```168 169### Exception: Async Opt-In170 171If you wish to have a Minipass stream with behavior that more172closely mimics Node.js core streams, you can set the stream in173async mode either by setting `async: true` in the constructor174options, or by setting `stream.async = true` later on.175 176```js177// hybrid module, either works178import { Minipass } from 'minipass'179// or:180const { Minipass } = require('minipass')181 182const asyncStream = new Minipass({ async: true })183asyncStream.on('data', () => console.log('data event'))184console.log('before write')185asyncStream.write('hello')186console.log('after write')187// output:188// before write189// after write190// data event <-- this is deferred until the next tick191```192 193Switching _out_ of async mode is unsafe, as it could cause data194corruption, and so is not enabled. Example:195 196```js197import { Minipass } from 'minipass'198const stream = new Minipass({ encoding: 'utf8' })199stream.on('data', chunk => console.log(chunk))200stream.async = true201console.log('before writes')202stream.write('hello')203setStreamSyncAgainSomehow(stream) // <-- this doesn't actually exist!204stream.write('world')205console.log('after writes')206// hypothetical output would be:207// before writes208// world209// after writes210// hello211// NOT GOOD!212```213 214To avoid this problem, once set into async mode, any attempt to215make the stream sync again will be ignored.216 217```js218const { Minipass } = require('minipass')219const stream = new Minipass({ encoding: 'utf8' })220stream.on('data', chunk => console.log(chunk))221stream.async = true222console.log('before writes')223stream.write('hello')224stream.async = false // <-- no-op, stream already async225stream.write('world')226console.log('after writes')227// actual output:228// before writes229// after writes230// hello231// world232```233 234### No High/Low Water Marks235 236Node.js core streams will optimistically fill up a buffer,237returning `true` on all writes until the limit is hit, even if238the data has nowhere to go. Then, they will not attempt to draw239more data in until the buffer size dips below a minimum value.240 241Minipass streams are much simpler. The `write()` method will242return `true` if the data has somewhere to go (which is to say,243given the timing guarantees, that the data is already there by244the time `write()` returns).245 246If the data has nowhere to go, then `write()` returns false, and247the data sits in a buffer, to be drained out immediately as soon248as anyone consumes it.249 250Since nothing is ever buffered unnecessarily, there is much less251copying data, and less bookkeeping about buffer capacity levels.252 253### Hazards of Buffering (or: Why Minipass Is So Fast)254 255Since data written to a Minipass stream is immediately written256all the way through the pipeline, and `write()` always returns257true/false based on whether the data was fully flushed,258backpressure is communicated immediately to the upstream caller.259This minimizes buffering.260 261Consider this case:262 263```js264const { PassThrough } = require('stream')265const p1 = new PassThrough({ highWaterMark: 1024 })266const p2 = new PassThrough({ highWaterMark: 1024 })267const p3 = new PassThrough({ highWaterMark: 1024 })268const p4 = new PassThrough({ highWaterMark: 1024 })269 270p1.pipe(p2).pipe(p3).pipe(p4)271p4.on('data', () => console.log('made it through'))272 273// this returns false and buffers, then writes to p2 on next tick (1)274// p2 returns false and buffers, pausing p1, then writes to p3 on next tick (2)275// p3 returns false and buffers, pausing p2, then writes to p4 on next tick (3)276// p4 returns false and buffers, pausing p3, then emits 'data' and 'drain'277// on next tick (4)278// p3 sees p4's 'drain' event, and calls resume(), emitting 'resume' and279// 'drain' on next tick (5)280// p2 sees p3's 'drain', calls resume(), emits 'resume' and 'drain' on next tick (6)281// p1 sees p2's 'drain', calls resume(), emits 'resume' and 'drain' on next282// tick (7)283 284p1.write(Buffer.alloc(2048)) // returns false285```286 287Along the way, the data was buffered and deferred at each stage,288and multiple event deferrals happened, for an unblocked pipeline289where it was perfectly safe to write all the way through!290 291Furthermore, setting a `highWaterMark` of `1024` might lead292someone reading the code to think an advisory maximum of 1KiB is293being set for the pipeline. However, the actual advisory294buffering level is the _sum_ of `highWaterMark` values, since295each one has its own bucket.296 297Consider the Minipass case:298 299```js300const m1 = new Minipass()301const m2 = new Minipass()302const m3 = new Minipass()303const m4 = new Minipass()304 305m1.pipe(m2).pipe(m3).pipe(m4)306m4.on('data', () => console.log('made it through'))307 308// m1 is flowing, so it writes the data to m2 immediately309// m2 is flowing, so it writes the data to m3 immediately310// m3 is flowing, so it writes the data to m4 immediately311// m4 is flowing, so it fires the 'data' event immediately, returns true312// m4's write returned true, so m3 is still flowing, returns true313// m3's write returned true, so m2 is still flowing, returns true314// m2's write returned true, so m1 is still flowing, returns true315// No event deferrals or buffering along the way!316 317m1.write(Buffer.alloc(2048)) // returns true318```319 320It is extremely unlikely that you _don't_ want to buffer any data321written, or _ever_ buffer data that can be flushed all the way322through. Neither node-core streams nor Minipass ever fail to323buffer written data, but node-core streams do a lot of324unnecessary buffering and pausing.325 326As always, the faster implementation is the one that does less327stuff and waits less time to do it.328 329### Immediately emit `end` for empty streams (when not paused)330 331If a stream is not paused, and `end()` is called before writing332any data into it, then it will emit `end` immediately.333 334If you have logic that occurs on the `end` event which you don't335want to potentially happen immediately (for example, closing file336descriptors, moving on to the next entry in an archive parse337stream, etc.) then be sure to call `stream.pause()` on creation,338and then `stream.resume()` once you are ready to respond to the339`end` event.340 341However, this is _usually_ not a problem because:342 343### Emit `end` When Asked344 345One hazard of immediately emitting `'end'` is that you may not346yet have had a chance to add a listener. In order to avoid this347hazard, Minipass streams safely re-emit the `'end'` event if a348new listener is added after `'end'` has been emitted.349 350Ie, if you do `stream.on('end', someFunction)`, and the stream351has already emitted `end`, then it will call the handler right352away. (You can think of this somewhat like attaching a new353`.then(fn)` to a previously-resolved Promise.)354 355To prevent calling handlers multiple times who would not expect356multiple ends to occur, all listeners are removed from the357`'end'` event whenever it is emitted.358 359### Emit `error` When Asked360 361The most recent error object passed to the `'error'` event is362stored on the stream. If a new `'error'` event handler is added,363and an error was previously emitted, then the event handler will364be called immediately (or on `process.nextTick` in the case of365async streams).366 367This makes it much more difficult to end up trying to interact368with a broken stream, if the error handler is added after an369error was previously emitted.370 371### Impact of "immediate flow" on Tee-streams372 373A "tee stream" is a stream piping to multiple destinations:374 375```js376const tee = new Minipass()377t.pipe(dest1)378t.pipe(dest2)379t.write('foo') // goes to both destinations380```381 382Since Minipass streams _immediately_ process any pending data383through the pipeline when a new pipe destination is added, this384can have surprising effects, especially when a stream comes in385from some other function and may or may not have data in its386buffer.387 388```js389// WARNING! WILL LOSE DATA!390const src = new Minipass()391src.write('foo')392src.pipe(dest1) // 'foo' chunk flows to dest1 immediately, and is gone393src.pipe(dest2) // gets nothing!394```395 396One solution is to create a dedicated tee-stream junction that397pipes to both locations, and then pipe to _that_ instead.398 399```js400// Safe example: tee to both places401const src = new Minipass()402src.write('foo')403const tee = new Minipass()404tee.pipe(dest1)405tee.pipe(dest2)406src.pipe(tee) // tee gets 'foo', pipes to both locations407```408 409The same caveat applies to `on('data')` event listeners. The410first one added will _immediately_ receive all of the data,411leaving nothing for the second:412 413```js414// WARNING! WILL LOSE DATA!415const src = new Minipass()416src.write('foo')417src.on('data', handler1) // receives 'foo' right away418src.on('data', handler2) // nothing to see here!419```420 421Using a dedicated tee-stream can be used in this case as well:422 423```js424// Safe example: tee to both data handlers425const src = new Minipass()426src.write('foo')427const tee = new Minipass()428tee.on('data', handler1)429tee.on('data', handler2)430src.pipe(tee)431```432 433All of the hazards in this section are avoided by setting `{434async: true }` in the Minipass constructor, or by setting435`stream.async = true` afterwards. Note that this does add some436overhead, so should only be done in cases where you are willing437to lose a bit of performance in order to avoid having to refactor438program logic.439 440## USAGE441 442It's a stream! Use it like a stream and it'll most likely do what443you want.444 445```js446import { Minipass } from 'minipass'447const mp = new Minipass(options) // options is optional448mp.write('foo')449mp.pipe(someOtherStream)450mp.end('bar')451```452 453### OPTIONS454 455- `encoding` How would you like the data coming _out_ of the456  stream to be encoded? Accepts any values that can be passed to457  `Buffer.toString()`.458- `objectMode` Emit data exactly as it comes in. This will be459  flipped on by default if you write() something other than a460  string or Buffer at any point. Setting `objectMode: true` will461  prevent setting any encoding value.462- `async` Defaults to `false`. Set to `true` to defer data463  emission until next tick. This reduces performance slightly,464  but makes Minipass streams use timing behavior closer to Node465  core streams. See [Timing](#timing) for more details.466- `signal` An `AbortSignal` that will cause the stream to unhook467  itself from everything and become as inert as possible. Note468  that providing a `signal` parameter will make `'error'` events469  no longer throw if they are unhandled, but they will still be470  emitted to handlers if any are attached.471 472### API473 474Implements the user-facing portions of Node.js's `Readable` and475`Writable` streams.476 477### Methods478 479- `write(chunk, [encoding], [callback])` - Put data in. (Note480  that, in the base Minipass class, the same data will come out.)481  Returns `false` if the stream will buffer the next write, or482  true if it's still in "flowing" mode.483- `end([chunk, [encoding]], [callback])` - Signal that you have484  no more data to write. This will queue an `end` event to be485  fired when all the data has been consumed.486- `pause()` - No more data for a while, please. This also487  prevents `end` from being emitted for empty streams until the488  stream is resumed.489- `resume()` - Resume the stream. If there's data in the buffer,490  it is all discarded. Any buffered events are immediately491  emitted.492- `pipe(dest)` - Send all output to the stream provided. When493  data is emitted, it is immediately written to any and all pipe494  destinations. (Or written on next tick in `async` mode.)495- `unpipe(dest)` - Stop piping to the destination stream. This is496  immediate, meaning that any asynchronously queued data will497  _not_ make it to the destination when running in `async` mode.498  - `options.end` - Boolean, end the destination stream when the499    source stream ends. Default `true`.500  - `options.proxyErrors` - Boolean, proxy `error` events from501    the source stream to the destination stream. Note that errors502    are _not_ proxied after the pipeline terminates, either due503    to the source emitting `'end'` or manually unpiping with504    `src.unpipe(dest)`. Default `false`.505- `on(ev, fn)`, `emit(ev, fn)` - Minipass streams are506  EventEmitters. Some events are given special treatment,507  however. (See below under "events".)508- `promise()` - Returns a Promise that resolves when the stream509  emits `end`, or rejects if the stream emits `error`.510- `collect()` - Return a Promise that resolves on `end` with an511  array containing each chunk of data that was emitted, or512  rejects if the stream emits `error`. Note that this consumes513  the stream data.514- `concat()` - Same as `collect()`, but concatenates the data515  into a single Buffer object. Will reject the returned promise516  if the stream is in objectMode, or if it goes into objectMode517  by the end of the data.518- `read(n)` - Consume `n` bytes of data out of the buffer. If `n`519  is not provided, then consume all of it. If `n` bytes are not520  available, then it returns null. **Note** consuming streams in521  this way is less efficient, and can lead to unnecessary Buffer522  copying.523- `destroy([er])` - Destroy the stream. If an error is provided,524  then an `'error'` event is emitted. If the stream has a525  `close()` method, and has not emitted a `'close'` event yet,526  then `stream.close()` will be called. Any Promises returned by527  `.promise()`, `.collect()` or `.concat()` will be rejected.528  After being destroyed, writing to the stream will emit an529  error. No more data will be emitted if the stream is destroyed,530  even if it was previously buffered.531 532### Properties533 534- `bufferLength` Read-only. Total number of bytes buffered, or in535  the case of objectMode, the total number of objects.536- `encoding` Read-only. The encoding that has been set.537- `flowing` Read-only. Boolean indicating whether a chunk written538  to the stream will be immediately emitted.539- `emittedEnd` Read-only. Boolean indicating whether the end-ish540  events (ie, `end`, `prefinish`, `finish`) have been emitted.541  Note that listening on any end-ish event will immediateyl542  re-emit it if it has already been emitted.543- `writable` Whether the stream is writable. Default `true`. Set544  to `false` when `end()`545- `readable` Whether the stream is readable. Default `true`.546- `pipes` An array of Pipe objects referencing streams that this547  stream is piping into.548- `destroyed` A getter that indicates whether the stream was549  destroyed.550- `paused` True if the stream has been explicitly paused,551  otherwise false.552- `objectMode` Indicates whether the stream is in `objectMode`.553- `aborted` Readonly property set when the `AbortSignal`554  dispatches an `abort` event.555 556### Events557 558- `data` Emitted when there's data to read. Argument is the data559  to read. This is never emitted while not flowing. If a listener560  is attached, that will resume the stream.561- `end` Emitted when there's no more data to read. This will be562  emitted immediately for empty streams when `end()` is called.563  If a listener is attached, and `end` was already emitted, then564  it will be emitted again. All listeners are removed when `end`565  is emitted.566- `prefinish` An end-ish event that follows the same logic as567  `end` and is emitted in the same conditions where `end` is568  emitted. Emitted after `'end'`.569- `finish` An end-ish event that follows the same logic as `end`570  and is emitted in the same conditions where `end` is emitted.571  Emitted after `'prefinish'`.572- `close` An indication that an underlying resource has been573  released. Minipass does not emit this event, but will defer it574  until after `end` has been emitted, since it throws off some575  stream libraries otherwise.576- `drain` Emitted when the internal buffer empties, and it is577  again suitable to `write()` into the stream.578- `readable` Emitted when data is buffered and ready to be read579  by a consumer.580- `resume` Emitted when stream changes state from buffering to581  flowing mode. (Ie, when `resume` is called, `pipe` is called,582  or a `data` event listener is added.)583 584### Static Methods585 586- `Minipass.isStream(stream)` Returns `true` if the argument is a587  stream, and false otherwise. To be considered a stream, the588  object must be either an instance of Minipass, or an589  EventEmitter that has either a `pipe()` method, or both590  `write()` and `end()` methods. (Pretty much any stream in591  node-land will return `true` for this.)592 593## EXAMPLES594 595Here are some examples of things you can do with Minipass596streams.597 598### simple "are you done yet" promise599 600```js601mp.promise().then(602  () => {603    // stream is finished604  },605  er => {606    // stream emitted an error607  }608)609```610 611### collecting612 613```js614mp.collect().then(all => {615  // all is an array of all the data emitted616  // encoding is supported in this case, so617  // so the result will be a collection of strings if618  // an encoding is specified, or buffers/objects if not.619  //620  // In an async function, you may do621  // const data = await stream.collect()622})623```624 625### collecting into a single blob626 627This is a bit slower because it concatenates the data into one628chunk for you, but if you're going to do it yourself anyway, it's629convenient this way:630 631```js632mp.concat().then(onebigchunk => {633  // onebigchunk is a string if the stream634  // had an encoding set, or a buffer otherwise.635})636```637 638### iteration639 640You can iterate over streams synchronously or asynchronously in641platforms that support it.642 643Synchronous iteration will end when the currently available data644is consumed, even if the `end` event has not been reached. In645string and buffer mode, the data is concatenated, so unless646multiple writes are occurring in the same tick as the `read()`,647sync iteration loops will generally only have a single iteration.648 649To consume chunks in this way exactly as they have been written,650with no flattening, create the stream with the `{ objectMode:651true }` option.652 653```js654const mp = new Minipass({ objectMode: true })655mp.write('a')656mp.write('b')657for (let letter of mp) {658  console.log(letter) // a, b659}660mp.write('c')661mp.write('d')662for (let letter of mp) {663  console.log(letter) // c, d664}665mp.write('e')666mp.end()667for (let letter of mp) {668  console.log(letter) // e669}670for (let letter of mp) {671  console.log(letter) // nothing672}673```674 675Asynchronous iteration will continue until the end event is reached,676consuming all of the data.677 678```js679const mp = new Minipass({ encoding: 'utf8' })680 681// some source of some data682let i = 5683const inter = setInterval(() => {684  if (i-- > 0) mp.write(Buffer.from('foo\n', 'utf8'))685  else {686    mp.end()687    clearInterval(inter)688  }689}, 100)690 691// consume the data with asynchronous iteration692async function consume() {693  for await (let chunk of mp) {694    console.log(chunk)695  }696  return 'ok'697}698 699consume().then(res => console.log(res))700// logs `foo\n` 5 times, and then `ok`701```702 703### subclass that `console.log()`s everything written into it704 705```js706class Logger extends Minipass {707  write(chunk, encoding, callback) {708    console.log('WRITE', chunk, encoding)709    return super.write(chunk, encoding, callback)710  }711  end(chunk, encoding, callback) {712    console.log('END', chunk, encoding)713    return super.end(chunk, encoding, callback)714  }715}716 717someSource.pipe(new Logger()).pipe(someDest)718```719 720### same thing, but using an inline anonymous class721 722```js723// js classes are fun724someSource725  .pipe(726    new (class extends Minipass {727      emit(ev, ...data) {728        // let's also log events, because debugging some weird thing729        console.log('EMIT', ev)730        return super.emit(ev, ...data)731      }732      write(chunk, encoding, callback) {733        console.log('WRITE', chunk, encoding)734        return super.write(chunk, encoding, callback)735      }736      end(chunk, encoding, callback) {737        console.log('END', chunk, encoding)738        return super.end(chunk, encoding, callback)739      }740    })()741  )742  .pipe(someDest)743```744 745### subclass that defers 'end' for some reason746 747```js748class SlowEnd extends Minipass {749  emit(ev, ...args) {750    if (ev === 'end') {751      console.log('going to end, hold on a sec')752      setTimeout(() => {753        console.log('ok, ready to end now')754        super.emit('end', ...args)755      }, 100)756      return true757    } else {758      return super.emit(ev, ...args)759    }760  }761}762```763 764### transform that creates newline-delimited JSON765 766```js767class NDJSONEncode extends Minipass {768  write(obj, cb) {769    try {770      // JSON.stringify can throw, emit an error on that771      return super.write(JSON.stringify(obj) + '\n', 'utf8', cb)772    } catch (er) {773      this.emit('error', er)774    }775  }776  end(obj, cb) {777    if (typeof obj === 'function') {778      cb = obj779      obj = undefined780    }781    if (obj !== undefined) {782      this.write(obj)783    }784    return super.end(cb)785  }786}787```788 789### transform that parses newline-delimited JSON790 791```js792class NDJSONDecode extends Minipass {793  constructor(options) {794    // always be in object mode, as far as Minipass is concerned795    super({ objectMode: true })796    this._jsonBuffer = ''797  }798  write(chunk, encoding, cb) {799    if (800      typeof chunk === 'string' &&801      typeof encoding === 'string' &&802      encoding !== 'utf8'803    ) {804      chunk = Buffer.from(chunk, encoding).toString()805    } else if (Buffer.isBuffer(chunk)) {806      chunk = chunk.toString()807    }808    if (typeof encoding === 'function') {809      cb = encoding810    }811    const jsonData = (this._jsonBuffer + chunk).split('\n')812    this._jsonBuffer = jsonData.pop()813    for (let i = 0; i < jsonData.length; i++) {814      try {815        // JSON.parse can throw, emit an error on that816        super.write(JSON.parse(jsonData[i]))817      } catch (er) {818        this.emit('error', er)819        continue820      }821    }822    if (cb) cb()823  }824}825```826 
Brunobkr/llama.cpp_AlgMor24_github · Team Ai