Implementation of splitter which looks at tokens.

Hierarchy

Implements

Constructors

Properties

allowedSpecial: string[] | "all"
chunkOverlap: number = 200
chunkSize: number = 1000
disallowedSpecial: string[] | "all"
encodingName: TiktokenEncoding
keepSeparator: boolean = false
lengthFunction: ((text) => number) | ((text) => Promise<number>)

Type declaration

    • (text): number
    • Parameters

      • text: string

      Returns number

Type declaration

    • (text): Promise<number>
    • Parameters

      • text: string

      Returns Promise<number>

Methods

Generated using TypeDoc