Class: SentenceSplitter
SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.
One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.
Constructors
constructor
• new SentenceSplitter(options?
)
Parameters
Name | Type |
---|---|
options? | Object |
options.chunkOverlap? | number |
options.chunkSize? | number |
options.chunkingTokenizerFn? | (text : string ) => null | RegExpMatchArray |
options.paragraphSeparator? | string |
options.splitLongSentences? | boolean |
options.tokenizer? | any |
options.tokenizerDecoder? | any |
Defined in
packages/core/src/TextSplitter.ts:67
Properties
chunkOverlap
• Private
chunkOverlap: number
Defined in
packages/core/src/TextSplitter.ts:60
chunkSize
• Private
chunkSize: number
Defined in
packages/core/src/TextSplitter.ts:59
chunkingTokenizerFn
• Private
chunkingTokenizerFn: (text
: string
) => null
| RegExpMatchArray
Type declaration
▸ (text
): null
| RegExpMatchArray
Parameters
Name | Type |
---|---|
text | string |
Returns
null
| RegExpMatchArray
Defined in
packages/core/src/TextSplitter.ts:64
paragraphSeparator
• Private
paragraphSeparator: string
Defined in
packages/core/src/TextSplitter.ts:63
splitLongSentences
• Private
splitLongSentences: boolean
Defined in
packages/core/src/TextSplitter.ts:65