Class: SentenceSplitter

SentenceSplitter is our default text splitter that supports splitting into sentences, paragraphs, or fixed length chunks with overlap.

One of the advantages of SentenceSplitter is that even in the fixed length chunks it will try to keep sentences together.

Constructors

new SentenceSplitter()

new SentenceSplitter(options?): SentenceSplitter

Parameters

• options?

• options.chunkOverlap?: number

• options.chunkSize?: number

• options.chunkingTokenizerFn?

• options.paragraphSeparator?: string | RegExp

• options.splitLongSentences?: boolean

• options.tokenizer?: any

• options.tokenizerDecoder?: any

Returns

SentenceSplitter

Source

packages/core/src/TextSplitter.ts:75

Properties

chunkOverlap

chunkOverlap: number

Source

packages/core/src/TextSplitter.ts:67

---

chunkSize

chunkSize: number

Source

packages/core/src/TextSplitter.ts:66

---

chunkingTokenizerFn()

private chunkingTokenizerFn: (text) => string[]

Parameters

• text: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:72

---

paragraphSeparator

private paragraphSeparator: string | RegExp

Source

packages/core/src/TextSplitter.ts:71

---

splitLongSentences

private splitLongSentences: boolean

Source

packages/core/src/TextSplitter.ts:73

---

tokenizer

private tokenizer: any

Source

packages/core/src/TextSplitter.ts:69

---

tokenizerDecoder

private tokenizerDecoder: any

Source

packages/core/src/TextSplitter.ts:70

Methods

combineTextSplits()

combineTextSplits(newSentenceSplits, effectiveChunkSize): TextSplit[]

Parameters

• newSentenceSplits: SplitRep[]

• effectiveChunkSize: number

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:216

---

getEffectiveChunkSize()

private getEffectiveChunkSize(extraInfoStr?): number

Parameters

• extraInfoStr?: string

Returns

number

Source

packages/core/src/TextSplitter.ts:111

---

getParagraphSplits()

getParagraphSplits(text, effectiveChunkSize?): string[]

Parameters

• text: string

• effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:128

---

getSentenceSplits()

getSentenceSplits(text, effectiveChunkSize?): string[]

Parameters

• text: string

• effectiveChunkSize?: number

Returns

string[]

Source

packages/core/src/TextSplitter.ts:158

---

processSentenceSplits()

private processSentenceSplits(sentenceSplits, effectiveChunkSize): SplitRep[]

Splits sentences into chunks if necessary.

This isn't great behavior because it can split down the middle of a word or in non-English split down the middle of a Unicode codepoint so the splitting is turned off by default. If you need it, please set the splitLongSentences option to true.

Parameters

• sentenceSplits: string[]

• effectiveChunkSize: number

Returns

SplitRep[]

Source

packages/core/src/TextSplitter.ts:187

---

splitText()

splitText(text, extraInfoStr?): string[]

Parameters

• text: string

• extraInfoStr?: string

Returns

string[]

Source

packages/core/src/TextSplitter.ts:310

---

splitTextWithOverlaps()

splitTextWithOverlaps(text, extraInfoStr?): TextSplit[]

Parameters

• text: string

• extraInfoStr?: string

Returns

TextSplit[]

Source

packages/core/src/TextSplitter.ts:282