Open-source implementation guide
How to split text by character limit in JavaScript
To split text by character limit in JavaScript without relying on raw substrings, Social Threader applies deterministic word, sentence, paragraph, and enumeration rules. The implementation is a browser ES module with table-driven and browser-level tests.
Quick verdict Treat maximum length as the invariant and readable boundaries as preferences. Reserve enumeration space before final output, and keep a hard-cut fallback for tokens that cannot fit.
Primary audienceWeb developers
Core outcomeAuditable chunking logic
Why substring slicing is not enough
A fixed slice(0, maximumLength) satisfies a
numeric limit but can break a word, discard useful
punctuation boundaries, or ignore the characters added by
post numbering. A practical algorithm needs a strict output
invariant and an ordered set of preferred fallbacks.
Social Threader normalizes line separators and whitespace, identifies paragraphs, tokenizes words with their trailing punctuation, optionally groups sentences, and then combines values while the next candidate remains inside the maximum.
JavaScript split-text-by-character-limit flow
- Normalize line endings, non-breaking spaces, tabs, and whitespace-only paragraph separators.
- Split text into word tokens while keeping punctuation attached to the preceding token.
- Optionally group tokens into heuristic sentences and process paragraphs independently.
- Build chunks within the effective maximum, then repeat with reduced space until enumeration overhead stabilizes.
This public function is copied directly from
js/core/chunking.js:
function getChunks(rawText, options) {
if (!options.enumerate) {
return buildBaseChunks(rawText, options);
}
let effectiveMaximumLength = Math.max(1, options.maximumLength);
/** @type {string[]} */
let baseChunks = [];
while (true) {
const iterationOptions = Object.assign({}, options, { maximumLength: effectiveMaximumLength });
baseChunks = buildBaseChunks(rawText, iterationOptions);
if (baseChunks.length === 0) {
return [];
}
const enumerationOverhead = getMaximumEnumerationOverhead(baseChunks.length);
const nextEffectiveMaximumLength = Math.max(1, options.maximumLength - enumerationOverhead);
if (nextEffectiveMaximumLength === effectiveMaximumLength) {
break;
}
effectiveMaximumLength = nextEffectiveMaximumLength;
}
return baseChunks.map((chunkText, index) => enumerateChunk(chunkText, index, baseChunks.length));
}
The source exports this behavior through the frozen
chunkingService object. It is a repository
module, not a separately documented package API.
Boundary precedence
| Boundary | When it is used | Fallback behavior |
|---|---|---|
| Paragraph | When breakOnParagraphs is enabled. |
Each paragraph recursively uses the remaining rules. |
| Sentence | When breakOnSentences is enabled and the sentence fits. |
An over-limit sentence proceeds to length-based splitting. |
| Space or punctuation | When a candidate must be cut inside the maximum. | The search moves backward from the maximum. |
| Hard character cut | When no usable boundary exists. | The remaining text continues through the same loop. |
Options and invariants
| Option | Effect | Invariant |
|---|---|---|
maximumLength |
Sets the final per-chunk character ceiling. | The implementation clamps the working value to at least 1. |
breakOnSentences |
Groups heuristic sentences before combining them. | It cannot keep an over-limit sentence intact. |
breakOnParagraphs |
Processes extracted paragraphs separately. | Nested processing disables paragraph recursion. |
enumerate |
Adds (current/total) suffixes. |
Suffix characters are reserved inside the original maximum. |
Tested edge cases
Abbreviations
Strict and flexible abbreviation sets cover documented
examples such as honorifics, months, e.g.,
and p.m..
Decimals and ordinals
Decimal-like tokens avoid false sentence boundaries, while early ordinal-looking tokens receive special handling.
Ellipses and wrappers
Sentence classification considers ellipses, the next significant character, and trailing quote or bracket characters.
Paragraphs and list markers
Tests cover blank-line variants, whitespace-wrapped separators, and documented list-marker transitions.
Browse the table-driven chunking tests and Puppeteer regression suite for executable examples.
Objections and limitations
Can I import this from npm?
The repository has no published library package contract. The web app imports the local ES module, and a project script synchronizes canonical web sources into the mobile tree.
Is sentence recognition linguistically complete?
No. It is a deterministic heuristic supported by the repository’s documented tests. Extend the public behavior and black-box cases before claiming broader punctuation or language coverage.
Does the module preserve input bytes exactly?
No byte-exact contract is documented. The implementation normalizes line separators, tabs, non-breaking spaces, and repeated whitespace as part of text preparation.
Frequently asked questions
Is Social Threader's chunking engine a published npm package?
No. It is a browser ES module in the Social Threader repository and is copied into the mobile source tree by the project's sync script. This guide does not present it as a versioned third-party package.
What happens when one word is longer than the character limit?
When no usable space or punctuation boundary exists inside the maximum, the documented chunkByLength function uses a hard character cut and continues with the remaining text.
How does the JavaScript preserve sentence boundaries?
When enabled, the engine groups words with a punctuation and abbreviation heuristic, then combines complete sentences while the combined value remains within the selected maximum.
Why does enumeration require repeated calculation?
Adding suffixes reduces available text space and can increase the chunk count, which can change the suffix width. The engine repeats until the effective maximum stops changing.
Does the algorithm understand the meaning of the text?
No. It uses deterministic whitespace, punctuation, abbreviation, paragraph, and length rules. It does not use a language model or claim semantic understanding.
Related Social Threader guides
Inspect the implementation, then test it in the browser
Read the public ES module and its black-box tests, or paste a draft into Social Threader to see the same options in the working interface.