# textConvert Architecture Overview This document provides a high-level overview of the architecture and design principles of the textConvert library — read this first if you're new to the codebase, before diving into a specific function's source. --- ## Project Structure ```text textConvert/ src/ text/ analysis/ statistics.ts # Text statistics (word/char/sentence counts, reading time) language.ts # Language detection wordFrequency.ts # Word-frequency counting validation/ email.ts # Email validation url.ts # URL validation phoneNumber.ts # Phone number validation internal/ scan.ts # Shared linear-scan helpers (not publicly exported) used by extract.ts and redact.ts detectors.ts # Shared, position-aware PII/secret detection logic (not publicly exported) used by redact.ts, extract.ts, and scan.ts graphemes.ts # Shared grapheme-cluster splitting via Intl.Segmenter (not publicly exported) used by reverse.ts and truncate.ts conventions.ts # Case conversion functions (camelCase, snakeCase, capitalize, titleCase, ...) slugify.ts # URL-safe slug generation clear.ts # Punctuation removal count.ts # Counting utilities (words, sentences, letters) reverse.ts # String reversal, grapheme-cluster aware spread.ts # Character array conversion truncate.ts # Length-limited text truncation with ellipsis, grapheme-safe cut isPalindrome.ts # Palindrome checking (built on clear.ts and reverse.ts) html.ts # HTML escaping/unescaping (escapeHtml, unescapeHtml) normalize.ts # Whitespace/line-ending normalization, diacritic stripping pluralize.ts # English pluralization randomString.ts # Cryptographically secure random string generation mask.ts # Partial string masking for display redact.ts # PII/secret detection and masking (built on mask.ts) scan.ts # PII/secret detection as structured matches (same detection logic as redact.ts, non-destructive) sanitize.ts # Configurable trim/normalize/redact/escape pipeline (composition, not new logic) extract.ts # Email/URL extraction from free-form text numbers/ numbersToWords.ts # Number to words conversion wordsToNumber.ts # Reverse of numbersToWords.ts ordinal.ts # Ordinal suffix form (21 -> '21st') ordinalToWords.ts # Ordinal word form (21 -> 'twenty-first'), built on numbersToWords.ts formatNumber.ts # Thousands-separator number formatting parseNumber.ts # Reverse of formatNumber.ts assets/ # Shared regex and constants bin/ textconvert.ts # CLI entry point (thin wiring, built as dist/cli.js) cli.ts # CLI argument parsing and command logic (testable, no direct IO side effects on import) textConvert.ts # Main entry point (exports all public functions) tests/ # Unit and integration tests docs/ # Documentation ... # Config, build, and meta files ``` --- ## Main Modules & Responsibilities - **text/analysis/**: Text statistics, language detection, word-frequency counting, and related analysis tools. - **text/validation/**: Validation functions (email, URL, phone number). - **text/internal/**: Shared logic used by more than one public function but not exported from the package itself — the linear-scan helpers (`scan.ts`), the position-aware PII/secret detection logic (`detectors.ts`) behind `extract.ts`, `redact.ts`, and `scan.ts` (the public one), and the grapheme-cluster splitting (`graphemes.ts`) behind `reverse.ts` and `truncate.ts`. - **text/conventions.ts**: Case conversion (camelCase, PascalCase, snake_case, kebab-case, capitalize, titleCase). - **text/slugify.ts**: URL-safe slug generation with Unicode accent normalization. - **text/clear.ts**: Remove punctuation and clean text. - **text/count.ts**: Count words, sentences, and letters. - **text/reverse.ts**: Reverse strings, grapheme-cluster aware (emoji, combining marks) via `text/internal/graphemes.ts`. - **text/spread.ts**: Convert strings to character arrays. - **text/truncate.ts**: Shorten text to a max length with an ellipsis, cutting on grapheme-cluster boundaries via `text/internal/graphemes.ts`. - **text/isPalindrome.ts**: Check whether text reads the same forwards and backwards, built on `clear.ts` and `reverse.ts`. - **text/html.ts**: Escape/unescape the five HTML special characters per OWASP's XSS Prevention Cheat Sheet Rule #1. - **text/normalize.ts**: Strip diacritics, collapse whitespace runs, and normalize CRLF/CR line endings to LF. - **text/pluralize.ts**: Return the plural form of an English word (regular suffix rules plus a maintained irregulars/uncountables list). - **text/randomString.ts**: Generate a cryptographically secure random string via `globalThis.crypto`, for IDs, tokens, or test fixtures. - **text/mask.ts**: Partially mask a string for display (the primitive `redact.ts` is built on). - **text/redact.ts**: Detect and mask PII (email, phone, credit card, public IPv4) and secrets (API keys, JWTs) in free-form text. - **text/scan.ts**: Same PII/secret detection as `redact.ts` (built on the same shared `text/internal/detectors.ts` logic), returning structured `{ type, value, start, end }` matches instead of masking them. - **text/sanitize.ts**: A configurable `trim`/`normalizeWhitespace`/`redactPII`/`escapeHtml` pipeline in a fixed step order — composition of the functions above, not new detection logic. - **text/extract.ts**: Find every email/URL embedded in a block of text. - **numbers/numbersToWords.ts**: Convert numbers to English words. - **numbers/wordsToNumber.ts**: Parse English number-words back into a number — the reverse of `numbersToWords.ts`, reusing its word lists. - **numbers/ordinal.ts**: Get a non-negative integer's ordinal suffix form (`21` -> `'21st'`). - **numbers/ordinalToWords.ts**: Get a non-negative integer's ordinal word form (`21` -> `'twenty-first'`), built on `numbersToWords.ts`'s cardinal output. - **numbers/formatNumber.ts**: Add thousands separators to a number, English/US style. - **numbers/parseNumber.ts**: Parse a formatted number string back into a number — the reverse of `formatNumber.ts`. - **assets/regex.ts**: Centralized regex patterns for reuse. - **bin/textconvert.ts** and **cli.ts**: The `npx textconvert` CLI (currently just `redact`) — `cli.ts` holds the testable argument-parsing/dispatch logic, `bin/textconvert.ts` is the thin entry point that wires it to real `process`/stdio and gets built into `dist/cli.js`. This is the one part of `src/` that legitimately depends on Node built-ins; everything else stays runtime-agnostic by design. - **textConvert.ts**: Aggregates and exports all public functions for library consumers. --- ## Design Principles - **Single Responsibility**: Each function/module does one thing well. - **Pure Functions**: Most utilities are pure, with no side effects. - **TypeScript First**: All code is written in TypeScript for safety and clarity. - **Test Coverage**: Every logic function has corresponding tests. - **Extensibility**: New functions can be added easily (see `docs/ADDING_FUNCTION.md`). - **No External Dependencies**: The library is dependency-free for maximum portability. - **ReDoS-Safe Scanning**: Functions that scan free-form text for candidate matches (`extract.ts`, `redact.ts`, `countSentences`) use linear character-by-character scans instead of backtracking-prone regexes, so they can't be driven into quadratic-time behavior by adversarial input. `text/internal/scan.ts` centralizes this pattern where more than one function needs it. --- ## Extension Points - **Adding New Functions**: Follow the guide in `docs/ADDING_FUNCTION.md`. - **Adding New Validators**: Place in `text/validation/`, export via `textConvert.ts`, and document. - **Adding New Analysis Tools**: Place in `text/analysis/`, export and document as above. - **Adding New Text Scanners**: If it scans free-form text for candidate substrings (like `extract.ts`/`redact.ts` do), check `text/internal/scan.ts` first — reuse its helpers instead of writing another backtracking regex. If it's a new PII/secret type specifically, add it to `text/internal/detectors.ts` so both `redact.ts` and `scan.ts` pick it up from one place. --- ## Build & Distribution - **Bundling**: Uses Rollup to produce a single ESM bundle (`dist/textConvert.mjs`); ESM-only since v3.0.0, no CJS output. - **Type Declarations**: `tsc` (with `emitDeclarationOnly`) generates the `.d.ts` tree consumers and Rollup's TypeScript plugin both rely on; Rollup itself produces the actual JS output. - **Testing**: Uses Vitest for fast, modern testing. - **Registries**: Published to npm, with a mirror publish to [JSR](https://jsr.io) on the same release. --- ## Future Directions - Expanding `detectLanguage`'s supported language set beyond the current seven. - IPv6 support for `redact`'s `'ip'` type -- deferred from #363, since its textual representation (zero-compression, embedded IPv4-mapped forms, zone IDs) is a materially larger parsing problem than IPv4. - More CLI commands beyond `redact`, if there's demand. --- For more details, see the README and other docs. Contributions and suggestions are welcome!