New blogMaking an AI Agent You Can Trust With Money
logo

★ Token-Based chunking, visualized

Token-Based Chunking for RAG

Token-based chunking is a fixed window measured in tokens instead of characters. The text is tokenized, cut every N tokens with an overlap, and each window is decoded back to text.

  • Runs in your browser
  • No signup

1,181 characters · loading tokenizer… · nothing leaves your browser

tokens
tokens

Counts tokens with the o200k_base tokenizer.

Fixed (tokens) · 80 tokens · overlap 10

5 chunks

67
avg tokens
15 / 80
min / max
40
overlap tokens
$6.70
embed 1M docs

Loading the tokenizer; token counts are estimates until it's ready.

# Refund policy for annual plans Annual plans are billed once a year, up front. This page explains when you can get money back and how long it takes. ## Eligibility You can request a full refund within 30 days of the first annual charge. After 30 days, refunds are prorated for the unused months, minus any discount you received for paying annually. Renewals follow the same rule: the 30-day window starts again on each renewal date. ## How to request a refund Open Settings, then Billing, and choose "Request refund". Pick the reason that fits best. If you are cancelling because of a bug, include the error message; it often gets fixed faster than the refund is processed. ### Processing times Card refunds usually appear within 5 to 10 business days, depending on your bank. Bank transfers can take longer. We send an email when the refund leaves our side. ## Exceptions Refunds are not available for usage-based charges such as extra API calls or storage, because those costs are incurred when you use them. Accounts closed for abuse are not eligible. ## Contact Still stuck? Email the billing team with your account ID and the last four digits of the card on file.

one chunkoverlapClick a chunk to inspect it.

Chunk inspector

Click any coloured chunk in the document to see its text, size, heading path and what it shares with its neighbours.

Chunk sizes

Tokens per chunk (estimated). A tight cluster means predictable chunks.

115294357471chunk size (tokens), bucket start

Embedding cost assumes text-embedding-3-small at $0.02 per 1M tokens (OpenAI, checked 8 Nov 2026), counting overlap twice.

Export config

Copy-ready code for the strategy and settings above. Comments flag where a library differs.

from langchain_text_splitters import TokenTextSplitter

splitter = TokenTextSplitter(encoding_name="o200k_base", chunk_size=80, chunk_overlap=10)
chunks = splitter.split_text(text)

Your retrieval is only as good as your chunks.

I build RAG systems with evals, so you know when the answers get better or worse.

Fix my RAG
01

How it works

Token-based chunking is a fixed window measured in tokens instead of characters. The text is tokenized, cut every N tokens with an overlap, and each window is decoded back to text.

This visualizer uses the o200k_base tokenizer. Your embedding model may tokenize differently, so treat the counts as close, not exact, unless you use the same tokenizer.

02

When to use it

Good fit

  • When you must stay under an embedding model's token limit with no surprises.
  • When cost per chunk needs to be predictable.
  • Mixed-language or code-heavy text, where characters per token vary a lot.

Watch out for

  • Still blind to structure: it cuts sentences and code wherever the window ends.
  • Tokenizer mismatch: counting with one tokenizer and embedding with another drifts.
  • Very small windows lose context; very large ones dilute the embedding.
03

Where to start

A few hundred tokens with an overlap of around a tenth of that is a common starting point. Tune it with an evaluation set rather than by feel.

Then test it: write down twenty real questions, check which chunks come back for each, and compare strategies on the same set. The main visualizer lets you put two strategies side by side.

04

Questions engineers actually ask

Which tokenizer should I count with?
The one your embedding model uses. OpenAI's current embedding models use the same family of tokenizers as this tool; other providers differ.
Does overlap help retrieval?
It helps when answers straddle a boundary. It also increases storage and can return near-duplicate chunks, so measure it.
Is token-based better than recursive?
It's better at respecting limits. Recursive is better at respecting meaning. Many pipelines use recursive splitting with a token length function to get both.
What chunk size should I use for RAG?
There is no universal number. A few hundred tokens with a small overlap is a common starting point, but the right size depends on your documents and questions. Build a small evaluation set and compare sizes on it.
Does my text leave the browser?
No. Splitting, tokenizing and the visualization all run in your browser. Nothing you paste is sent anywhere.
05

Other strategies

06

More from the workbench

07 · AI & LLMsToken CounterToken counts per model, cost per call and how much of the context window you are using.02 · AI & LLMsLLM API Cost CalculatorYour monthly bill across Claude, GPT, Gemini and open models, side by side.13 · AI & LLMsAI Agent vs HireThe fully loaded cost of a hire versus an AI agent doing the same work, review included.

Fine print

Tools: on the house.
Engineer: by the hour.

The visualizer was free. Production RAG with evals is what I build.

I'm Himanshu, an AI and full-stack engineer. I build MVPs, AI agents and the growth plumbing around them for early-stage startups.

Hire me →My projects