Chunking
Read · 1 of 2
Why overlap?
Chunks that are too big drown the useful sentence in noise; too small and they lose context.
Cut at fixed sizes and an idea can straddle the cut. A small overlap (repeating the last few words of one chunk at the start of the next) keeps it whole in at least one chunk.
Read this lesson as text
Small pieces with a little overlap, so no idea is cut in half.
Why overlap?
Chunks that are too big drown the useful sentence in noise; too small and they lose context.
Cut at fixed sizes and an idea can straddle the cut. A small overlap (repeating the last few words of one chunk at the start of the next) keeps it whole in at least one chunk.
Exercise: Chunk with overlap
Input: the chunk size S and the overlap O (in words, O < S), then one line of text.
Make chunks of S words; each new chunk starts O words before the previous one ended. Stop once a chunk reaches the last word.
Print each chunk as [i] words… starting from 1.