Full-Text Search (FTS) Index

Create and tune BM25-based full-text search indexes in LanceDB.

LanceDB Cloud and Enterprise provide performant full-text search based on BM25, allowing you to incorporate keyword-based search in your retrieval solutions.

The create_fts_index API returns immediately, but index building happens asynchronously.

Creating FTS Indexes

Synchronous API

Use create_fts_index with synchronous LanceDB connections:

Check FTS index status using the API:

Asynchronous API

When using async connections (connect_async), use create_index with the FTS configuration:

The create_fts_index method is not available on AsyncTable. Use create_index with FTS config instead.

Configuration Options

FTS Parameters

Parameter	Type	Default	Description
with_position	bool	False	Store token positions (required for phrase queries)
base_tokenizer	str	"simple"	Text splitting method (simple, whitespace, or raw)
language	str	"English"	Language for stemming/stop words
max_token_length	int	40	Maximum token size; longer tokens are omitted
lower_case	bool	True	Lowercase tokens
stem	bool	True	Apply stemming (running → run)
remove_stop_words	bool	True	Drop common stop words
ascii_folding	bool	True	Normalize accented characters

• max_token_length can filter out base64 blobs or long URLs.
• Disabling with_position reduces index size but disables phrase queries.
• ascii_folding helps with international text (e.g., “café” → “cafe”).

Phrase Query Configuration

Enable phrase queries by setting:

Parameter	Required Value	Purpose
with_position	True	Track token positions for phrase matching
remove_stop_words	False	Preserve stop words for exact phrase matching