Semantic search and an AI assistant for your content

We've built two Statamic addons that layer Retrieval-Augmented Generation (RAG) onto CMS content:

• byte5/ai-entry-embeddings: extracts text from entries and generates vector embeddings on every save.
  
  
• byte5/ai-entries-assistant: a conversational assistant in the Control Panel that answers questions exclusively from that content.

In this post, we show how both addons work, how you configure them, and what value they create for editors and visitors.

Classic keyword search compares strings, not meaning. An example: a visitor searches for "cancel contract," but your site talks about "end subscription," and the keyword search returns nothing.

An embedding is a numerical vector representation of a piece of text, placed in a vector space so that semantically similar texts lie close together. Instead of comparing words, you measure the distance between vectors, and that turns the two phrasings into neighbors.

Statamic entries aren't flat text. A page might be a Bard field, a nested Replicator, a Grid, Markdown, or a simple text field. On top of that comes presentation noise like HTML tags, CSS classes, alignment flags, and IDs. Feed this raw data straight into an embedding model, and the results are correspondingly poor.

1. An entry is saved. Statamic fires the EntrySaved event.

2. A listener checks whether the entry's collection is configured and dispatches a queue job.

3. An extraction pipeline runs through the configured fields and picks a suitable extractor for each field type.

4. Each extractor returns one or more ContentChunk objects.

5. The chunks replace the entry's previous chunks in PostgreSQL.

6. A second queue job generates the embeddings.

For every field type there's a dedicated extractor that knows how to pull meaningful text out of the field:

• text / textarea take the raw string value.
  
  
• markdown is rendered to HTML and then reduced to plain text.
  
  
• bard splits body text and sets into separate chunks.
  
  
• replicator creates one chunk per set, including nested fields.
  
  
• grid creates one chunk per row, including nested columns.
  
  
• select takes the human-readable option label(s).

A ContentChunk carries with it:

• the text itself,
• the field handle it came from,
• a path in dot notation, e.g. page_builder.pricing_block.0,
• structured metadata (set type, index, etc.).

Because every chunk records which section of which entry it came from, the assistant can name the exact block that answered a question, not just the page.

For fields too irregular for rule-based parsing, you can use ExtractFieldWithAi. The raw field data is sent to a small, cheap LLM agent. It's instructed to extract only human-readable text, to ignore tags, classes, flags, and technical identifiers, and to never translate or summarize. Via structured output (a JSON schema), the result is always a clean array of chunks.

The feature is opt-in per field or per field type and costs one API call per entry.

Chunks are stored in PostgreSQL using the pgvector extension. The vectors live in the same database, so you don't have to run a separate vector store. Two tables are involved:

• ai_entry_embeddings: one row per entry, with status (extracting, generating, generated, failed) and chunk counts.
• ai_entry_embedding_chunks: one row per chunk, with text, path, metadata, and an indexed vector column.

After extraction, GenerateEntryEmbeddingsJob processes the chunk texts in batches via the Laravel AI SDK's embeddings API, writes the vectors back, and sets the entry to generated. If the provider is unreachable, the job retries with backoff and otherwise marks the entry as failed.

The entire flow is asynchronous and event-driven: your editors save as usual, and the embedding happens in the background on the queue. Deleting an entry cleans up its embeddings automatically (configurable).

By default, nothing is embedded. You list collections and fields explicitly:

This is a deliberate security decision: it keeps internal notes, admin-only fields, and sensitive data away from the AI provider. Further options can be configured, such as the embedding dimensions (default 1536, which must match the model), whether drafts are skipped, and the queue connection and name.

The Control Panel adds an "AI Tools" section with entry and chunk counts plus status per collection, secured behind a permission.

A chat in the Control Panel that answers questions about your content.

1. A user sends a message. It's saved and UserMessageAddedToConversation is dispatched.

2. A queue listener calls the EntriesAssistantAgent.

3. The agent always calls a similarity-search tool first: a pgvector nearest-neighbor query that returns the top 25 chunks.

4. The agent composes a Markdown answer, exclusively from those results. No general knowledge, no internet, no making things up. If nothing relevant is found, it says so.

5. The answer is saved as an assistant message and pushed to the frontend.

That's the core of RAG: the model doesn't answer from its training data but retrieves your content and reasons over it. This keeps answers grounded, up to date, and specific to your website, and reduces hallucinations.

• Provider-agnostic: Anthropic, OpenAI, Gemini, Mistral, Ollama, and others, switchable via config.

• Conversational: previous messages are passed to the model again each round, so follow-up questions keep their context.

• Answers run on a queue and don't block the request (generation takes a few seconds).

• Broadcasting via Laravel Echo: the answer is pushed to a private channel conversation.{id} the moment it's ready. Each user can only subscribe to their own conversations.

• Polling fallback: if you don't run a websocket server, you let the frontend poll instead, with no additional infrastructure at all.

• Broadcasting is off by default and can be enabled via an env flag.
  The assistant lives under "AI Tools" in the Control Panel, behind its own permission.

• Semantic on-site search: the embedding layer can power a frontend search where visitors find what they mean, not just what they type. Relevant for knowledge bases, documentation, product catalogs, and large editorial archives, and it means fewer "not found" support tickets.
  
  
• Trustworthy self-service support: the assistant only answers from your content and declines when it doesn't know something. So there are no invented policies or prices. The CMS stays the single source of truth, with no separate knowledge base to sync.
  
  
• Faster editorial work: questions like "What have we published about X?" or "Where do we mention the old prices?" return the exact entries and sections instead of manual searching.
  
  
• Low operational overhead: runs in your existing stack (Statamic, Laravel, PostgreSQL). pgvector means no separate vector database to license, host, and monitor, and the embedding runs on your existing queue workers. The provider is swappable, so you can optimize for cost or quality.
  
  
• Secure by default: explicit allow-listing of fields, exclusion of drafts, permission gating, and "answer only from the retrieved content" are built in.

Both addons are open source and available on GitHub: ai-entry-embeddings and ai-entries-assistant. Take a look, give them a try, and we welcome issues, pull requests, and feedback.