File Watching

Keep the grepai index up to date with grepai watch, run it in the background, or update it once with grepai index.

grepai watch keeps your index in sync with your code. It runs an initial scan, then watches the filesystem and re-indexes files as they change. Use grepai index instead when you only need a one-time update.

Quick Start

# Create .grepai/config.yaml (once per project)
grepai init

# Index, then keep watching for changes (Ctrl+C or q to stop)
grepai watch

In an interactive terminal, grepai watch opens a monitoring UI. Use --no-ui, or pipe the output, to get plain text logs instead.

To keep it running without a terminal, see Background Daemon Mode.

Features

• Initial scan: brings the index in line with the files on disk at startup.
• Real-time updates: re-indexes created and modified files, removes deleted ones.
• Incremental: unchanged files are not re-embedded.
• Symbol extraction: builds the call graph data used by grepai trace.
• Debouncing: groups rapid changes before processing them.
• Linked worktrees: also watches linked git worktrees by default. See Git Worktrees.

One-Time Indexing

grepai index runs the same initial synchronization as grepai watch (vector, symbol and, if enabled, RPG indexes), then exits. It does not start a watcher or daemon. Unchanged files are scanned but not re-embedded.

# Update the current project, then exit
grepai index

# Update every project in a workspace, then exit
grepai index --workspace myworkspace

grepai index and grepai watch both take the project’s writer lock, so grepai index fails while a watcher is running on the same project.

Use it in Git hooks when a long-running watcher would pick up temporary files created by tests or build tools. For example, .git/hooks/pre-commit:

#!/bin/sh
grepai index

grepai index indexes the current working tree, including unstaged changes. It does not index only the staged snapshot.

To set up and index a new linked worktree without leaving a watcher behind:

git worktree add ../my-feature feature/my-feature
(cd ../my-feature && grepai index)

When the main worktree has a .grepai/ directory, the linked worktree gets its own copy automatically. Each linked worktree keeps its own index, and a one-time index does not update other worktrees.

How It Works

+------------------+     +------------------+     +------------------+
|   Initial Scan   | --> |   File Watcher   | --> |  Index Update    |
|  (sync with disk)|     |   (fsnotify)     |     |  (per file)      |
+------------------+     +------------------+     +------------------+
                                |
                                v
                        +------------------+
                        |   Debouncing     |
                        |   (500ms)        |
                        +------------------+

1. Initial scan: lists indexable files, removes index entries for files that no longer exist, and decides which files need (re)indexing (see below).
2. Embedding: new and changed files are chunked and embedded. With the openai provider, chunks from many files are sent in batches, using embedder.parallelism workers (default 4). Other providers embed file by file.
3. Symbol index: symbols are extracted for languages listed in trace.enabled_languages.
4. RPG (optional): when rpg.enabled: true, the RPG graph is built after symbol extraction.
5. File watching: filesystem events (create, modify, delete, rename) are collected and debounced (watch.debounce_ms, default 500).
6. Per-file update: each changed file is re-scanned. If its content hash is unchanged, it is skipped. Otherwise its chunks are replaced, so no duplicate vectors are left behind.

Fast restarts with last_index_time

The initial scan avoids reading every file on restart:

• A file whose modification time is at or before watch.last_index_time and that already has chunks in the index is skipped without being read.
• Other files are read and hashed. Files with an unchanged hash are not re-embedded.
• Files with no chunks yet are always re-indexed, whatever their modification time.

grepai stores watch.last_index_time in .grepai/config.yaml. It is set to the start time of the initial scan when the scan indexed something (or when no value was stored yet). While watching, it advances after each indexed file and is written to disk at most every 30 seconds and on clean shutdown. You do not need to edit it. Remove it to force every file to be re-hashed on the next start.

What Gets Indexed

A file is indexed when it is not ignored and has a supported extension. The built-in list covers common source, markup and config formats, including:

To index more extensions, add them under chunking.custom_extensions:

chunking:
  custom_extensions: [".tengo", ".el"]

What Gets Skipped

Files are skipped when they match any of these:

1. .gitignore files: at the root and in subdirectories.
2. .grepaiignore files: same syntax as .gitignore, and can re-include paths with !.
3. ignore in .grepai/config.yaml: each entry is compared to the file or directory name (not a glob or path).
4. external_gitignore: an extra gitignore file, such as your global one.
5. Minified files: names ending in .min.js, .min.css, .bundle.js or .bundle.css.
6. Large files: over 1 MB.
7. Binary files: content that is not valid UTF-8 or contains null bytes.

Default ignore list written by grepai init:

ignore:
  - .git
  - .grepai
  - node_modules
  - vendor
  - bin
  - dist
  - __pycache__
  - .venv
  - venv
  - .idea
  - .vscode
  - target
  - .zig-cache
  - zig-out
  - qdrant_storage

Real-Time Updates

With --no-ui, each change is logged as it is processed:

2026/10/25 14:02:11 Indexed src/auth/handler.go (4 chunks)
2026/10/25 14:02:11 Extracted 12 symbols from src/auth/handler.go
2026/10/25 14:02:30 Skipped unchanged src/api/routes.go
2026/10/25 14:02:45 Removed src/old/deprecated.go from index

Editors and coding agents often save by writing a temporary file and renaming it over the original. When a delete or rename event arrives but the file is still on disk, grepai treats it as a modification instead of removing the file from the index.

Symbol Indexing

The watcher also builds a symbol index for call graph analysis:

• Symbols: functions, methods, classes and types.
• References: function calls between symbols.
• Languages: those listed in trace.enabled_languages.
• Stored in: .grepai/symbols.gob.

See Call Graph Analysis for more details.

Configuration

Watcher settings in .grepai/config.yaml:

watch:
  debounce_ms: 500
  # Auto-discover and watch linked git worktrees (default: true)
  discover_worktrees: true
  # Only used when rpg.enabled is true
  rpg_derived_debounce_ms: 300
  rpg_persist_interval_ms: 1000
  rpg_full_reconcile_interval_sec: 300
  rpg_max_dirty_files_per_batch: 128
  # Managed by grepai, see "Fast restarts with last_index_time"
  # last_index_time: 2026-10-25T14:00:00+02:00

To watch only the main worktree, set discover_worktrees: false in the main worktree’s config. See Git Worktrees.

Persistence

• Periodic save: the vector and symbol indexes are saved every 30 seconds while watching.
• Shutdown save: they are saved again on Ctrl+C, q in the UI, SIGTERM or grepai watch --stop.
• Location: .grepai/index.gob and .grepai/symbols.gob with the GOB backend. PostgreSQL and Qdrant store vectors in the database; symbols stay in .grepai/symbols.gob.

Only one process may write a project’s index at a time. grepai watch (foreground, background or workspace) and grepai index all take a lock on .grepai/writer.lock. A second writer exits immediately with an error like cannot write project ...: another writer is active. The lock is released when the process exits, including after a crash.

grepai search, grepai trace and the MCP server only read the index, so they run normally while a watcher is active.

Fatal watcher errors

If grepai cannot keep full coverage of the project, it stops instead of serving a stale index. This happens when a directory cannot be watched (at startup or when a new directory appears), when fsnotify stops or reports an error (for example an event overflow), or when the project root is removed or renamed. In that case:

• New index updates are rejected, and in-flight work is canceled.
• Derived indexes are not saved, because their state may be wrong.
• The next grepai watch or grepai index re-runs the initial scan, which reconciles the index with the disk.

In workspace and multi-worktree mode, one fatal error stops the whole watcher.

Background Daemon Mode

Run the watcher as a background process:

# Start the watcher in background
grepai watch --background

# Check if the watcher is running
grepai watch --status

# Stop the background watcher gracefully
grepai watch --stop

--background, --status and --stop cannot be combined. The background process always logs plain text (no UI).

Starting the Daemon

$ grepai watch --background
Background watcher started (PID 12345)
Logs: /Users/you/Library/Logs/grepai/grepai-worktree-0adda787cae4.log
Worktree ID: 0adda787cae4

Use 'grepai watch --status' to check status
Use 'grepai watch --stop' to stop the watcher

The command waits up to 30 seconds (60 for a workspace) for the background process to finish its initial scan and watch every directory. If the startup checks passed (config, embedder, store) but the initial index takes longer, the command returns with initial indexing still in progress and the watcher keeps running: follow it with grepai watch --status or the log file. The command fails, and points to the log file, only if the process exits early or never gets through its startup checks.

Inside a git repository, the log, PID and ready files are named after the worktree ID (grepai-worktree-<id>.*), so each worktree can have its own daemon. Outside git, they are named grepai-watch.*.

Checking Status

$ grepai watch --status
Status: running
PID: 12345
Log directory: /Users/you/Library/Logs/grepai
Log file: /Users/you/Library/Logs/grepai/grepai-worktree-0adda787cae4.log
Worktree ID: 0adda787cae4

When no daemon is running, it prints Status: not running and the log directory.

Stopping the Daemon

$ grepai watch --stop
Stopping background watcher (PID 12345)...
Background watcher stopped

The daemon saves the index before exiting. --stop waits up to 30 seconds and prints Waiting for graceful shutdown... every 5 seconds meanwhile. If nothing is running, it prints No background watcher is running.

Log Locations

Logs are stored in OS-specific directories:

Background log lines start with [grepai-watch] and a timestamp with microseconds.

Use --log-dir to override the directory:

grepai watch --background --log-dir /custom/path
grepai watch --status --log-dir /custom/path
grepai watch --stop --log-dir /custom/path

--stop also finds a custom log directory recorded when the daemon was started. --status only checks the directory you give it (or the default).

Log Management

Logs are not rotated automatically. To limit disk usage, truncate the file from time to time or use your system’s rotation tool:

# Truncate the log file (macOS example)
: > ~/Library/Logs/grepai/grepai-worktree-<id>.log

# Or set up logrotate (Linux) / newsyslog (macOS)

PID File Management

The daemon writes a PID file and locks it, which lets grepai:

• Refuse to start a second background watcher for the same project
• Make a foreground grepai watch refuse to start while a background watcher runs
• Clean up stale PID files left by crashed processes

Alternative: Terminal Multiplexers

You can also use tmux or screen. Add --no-ui if you will read the output later:

# Using tmux
tmux new-session -d -s grepai 'grepai watch --no-ui'

# Using screen
screen -dmS grepai grepai watch --no-ui

Foreground UI Controls

In an interactive terminal (stdin and stdout are terminals and TERM is set and not dumb), grepai watch shows a UI with the scan and embedding progress, an event ledger and the watcher health. It is not used with --no-ui, --background or --workspace.

With --no-ui, the same run prints plain text (abridged):

Starting grepai watch in /path/to/project
Provider: ollama (nomic-embed-text)
Backend: gob
RPG: disabled
...
Performing initial scan...
Initial scan complete: 245 files indexed, 1842 chunks created, 0 files removed, 3 skipped (took 45.2s)
Checking symbol index...
Symbol index built: 3421 symbols extracted
...
Watching for changes... (Press Ctrl+C to stop)

On a restart with nothing to update, the scan line reads Index up to date: 245 files checked, 0 reindexed (took 120ms). The omitted lines include session status lines such as [RUNNING] /path/to/project - steady.

Troubleshooting

System Limits (Linux)

On Linux, large projects may need a higher inotify watch limit:

# Check current limit
cat /proc/sys/fs/inotify/max_user_watches

# Increase temporarily
sudo sysctl fs.inotify.max_user_watches=524288

# Increase permanently
echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf

If the watcher reports inotify watch limit reached (no space left on device, ENOSPC) while adding a directory, this is the per-user inotify quota, not disk space. The quota is shared by all your processes, so an editor, language server or another watcher can use it up after grepai starts. Raise fs.inotify.max_user_watches or stop other watchers, then restart grepai watch.

Use Cases

Development Workflow

Keep the watcher running while you code:

# Terminal 1: watch for changes
grepai watch

# Terminal 2: search as needed
grepai search "current feature implementation"

CI/CD Integration

In CI, update the index once and search:

grepai index
grepai search "security vulnerabilities" --json --compact

The embedder configured in .grepai/config.yaml must be reachable from the CI runner.

Workspace Mode

For multi-project setups, the watcher can index every project of a workspace into a shared vector store:

# Start workspace watcher (foreground)
grepai watch --workspace my-fullstack

# Start workspace watcher (background)
grepai watch --workspace my-fullstack --background

# Check workspace watcher status
grepai watch --workspace my-fullstack --status

# Stop workspace watcher
grepai watch --workspace my-fullstack --stop

The workspace watcher always prints plain text (no UI).

How Workspace Watching Differs

Coexistence

A per-project watcher and a workspace watcher both write the project’s symbol index in .grepai/, so they take the same writer lock (.grepai/writer.lock). The second one to start exits with an another writer is active error. Run one or the other for a given project, not both at once.

Log Locations (Workspace)

Workspace logs use the same OS-specific directories as single-project logs:

See Workspace Management for setup and configuration.

Commands Reference

• grepai watch: full CLI reference
• grepai index: one-time indexing
• grepai status: check index status
• grepai init: initialize configuration