File Watching

Real-time index updates with grepai watch

File Watching

grepai watch is the core daemon that maintains your codebase index. It performs an initial full scan and then monitors for file changes in real-time.

Features

• Initial indexing: Full codebase scan on startup
• Real-time updates: Monitors filesystem for changes
• Incremental sync: Only re-indexes modified files
• Symbol extraction: Builds call graph data for grepai trace
• Debounced processing: Batches rapid changes efficiently

Quick Start

# Initialize grepai in your project
grepai init

# Start the watcher daemon
grepai watch

Output:

Starting grepai watch in /path/to/project
Provider: ollama (nomic-embed-text)
Backend: gob

Performing initial scan...
Indexing [================] 100% (245/245) src/auth/handler.go
Initial scan complete: 245 files indexed, 1842 chunks created (took 45.2s)
Building symbol index...
Symbol index built: 3421 symbols extracted

Watching for changes... (Press Ctrl+C to stop)

How It Works

+------------------+     +------------------+     +------------------+
|   Initial Scan   | --> |   File Watcher   | --> |  Index Update    |
|   (full index)   |     |   (fsnotify)     |     |  (incremental)   |
+------------------+     +------------------+     +------------------+
                                |
                                v
                        +------------------+
                        |   Debouncing     |
                        |   (500ms)        |
                        +------------------+

1. Initial scan: Compares disk state with existing index, removes obsolete entries, indexes new files
2. File watching: Monitors filesystem events (create, modify, delete, rename)
3. Debouncing: Batches rapid changes to avoid redundant indexing
4. Atomic updates: Prevents duplicate vectors during updates

What Gets Indexed

The watcher indexes files with these extensions:

What Gets Skipped

Files are skipped based on:

1. .gitignore patterns: Automatically respected
2. Config ignore patterns: From .grepai/config.yaml
3. Binary files: Non-text files are excluded
4. Large files: Files exceeding size limits

Default ignore patterns:

scanner:
  ignore:
    - "*.min.js"
    - "*.min.css"
    - "vendor/"
    - "node_modules/"
    - ".git/"
    - "target/"
    - ".zig-cache/"
    - "zig-out/"

Real-Time Updates

When files change, the watcher logs updates:

[MODIFY] src/auth/handler.go
Indexed src/auth/handler.go (4 chunks)
Extracted 12 symbols from src/auth/handler.go

[CREATE] src/api/routes.go
Indexed src/api/routes.go (3 chunks)
Extracted 8 symbols from src/api/routes.go

[DELETE] src/old/deprecated.go
Removed src/old/deprecated.go from index

Symbol Indexing

The watcher also builds a symbol index for call graph analysis:

• Symbols extracted: Functions, methods, classes, types
• References tracked: Function calls and references
• Used by: grepai trace for call graph analysis

See Call Graph Analysis for more details.

Configuration

Configure watcher behavior in .grepai/config.yaml:

# Chunking parameters
chunking:
  size: 512      # Tokens per chunk
  overlap: 50    # Overlap for context

# Additional ignore patterns
scanner:
  ignore:
    - "*.generated.go"
    - "dist/"
    - "build/"

Persistence

The watcher periodically saves the index:

• Auto-save: Automatic persistence during operation
• Shutdown save: Clean save on Ctrl+C or SIGTERM
• Location: .grepai/index.gob (or PostgreSQL)

Background Daemon Mode

Run the watcher as a background daemon with built-in lifecycle management:

# Start the watcher in background
grepai watch --background

# Check if the watcher is running
grepai watch --status

# Stop the background watcher gracefully
grepai watch --stop

Starting the Daemon

$ grepai watch --background
Background watcher started (PID 12345)
Logs: /Users/you/Library/Logs/grepai/grepai-watch.log

Use 'grepai watch --status' to check status
Use 'grepai watch --stop' to stop the watcher

The daemon waits for full initialization (embedder connection, initial scan) before returning success.

Checking Status

$ grepai watch --status
Status: running
PID: 12345
Log directory: /Users/you/Library/Logs/grepai
Log file: /Users/you/Library/Logs/grepai/grepai-watch.log

Stopping the Daemon

$ grepai watch --stop
Stopping background watcher (PID 12345)...
Background watcher stopped

The daemon performs a graceful shutdown, persisting the index before exiting.

Log Locations

Logs are stored in OS-specific directories:

Use --log-dir to override:

grepai watch --background --log-dir /custom/path
grepai watch --status --log-dir /custom/path
grepai watch --stop --log-dir /custom/path

Log Management

Logs are not rotated automatically. To prevent disk usage growth:

# Truncate log file (macOS example)
echo "" > ~/Library/Logs/grepai/grepai-watch.log

# Or set up logrotate (Linux) / newsyslog (macOS)

PID File Management

The daemon uses PID files with file locking to:

• Prevent multiple watchers from starting simultaneously
• Automatically clean up stale PID files from crashed processes
• Detect if a watcher is already running

Alternative: Terminal Multiplexers

You can also use traditional tools if preferred:

# Using tmux
tmux new-session -d -s grepai 'grepai watch'

# Using screen
screen -dmS grepai grepai watch

Troubleshooting

System Limits (Linux)

On Linux, you may need to increase inotify watchers for large projects:

# Check current limit
cat /proc/sys/fs/inotify/max_user_watches

# Increase temporarily
sudo sysctl fs.inotify.max_user_watches=524288

# Increase permanently
echo "fs.inotify.max_user_watches=524288" | sudo tee -a /etc/sysctl.conf

Use Cases

Development Workflow

Keep the watcher running in a terminal while you code:

# Terminal 1: Watch for changes
grepai watch

# Terminal 2: Search as you need
grepai search "current feature implementation"

CI/CD Integration

For CI environments, run a one-time index:

grepai watch &
sleep 60  # Wait for initial indexing
grepai search "security vulnerabilities" --json --compact

Workspace Mode

For multi-project setups, the watcher can index all projects in a workspace using a shared vector store:

# Start workspace watcher (foreground)
grepai watch --workspace my-fullstack

# Start workspace watcher (background)
grepai watch --workspace my-fullstack --background

# Check workspace watcher status
grepai watch --workspace my-fullstack --status

# Stop workspace watcher
grepai watch --workspace my-fullstack --stop

How Workspace Watching Differs

Coexistence

Per-project and workspace watchers can run simultaneously. The per-project watcher indexes into the local store (e.g., .grepai/index.gob), while the workspace watcher indexes into the shared store (e.g., Qdrant or PostgreSQL). This is useful if you want both fast local search and cross-project search.

Log Locations (Workspace)

Workspace watcher logs follow the same OS-specific directories as single-project logs:

See Workspace Management for full workspace setup and configuration.

Commands Reference

• grepai watch - Full CLI reference
• grepai status - Check index status
• grepai init - Initialize configuration