# Testing FSCrawler CLI

A convenience shell script is provided to spin up the full EDOT observability
stack and run FSCrawler against the built-in test documents in a single command.

```{contents}
:backlinks: entry
```

## Prerequisites

* Java 17+
* Maven 3.3+
* Docker and either `docker compose` (v2) or `docker-compose` (v1)
* `curl`, `unzip`

## Quick start

From the project root
```
# Full run: build → start docker stack → crawl with OTel tracing
./distribution/test-scripts/test-fscrawler-cli.sh
```

```{note}

 `distribution/test-scripts/test-fscrawler-cli.sh` is a **generated file** — the project
 version is injected by Maven resource filtering at build time.
 The source template is `distribution/src/test/scripts/test-fscrawler-cli.sh`.
 After a version bump, regenerate it with::

     mvn generate-test-resources -pl distribution
```

The script will:

1. Build the distribution ZIP (`mvn clean package -DskipTests -Ddocker.skip`)
2. Start **Elasticsearch + Kibana + EDOT Collector** via docker-compose
3. Unzip the distribution into `/tmp/fscrawler-edot-test/`
4. Create a job config pointing to `test-documents/src/main/resources/documents/`
5. Set `OTEL_*` environment variables and launch FSCrawler with the REST API enabled (`--rest`)

FSCrawler runs until you press **Ctrl+C**. While it is running:

* **Elasticsearch** — `http://localhost:9200/test-edot/_search`
* **Kibana APM** — `http://localhost:5601` → *Observability → APM* → service `fscrawler`

## Options

| Flag                  | Effect                                                                        |
|-----------------------|-------------------------------------------------------------------------------|
| `--skip-build`        | Reuse the existing distribution ZIP (skip Maven build)                        |
| `--skip-docker`       | Assume the docker-compose stack is already running                            |
| `--no-otel`           | Disable OTel tracing (`OTEL_SDK_DISABLED=true`); crawl without EDOT Collector |
| `--log-level=<level>` | Set the FSCrawler log level (default: `info`; try `debug` or `trace`)         |
| `--help`              | Print usage                                                                   |

Examples
```
# Iterate quickly: keep docker stack running, only rebuild + recrawl
./distribution/test-scripts/test-fscrawler-cli.sh --skip-docker

# Rebuild and crawl, but disable tracing (baseline comparison)
./distribution/test-scripts/test-fscrawler-cli.sh --no-otel

# Fastest iteration: nothing to (re)build, stack already up
./distribution/test-scripts/test-fscrawler-cli.sh --skip-build --skip-docker

# Debug log level with stack already running
./distribution/test-scripts/test-fscrawler-cli.sh --skip-build --skip-docker --log-level=debug
```

## What to look for in Kibana APM

After the crawl, open Kibana at http://localhost:5601.
Navigate to **Observability → APM → Services → fscrawler**.

You should see traces containing the following spans in a waterfall view:

```text
fscrawler.crawl                      ← one span per run
└─ fscrawler.directory.traverse
   └─ fscrawler.directory.process    ← one per directory
      └─ fscrawler.file.index        ← one per file
         └─ fscrawler.tika.extract   ← Tika text extraction
fscrawler.es.bulk                    ← Elasticsearch bulk calls
```

Auto-instrumented spans (HTTP, ES client, etc.) also appear as children of the
root trace thanks to the `elastic-otel-javaagent`.

## Stopping the stack

When you're done, stop the docker-compose services
```
docker compose -f contrib/docker-compose-example-edot/docker-compose.yml down
```

Or to also remove the volumes (wipes Elasticsearch data)
```
docker compose -f contrib/docker-compose-example-edot/docker-compose.yml down -v
```