Document IDs

Added in version 3.0.

By default, FSCrawler derives the Elasticsearch document _id by hashing the file path (and similarly hashes folder paths and path.root values used for housekeeping queries). You can control the digest algorithm with fs.hash_algorithm, or skip hashing entirely with fs.filename_as_id.

See also Using filename as elasticsearch _id and Document ID.

fs.hash_algorithm

Name

Environment Variable

Default when unset

New jobs (--setup)

fs.hash_algorithm

FSCRAWLER_FS_HASH_ALGORITHM

MD5

SHA-256

Any algorithm supported by Java MessageDigest is accepted (for example MD5, SHA-1, SHA-256). Invalid names disable the crawler at startup.

name: "test"
fs:
  url: "/path/to/data/dir"
  hash_algorithm: "SHA-256"

Defaults and compatibility

  • Existing jobs that do not set fs.hash_algorithm keep MD5, including the historical encoding used for document _ids, so upgrading FSCrawler does not rewrite existing ids.

  • New jobs created with fscrawler --setup get an example _settings.yaml with hash_algorithm: "SHA-256".

fs.hash_algorithm is independent from fs.checksum. The latter hashes file content for the file.checksum field; fs.hash_algorithm only affects document and folder _ids (and related path.root hashes).

When fs.filename_as_id is true, the raw filename is used as _id and fs.hash_algorithm is ignored.

The same setting applies to the crawler and to the REST _document endpoint when an id is not provided explicitly.

Changing the algorithm (reindex checklist)

Changing fs.hash_algorithm (or switching between hashed ids and filename_as_id) produces new _ids for the same files. FSCrawler will treat them as new documents unless you start from a clean index. There is no automatic id-migration tool.

Recommended steps:

  1. Stop FSCrawler.

  2. Create a new Elasticsearch index (or delete / empty the existing documents and folder index).

  3. Update fs.hash_algorithm in _settings.yaml (or remove it to fall back to MD5).

  4. Optionally point elasticsearch.index / elasticsearch.index_folder at the new indices.

  5. Restart with --restart so the checkpoint is cleared and the filesystem is fully re-scanned.

  6. After verification, remove the old index or update aliases.

  7. Remember that path.root and folder documents are hashed with the same algorithm; they must stay consistent with file documents.

See also the Elasticsearch Reindex API if you need to copy other data between indices.