<a id="local-fs-settings"></a>

# Local FS settings

> ##### Contents
> 
> * [Local FS settings](#local-fs-settings)
>   * [Crawler Provider](#crawler-provider)
>   * [Root directory](#root-directory)
>   * [Update rate](#update-rate)
>   * [Includes and excludes](#includes-and-excludes)
>   * [Filter content](#filter-content)
>   * [Indexing JSON docs](#indexing-json-docs)
>   * [Indexing XML docs](#indexing-xml-docs)
>   * [Add as Inner Object](#add-as-inner-object)
>   * [Index folders](#index-folders)
>   * [Dealing with multiple types and multiple dirs](#dealing-with-multiple-types-and-multiple-dirs)
>   * [Dealing with multiple types within the same dir](#dealing-with-multiple-types-within-the-same-dir)
>   * [Using filename as elasticsearch `_id`](#using-filename-as-elasticsearch-id)
>   * [Adding file attributes](#adding-file-attributes)
>   * [Collecting ACL metadata](#collecting-acl-metadata)
>   * [Enabling raw metadata](#enabling-raw-metadata)
>   * [Disabling file size field](#disabling-file-size-field)
>   * [Ignore deleted files](#ignore-deleted-files)
>   * [Ignore content](#ignore-content)
>   * [Continue on Error](#continue-on-error)
>   * [Language detection](#language-detection)
>   * [Storing binary source document](#storing-binary-source-document)
>   * [Extracted characters](#extracted-characters)
>   * [Ignore Above](#ignore-above)
>   * [File checksum](#file-checksum)
>   * [Temporary Directory](#temporary-directory)
>   * [Follow Symlinks](#follow-symlinks)
>   * [Tika Config Path](#tika-config-path)

Here is a list of Local FS settings (under `fs.` prefix):

| Name                     | Environment Variable               | Default value             | Documentation                                                                       |
|--------------------------|------------------------------------|---------------------------|-------------------------------------------------------------------------------------|
| `fs.provider`            | `FSCRAWLER_FS_PROVIDER`            | `"local"`                 | [Crawler Provider]()                                                                |
| `fs.url`                 | `FSCRAWLER_FS_URL`                 | `"/tmp/es"`               | [Root directory]()                                                                  |
| `fs.update_rate`         | `FSCRAWLER_FS_UPDATE_RATE`         | `"15m"`                   | [Update Rate]()                                                                     |
| `fs.includes`            | `FSCRAWLER_FS_INCLUDES`            | `null`                    | [Includes and excludes]()                                                           |
| `fs.excludes`            | `FSCRAWLER_FS_EXCLUDES`            | `["*/~*", "*/.ds_store"]` | [Includes and excludes]()                                                           |
| `fs.filters`             | `FSCRAWLER_FS_FILTERS`             | `null`                    | [Filter content]()                                                                  |
| `fs.json_support`        | `FSCRAWLER_FS_JSON_SUPPORT`        | `false`                   | [Indexing JSON docs]()                                                              |
| `fs.xml_support`         | `FSCRAWLER_FS_XML_SUPPORT`         | `false`                   | [Indexing XML docs]()                                                               |
| `fs.add_as_inner_object` | `FSCRAWLER_FS_ADD_AS_INNER_OBJECT` | `false`                   | [Add as Inner Object]()                                                             |
| `fs.index_folders`       | `FSCRAWLER_FS_INDEX_FOLDERS`       | `true`                    | [Index folders]()                                                                   |
| `fs.attributes_support`  | `FSCRAWLER_FS_ATTRIBUTES_SUPPORT`  | `false`                   | [Adding file attributes]()                                                          |
| `fs.acl_support`         | `FSCRAWLER_FS_ACL_SUPPORT`         | `false`                   | [Collecting ACL metadata]()                                                         |
| `fs.raw_metadata`        | `FSCRAWLER_FS_RAW_METADATA`        | `false`                   | [Enabling raw metadata]()                                                           |
| `fs.filename_as_id`      | `FSCRAWLER_FS_FILENAME_AS_ID`      | `false`                   | [Using filename as elasticsearch \_id](#filename-as-id) |
| `fs.hash_algorithm`      | `FSCRAWLER_FS_HASH_ALGORITHM`      | `"MD5"`                   | [Document IDs](document-ids.html.md#document-ids)            |
| `fs.add_filesize`        | `FSCRAWLER_FS_ADD_FILESIZE`        | `true`                    | [Disabling file size field]()                                                       |
| `fs.remove_deleted`      | `FSCRAWLER_FS_REMOVE_DELETED`      | `true`                    | [Ignore deleted files]()                                                            |
| `fs.store_source`        | `FSCRAWLER_FS_STORE_SOURCE`        | `false`                   | [Storing binary source document](#store-binary)       |
| `fs.index_content`       | `FSCRAWLER_FS_INDEX_CONTENT`       | `true`                    | [Ignore content]()                                                                  |
| `fs.lang_detect`         | `FSCRAWLER_FS_LANG_DETECT`         | `false`                   | [Language detection]()                                                              |
| `fs.continue_on_error`   | `FSCRAWLER_FS_CONTINUE_ON_ERROR`   | `false`                   | [Continue on Error](#continue-on-error)                    |
| `fs.indexed_chars`       | `FSCRAWLER_FS_INDEXED_CHARS`       | `100000.0`                | [Extracted characters]()                                                            |
| `fs.ignore_above`        | `FSCRAWLER_FS_IGNORE_ABOVE`        | `null`                    | [Ignore above]()                                                                    |
| `fs.checksum`            | `FSCRAWLER_FS_CHECKSUM`            | `null`                    | [File Checksum]()                                                                   |
| `fs.temp_dir`            | `FSCRAWLER_FS_TEMP_DIR`            | `null`                    | [Temporary Directory]()                                                             |
| `fs.follow_symlinks`     | `FSCRAWLER_FS_FOLLOW_SYMLINKS`     | `false`                   | [Follow Symlinks]()                                                                 |
| `fs.tika_config_path`    | `FSCRAWLER_FS_TIKA_CONFIG_PATH`    | `null`                    | [Tika Config Path]()                                                                |
| `fs.ocr.enabled`         | `FSCRAWLER_FS_OCR_ENABLED`         | `true`                    | [OCR integration](../../user/ocr.html.md#ocr-integration)      |

<a id="crawler-provider"></a>

## Crawler Provider

#### Versionadded
Added in version 3.0.

The `fs.provider` setting specifies which crawler plugin to use for scanning files.
Available providers are:

* `local` (default): Crawl files from the local filesystem
* `ftp`: Crawl files from a remote FTP server (see [FTP settings](ftp.html.md#ftp-settings))
* `ssh`: Crawl files from a remote server via SSH/SFTP (see [SSH settings](ssh.html.md#ssh-settings))

```yaml
name: "test"
fs:
  provider: "local"
  url: "/path/to/data/dir"
```

#### NOTE
The `fs.provider` setting replaces the deprecated `server.protocol` setting.
SSH/FTP connection details belong under `fs.providers.ssh` / `fs.providers.ftp`
(see [SSH settings](ssh.html.md#ssh-settings) and [FTP settings](ftp.html.md#ftp-settings)). The top-level `server.*` block is
deprecated and will be removed in a future version.

Old configuration (deprecated):

```yaml
name: "test"
fs:
  url: "/path/to/data/dir"
server:
  hostname: "mynode.mydomain.com"
  protocol: "ftp"
```

New configuration (recommended):

```yaml
name: "test"
fs:
  provider: "ftp"
  url: "/path/to/data/dir"
  providers:
    ftp:
      hostname: "mynode.mydomain.com"
```

<a id="root-directory"></a>

## Root directory

Define `fs.url` property in your `~/.fscrawler/test/_settings.yaml`
file:

```yaml
name: "test"
fs:
  url: "/path/to/data/dir"
```

For Windows users, use a form like `c:/tmp` or `c:\\tmp`.

<a id="local-fs-update-rate"></a>

## Update rate

By default, `update_rate` is set to `15m`. You can modify this value using any compatible
[time unit](https://www.elastic.co/guide/en/elasticsearch/reference/current/common-options.html#time-units).

For example, here is a 15 minutes update rate:

```yaml
name: "test"
fs:
  update_rate: "15m"
```

Or a 3 hours update rate:

```yaml
name: "test"
fs:
  update_rate: "3h"
```

`update_rate` is the pause duration between the last time we read the
file system and another run. Which means that if you set it to `15m`,
the next scan will happen on 15 minutes after the end of the current
scan, whatever its duration.

The supported units for duration are:

* `d` for days
* `h` for hours
* `m` for minutes
* `s` for seconds
* `ms` for milliseconds

#### NOTE
If you don’t want to wait for the next scan, you can manually edit the `~/.fscrawler/{job_name}/_checkpoint.json` file and
set `next_check` to the current time or to `null`. FSCrawler will then start a new scan at most after 5 seconds.

See [Checkpoint file](../status.html.md#status-files) for more information.

<a id="includes-excludes"></a>

## Includes and excludes

Let’s say you want to index only docs like `*.doc` and `*.pdf` but
`resume*`. So `resume_david.pdf` won’t be indexed.

Define `fs.includes` and `fs.excludes` properties in your
`~/.fscrawler/test/_settings.yaml` file:

```yaml
name: "test"
fs:
  includes:
  - "*/*.doc"
  - "*/*.pdf"
  excludes:
  - "*/resume*"
```

By default, FSCrawler will exclude files starting with `~` and macOS Finder
metadata files named `.DS_Store` (matched case-insensitively via `*/.ds_store`).

It also applies to directory names. So if you want to ignore `.ignore`
dir, just add `.ignore` as an excluded name. Note that `includes` and `excludes`
apply to directory names as well.

Let’s take the following example with the `root` dir as `/tmp`:

```default
 /tmp
 ├── folderA
 │   ├── subfolderA
 │   ├── subfolderB
 │   └── subfolderC
 ├── folderB
 │   ├── subfolderA
 │   ├── subfolderB
 │   └── subfolderC
 └── folderC
     ├── subfolderA
     ├── subfolderB
     └── subfolderC
```

If you define the following `fs.excludes` property in your
`~/.fscrawler/test/_settings.yaml` file:

```yaml
name: "test"
fs:
  excludes:
  - "/folderB/subfolder*"
```

Then all files but the ones in `/folderB/subfolderA`, `/folderB/subfolderB` and
`/folderB/subfolderC` will be indexed.

If you want to exclude a specific folder, you need to use a wildcard character at the end of the folder name, like:

```yaml
name: "test"
fs:
  excludes:
  - "/folderB/subfolderB/*"
```

Since the includes and excludes work on the entire *path of the file* you must consider that when using wildcards. Below are some includes and excludes pattern to help convey the idea better.

| Pattern         | Includes                                      | Excludes                                      |
|-----------------|-----------------------------------------------|-----------------------------------------------|
| `*.jpg`         | Include all jpg files                         | exclude all jpg files                         |
| `/images/*.jpg` | Include all jpg files in the images directory | Exclude all jpg files in the images directory |
| `*/old-*.jpg`   | Include all jpg files that start with `old-`  | Exclude all jpg files that start with `old-`  |

If a folder contains a file named `.fscrawlerignore`, this folder and its subfolders will be entirely skipped.

## Filter content

You can filter out documents you would like to index by adding one or more
regular expression that match the extracted content.
Documents which are not matching will be simply ignored and not indexed.

If you define the following `fs.filters` property in your
`~/.fscrawler/test/_settings.yaml` file:

```yaml
name: "test"
fs:
  filters:
  - ".*foo.*"
  - "^4\\d{3}([\\ \\-]?)\\d{4}\\1\\d{4}\\1\\d{4}$"
```

With this example, only documents which contains the word `foo` and a VISA credit card number
with the form like `4012888888881881`, `4012 8888 8888 1881` or `4012-8888-8888-1881`
will be indexed.

## Indexing JSON docs

If you want to index JSON files directly without parsing with Tika, you
can set `json_support` to `true`. JSON contents will be stored
directly under \_source. If you need to keep JSON documents synchronized
to the index, set option [Add as Inner Object]()
which stores additional metadata and the JSON contents under field
`object`.

```yaml
name: "test"
fs:
  json_support: true
```

Of course, if you did not define a mapping before launching the crawler,
Elasticsearch will auto guess the mapping.

## Indexing XML docs

If you want to index XML files and convert them to JSON, you can set
`xml_support` to `true`. The content of XML files will be added
directly under \_source. If you need to keep XML documents synchronized
to the index, set option [Add as Inner Object]()
which stores additional metadata and the XML contents under field
`object`.

```yaml
name: "test"
fs:
  xml_support: true
```

Of course, if you did not define a mapping before launching the crawler,
Elasticsearch will auto guess the mapping.

## Add as Inner Object

The default settings store the contents of JSON and XML documents
directly onto the \_source element of Elasticsearch documents. Thereby,
there is no metadata about file and path settings, which are necessary
to determine if a document is deleted or updated. New files will however
be added to the index, (determined by the file timestamp).

If you need to keep JSON or XML documents synchronized to elasticsearch,
you should set this option.

```yaml
name: "test"
fs:
  add_as_inner_object: true
```

## Index folders

By default, FSCrawler will index folder names in the folder index. If
you don’t want to index those folders, you can set `index_folders` to
`false`.

Note that in that case, FSCrawler won’t be able to detect removed
folders so any document has been indexed in Elasticsearch, it won’t be
removed when you remove or move the folder away.

See `elasticsearch.index_folder` below for the name of the index to be used to store the folder data (if `es.index_folders` is set to `true`).

```yaml
name: "test"
fs:
  index_folders: false
```

## Dealing with multiple types and multiple dirs

If you have more than one type, create as many crawlers as types and/or folders:

`~/.fscrawler/test_type1/_settings.yaml`:

```yaml
name: "test_type1"
fs:
  url: "/tmp/type1"
  json_support: true
elasticsearch:
  index: "mydocs1"
  index_folder: "myfolders1"
```

`~/.fscrawler/test_type2/_settings.yaml`:

```yaml
name: "test_type2"
fs:
  url: "/tmp/type2"
  json_support: true
elasticsearch:
  index: "mydocs2"
  index_folder: "myfolders2"
```

`~/.fscrawler/test_type3/_settings.yaml`:

```yaml
name: "test_type3"
fs:
  url: "/tmp/type3"
  xml_support: true
elasticsearch:
  index: "mydocs3"
  index_folder: "myfolders3"
```

## Dealing with multiple types within the same dir

You can also index many types from one single dir using two crawlers
scanning the same dir and by setting `includes` parameter:

`~/.fscrawler/test_type1.yaml`:

```yaml
name: "test_type1"
fs:
  url: "/tmp"
  includes:
  - "type1*.json"
  json_support: true
elasticsearch:
  index: "mydocs1"
  index_folder: "myfolders1"
```

`~/.fscrawler/test_type2.yaml`:

```yaml
name: "test_type2"
fs:
  url: "/tmp"
  includes:
  - "type2*.json"
  json_support: true
elasticsearch:
  index: "mydocs2"
  index_folder: "myfolders2"
```

`~/.fscrawler/test_type3.yaml`:

```yaml
name: "test_type3"
fs:
  url: "/tmp"
  includes:
  - "*.xml"
  xml_support: true
elasticsearch:
  index: "mydocs3"
  index_folder: "myfolders3"
```

<a id="filename-as-id"></a>

## Using filename as elasticsearch `_id`

Please note that by default the document `_id` is generated as a hash of the file path
(see [Document IDs](document-ids.html.md#document-ids)). You can force the `_id` to be the filename using the
`filename_as_id` attribute:

```yaml
name: "test"
fs:
  filename_as_id: true
```

When `filename_as_id` is `true`, `fs.hash_algorithm` is ignored.

## Adding file attributes

If you want to add file attributes such as `attributes.owner`, `attributes.group`
and `attributes.permissions`, you can set `attributes_support` to `true`.

```yaml
name: "test"
fs:
  attributes_support: true
```

#### NOTE
On Windows systems, `attributes.group` and `attributes.permissions` are
not generated.

## Collecting ACL metadata

To extract NTFS access control entries (principal, type, permissions and flags),
enable both `attributes_support` and `acl_support`:

```yaml
name: "test"
fs:
  attributes_support: true
  acl_support: true
```

When `acl_support` is disabled, FSCrawler skips resolving ACLs even if
`attributes_support` is active.

<a id="enabling-raw-metadata"></a>

## Enabling raw metadata

FSCrawler can extract all found metadata within a `meta.raw` object in addition
to the standard metadata fields.
If you want to enable this feature, you can set `raw_metadata` to `true`.

```yaml
name: "test"
fs:
  raw_metadata: true
```

#### Versionchanged
Changed in version 3.0: The names of the keys stored in `meta.raw` changed compared to FSCrawler 2.9 because of
the Apache Tika 4 upgrade. For example, `resourceName` is now `tk:resource-name`, Tika
internal keys use the `tk:` prefix instead of `X-TIKA:`, ICC keys use `icc:` instead of
`ICC:`, and many PDF keys now use hyphens (`pdf:pdf-version` instead of `pdf:PDFVersion`).
See the [3.0 release notes](../../release/3.0.html.md#release-notes-3-0) for a summary of the renamed keys.

Generated raw metadata depends on the file format itself.

For example, a PDF document could generate:

```json
{
   "access-permission:assemble-document" : "true",
   "access-permission:can-modify" : "true",
   "access-permission:can-print" : "true",
   "access-permission:can-print-faithful" : "true",
   "access-permission:extract-content" : "true",
   "access-permission:extract-for-accessibility" : "true",
   "access-permission:fill-in-form" : "true",
   "access-permission:modify-annotations" : "true",
   "Content-Length" : "101643",
   "Content-Type" : "application/pdf",
   "dc:creator" : "David Pilato",
   "dc:format" : "application/pdf; version=1.5",
   "dc:language" : "en-US",
   "dc:subject" : "keyword1, keyword2",
   "dc:title" : "Test Tika title",
   "dcterms:created" : "2016-07-07T08:37:42Z",
   "dcterms:modified" : "2016-07-07T08:37:42Z",
   "pdf:chars-per-page" : "42",
   "pdf:contains-damaged-font" : "false",
   "pdf:contains-non-embedded-font" : "false",
   "pdf:docinfo:created" : "2016-07-07T08:37:42Z",
   "pdf:docinfo:creator" : "David Pilato",
   "pdf:docinfo:creator-tool" : "Microsoft Word",
   "pdf:docinfo:keywords" : "keyword1, keyword2",
   "pdf:docinfo:modified" : "2016-07-07T08:37:42Z",
   "pdf:docinfo:subject" : "Test Tika Object",
   "pdf:docinfo:title" : "Test Tika title",
   "pdf:encrypted" : "false",
   "pdf:eof-offsets" : "101460",
   "pdf:has-collection" : "false",
   "pdf:has-marked-content" : "true",
   "pdf:has-xfa" : "false",
   "pdf:has-xmp" : "false",
   "pdf:incremental-update-count" : "1",
   "pdf:num-3d-annotations" : "0",
   "pdf:ocr-page-count" : "0",
   "pdf:overall-percentage-unmapped-unicode-chars" : "0.0",
   "pdf:pdf-version" : "1.5",
   "pdf:total-unmapped-unicode-chars" : "0",
   "pdf:unmapped-unicode-chars-per-page" : "0",
   "tk:content-type-magic-detected" : "application/pdf",
   "tk:parsed-by" : "org.apache.tika.parser.pdf.PDFParser",
   "tk:parsed-by-full-set" : "org.apache.tika.parser.pdf.PDFParser",
   "tk:resource-name" : "test.pdf",
   "tk:version-count" : "1",
   "xmp:CreatorTool" : "Microsoft Word",
   "xmpTPg:NPages" : "2"
}
```

Where a MP3 file would generate:

```json
{
   "audio:bitrate" : "192000",
   "audio:channels" : "2",
   "audio:is-variable-bitrate" : "false",
   "audio:raw-track-number" : "1",
   "Content-Type" : "audio/mpeg",
   "dc:creator" : "David Pilato",
   "dc:title" : "Test Tika",
   "mp3:version" : "MPEG 3 Layer III Version 1",
   "tk:content-type-magic-detected" : "audio/mpeg",
   "tk:parsed-by" : "org.apache.tika.parser.DefaultParser",
   "tk:parsed-by-full-set" : "org.apache.tika.parser.DefaultParser",
   "tk:resource-name" : "test.mp3",
   "xmpDM:album" : "FS Crawler",
   "xmpDM:artist" : "David Pilato",
   "xmpDM:audioChannelType" : "Stereo",
   "xmpDM:audioCompressor" : "MP3",
   "xmpDM:audioSampleRate" : "44100",
   "xmpDM:duration" : "1.0187751054763794",
   "xmpDM:genre" : "Vocal",
   "xmpDM:logComment" : "Hello but reverted",
   "xmpDM:releaseDate" : "2016",
   "xmpDM:trackNumber" : "1"
}
```

#### NOTE
All fields are generated as text even though they can be valid booleans or numbers.

The `meta.raw.*` fields have a default mapping applied:

```json
{
  "type": "text",
  "fields": {
    "keyword": {
      "type": "keyword",
      "ignore_above": 256
    }
  }
}
```

If you want specifically tell elasticsearch to use a date type or a
numeric type for some fields, you need to modify the default template
provided by FSCrawler.

#### NOTE
Note that dots in metadata names will be replaced by a `:`. For example `PTEX.Fullbanner` will be indexed as
`PTEX:Fullbanner`.

#### NOTE
Note that if you have a lot of different type of files, that can generate a lot of raw metadata which can make you hit
the total number of field limit in elasticsearch mappings. In which case you will need to change the index settings
`foo`.

See [elasticsearch documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#mapping-limit-settings)

## Disabling file size field

By default, FSCrawler will create a field to store the original file
size in octets. You can disable it using \`add_filesize’ option:

```yaml
name: "test"
fs:
  add_filesize: false
```

## Ignore deleted files

If you don’t want to remove indexed documents when you remove a file or a directory, you can set `remove_deleted` to
`false` (default to `true`):

```yaml
name: "test"
fs:
  remove_deleted: false
```

#### NOTE
Setting `remove_deleted` is forced to `false` when using the Workplace Search output.

## Ignore content

If you don’t want to extract file content but only index filesystem
metadata such as filename, date, size and path, you can set
`index_content` to `false` (default to `true`):

```yaml
name: "test"
fs:
  index_content: false
```

<a id="continue-on-error"></a>

## Continue on Error

By default, FSCrawler will immediately stop indexing if he hits a
Permission denied exception. If you want to just skip this File and
continue with the rest of the directory tree you can set
`continue_on_error` to `true` (default to `false`):

```yaml
name: "test"
fs:
  continue_on_error: true
```

## Language detection

You can ask for language detection using `lang_detect` option:

```yaml
name: "test"
fs:
  lang_detect: true
```

In that case, a new field named `meta.language` is added to the
generated JSON document.

If you are using elasticsearch 5.0 or superior, you can use this value
to send your document to a specific index using a [Node Ingest pipeline](elasticsearch.html.md#ingest-node).

For example, you can define a pipeline named `langdetect` with:

```sh
PUT _ingest/pipeline/langdetect
{
  "description" : "langdetect pipeline",
  "processors" : [
    {
      "set": {
        "field": "_index",
        "value": "myindex-{{meta.language}}"
      }
    }
  ]
}
```

In FSCrawler settings, set both `fs.lang_detect` and
`elasticsearch.pipeline` options:

```yaml
name: "test"
fs:
  lang_detect: true
elasticsearch:
  pipeline: "langdetect"
```

And then, a document containing french text will be sent to
`myindex-fr`. A document containing english text will be sent to
`myindex-en`.

You can also imagine changing the field name from `content` to
`content-fr` or `content-en`. That will help you to define the
correct analyzer to use.

Language detection might detect more than one language in a given text
but only the most accurate will be set. Which means that if you have a
document containing 80% of french and 20% of english, the document will
be marked as `fr`.

Note that language detection is CPU and time consuming.

<a id="store-binary"></a>

## Storing binary source document

You can store in elasticsearch itself the binary document (BASE64 encoded)
using `store_source` option:

```yaml
name: "test"
fs:
  store_source: true
```

In that case, a new field named `attachment` is added to the generated
JSON document. This field is not indexed. Default mapping for
`attachment` field is:

```json
{
  "_doc" : {
    "properties" : {
      "attachment" : {
        "type" : "binary",
        "doc_values" : false
      }
      // ... Other properties here
    }
  }
}
```

## Extracted characters

By default, FSCrawler will extract only the first 100 000 characters.
But, you can set `indexed_chars` to `5000` in FSCrawler settings in
order to overwrite this default settings.

```yaml
name: "test"
fs:
  indexed_chars: "5000"
```

This number can be either a fixed size, number of characters that is, or
a percent using `%` sign. The percentage value will be applied to the
filesize to determine the number of character the crawler needs to
extract.

If you want to index only `80%` of filesize, define `indexed_chars`
to `"80%"`. Of course, if you want to index the full document, you can
set this property to `"100%"`. Double values are also supported so
`"0.01%"` is also a correct value.

**Compressed files**: If your file is compressed, you might need to
increase `indexed_chars` to more than `"100%"`. For example,
`"150%"`.

If you want to extract the full content, define `indexed_chars` to
`"-1"`.

#### NOTE
Tika requires to allocate in memory a data structure to
extract text. Setting `indexed_chars` to a high number will require
more memory!

## Ignore Above

By default, and if `index_content` set to `true`, FSCrawler will send every single file to Tika, whatever its size.
But some files on your file system might be a way too big to be parsed.

Set `ignore_above` to the desired value of the limit.

```yaml
name: "test"
fs:
  ignore_above: "512mb"
```

<a id="file-checksum"></a>

## File checksum

If you want FSCrawler to generate a checksum for each file, set
`checksum` to the algorithm you wish to use to compute the checksum,
such as `MD5` or `SHA-1`.

#### NOTE
You MUST set `index_content` to true to allow this feature to work. Nevertheless you MAY set `indexed_chars` to 0 if you do not need any content in the index.

You MUST NOT set `json_support` or `xml_support` to allow this feature to work also.

```yaml
name: "test"
fs:
   # required
  index_content: true
  #indexed_chars: 0
  checksum: "MD5"
```

## Temporary Directory

#### Versionadded
Added in version 3.0.

When `checksum` or `store_source` is enabled, FSCrawler may need to create
temporary files to process large documents without loading them entirely into memory.
By default, temporary files are created in `~/.fscrawler/<job_name>/tmp/`.

You can override this location using the `temp_dir` option:

```yaml
name: "test"
fs:
  checksum: "MD5"
  temp_dir: "/path/to/custom/temp"
```

#### NOTE
For small files (64KB or less), FSCrawler uses an in-memory buffer instead of
temporary files for better performance. Temporary files are only created for
larger files to avoid `OutOfMemoryError`.

#### NOTE
Temporary files are automatically deleted after processing each document.

## Follow Symlinks

If you want FSCrawler to follow the symbolic links, you need to be explicit about it and set
`follow_symlinks` to `true`. Starting from version 2.7, symbolic links are not followed anymore.

```yaml
name: "test"
fs:
  follow_symlinks: true
```

## Tika Config Path

#### Versionadded
Added in version 3.0.

If you want to override the default Tika parser configuration, you can set the path to a custom Tika
configuration file, which will be used instead.

#### NOTE
Apache Tika 4 replaced the XML-based configuration file mechanism with a JSON one. If you were using an
XML Tika configuration with a previous version of FSCrawler, you need to convert it to the equivalent
JSON configuration. If the configuration file cannot be loaded, FSCrawler fails fast with a
configuration error instead of silently ignoring the setting.

```yaml
name: "test"
fs:
  tika_config_path: '/path/to/tikaConfig.json'
```

An example Tika JSON config file is shown below. It excludes the HTML parser (`jsoup-parser`) from the
default parser (HTML files are then handled as plain text), routes XHTML files to the XML parser, and
keeps the Vision Language Model (VLM) OCR parsers disabled (see [Using a Vision Language Model (VLM) for OCR](../../user/ocr.html.md#vlm-ocr)):

```json
{
  "parsers": [
    { "default-parser": {
        "exclude": [
          "jsoup-parser",
          "openai-vlm-parser",
          "openai-vlm-deterministic-parser",
          "claude-vlm-parser",
          "gemini-vlm-parser"
        ]
      } },
    { "xml-parser": { "_mime-include": ["application/xhtml+xml"] } }
  ]
}
```

#### NOTE
Since FSCrawler 3.0, the distribution includes Apache Tika’s `tika-vlm` module. If your JSON
configuration uses `default-parser`, add the VLM parser component names to the `exclude` list unless
you explicitly configure one of them in the `parsers` array. Without these exclusions, Tika may load
the VLM parsers through SPI and run a startup health check against their configured endpoint.

The `parsers` array lists the parsers to load, in order. Each entry is either a Tika component name (like
`pdf-parser` or `xml-parser`) with its configuration object, or the special
`default-parser` marker which enables SPI loading of all remaining parsers (with optional `exclude`
list). Parsers support mime filtering via the `_mime-include` and `_mime-exclude` decorations.