Configuring the logger

FSCrawler comes with a default logger configuration which can be found in the FSCrawler installation dir as config/log4j2.xml file.

You can modify it to suit your needs. It will be automatically reloaded every 30 seconds.

There are some properties to make your life easier to change the log levels or the log dir:

<Properties>
   <Property name="LOG_LEVEL">info</Property>
   <Property name="DOC_LEVEL">info</Property>
   <Property name="LOG_DIR">logs</Property>
</Properties>

You can control where FSCrawler will store the logs and the log levels by setting LOG_DIR, LOG_LEVEL and DOC_LEVEL Java properties.

FS_JAVA_OPTS="-DLOG_DIR=path/to/logs_dir -DLOG_LEVEL=trace -DDOC_LEVEL=debug" bin/fscrawler

By default, it will log everything in the logs directory inside the installation folder.

Two log files are generated:

  • One is used to log FSCrawler code execution, named fscrawler.log. It’s automatically rotated every day or after 20mb of logs and gzipped. Logs are removed after 7 days.

  • One is used to trace all information about documents, named documents.log. It’s automatically rotated every day or after 20mb of logs and gzipped. Logs are removed after 7 days.

  • One is used when bulk indexing fails, named bulk-failures.log. Each line starts with the failure reason in brackets (for example [Read timed out]), then the bulk executionId and the affected actions (_index / _id). Enable TRACE on logger fscrawler.bulk.failure to also dump truncated document payloads (8 KiB max per document). Console / fscrawler.log WARNs point to this file.

You can change this strategy by modifying the config/log4j2.xml file. Please read Log4J2 documentation on how to configure Log4J.

By default, noisy third-party warnings that do not stop indexing are dialed down. In particular:

  • org.apache.pdfbox is set to error so messages like No Unicode mapping for CID+… (subset fonts missing a ToUnicode CMap) do not flood the logs.

  • org.apache.fontbox is set to error so messages like No PostScript name data is provided for the font … do not flood the logs.

Raise those loggers if you need to diagnose PDF / font extraction issues.

To dump failed bulk payloads:

<Logger name="fscrawler.bulk.failure" level="trace" additivity="false">
   <AppenderRef ref="BulkFailures" />
</Logger>

Note

FSCrawler detects automatically on Linux machines when it’s running in background or foreground. When in background, the logger configuration file used is config/log4j2-file.xml.

In the Docker context, you can modify the logs level by setting the FS_JAVA_OPTS environment variable:

docker run -it --rm \
     -v ~/.fscrawler:/root/.fscrawler \
     -v ~/tmp:/tmp/es:ro \
     -v ~/logs:/root/logs \
     -e FS_JAVA_OPTS="-DLOG_LEVEL=debug -DDOC_LEVEL=debug" \
     dadoonet/fscrawler job_name

Then the logs will be readable from the ~/logs directory.

Read Using Docker for more information.

Same for Docker Compose, you can modify your docker-compose.yml file:

version: '3'
services:
  fscrawler:
    image: dadoonet/fscrawler
    volumes:
      - ~/.fscrawler:/root/.fscrawler
      - ~/tmp:/tmp/es:ro
      - ~/logs:/root/logs
    environment:
      - FS_JAVA_OPTS=-DLOG_LEVEL=debug -DDOC_LEVEL=debug
    command: job_name

Read Using Docker Compose for more information.