Welcome to FSCrawler’s documentation!¶
This documentation is for the version of FSCrawler currently under development. Were you looking for the documentation of the latest stable version?
Welcome to the FS Crawler for Elasticsearch.
This crawler helps to index binary documents such as PDF, Open Office, MS Office.
- Local file system (or a mounted drive) crawling and index new files, update existing ones and removes old ones.
- Remote file system over SSH/FTP crawling.
- REST interface to let you “upload” your binary documents to elasticsearch.
FS Crawler 2.8-SNAPSHOT is using Tika 2.1.0 and:
- Getting Started
- Crawler options
- OCR integration
- Starting with a REST gateway
- Supported formats
- Tips and tricks
- Status files
- CLI options
- JVM Settings
- Configuring the logger
- Job file specification
- The most simple crawler
- Local FS settings
- Root directory
- Update rate
- Includes and excludes
- Filter content
- Indexing JSon docs
- Indexing XML docs
- Add as Inner Object
- Index folders
- Dealing with multiple types and multiple dirs
- Dealing with multiple types within the same dir
- Using filename as elasticsearch
- Adding file attributes
- Disabling raw metadata
- Disabling file size field
- Ignore deleted files
- Ignore content
- Continue on Error
- Language detection
- Storing binary source document
- Extracted characters
- Ignore Above
- File checksum
- Follow Symlinks
- SSH settings
- FTP settings
- Elasticsearch settings
- Workplace Search settings
- REST service
- Building the project
- Clone the project
- Build the artifact
- Integration tests
- Check for vulnerabilities (CVE)
- Docker build
- DockerHub publication
- Writing documentation
- Release the project
This software is licensed under the Apache 2 license, quoted below.
Copyright 2011-2021 David Pilato
Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at
Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Incompatible 3rd party library licenses¶
Some libraries are not Apache2 compatible. Therefore they are not
packaged with FSCrawler so you need to download and add manually them to
- for TIFF images, you need to add jai-imageio-core:1.4.0 library
- for JPEG 2000 (JPX) images, you need to add jai-imageio-jpeg2000:1.4.0 library
See pdfbox documentation for more details.