Building the project¶
Clone the project¶
Use git to clone the project locally:
git clone email@example.com:dadoonet/fscrawler.git cd fscrawler
Build the artifact¶
To build the project, run:
mvn clean package
The final artifacts are available in
distribution/esX/target directory where
X is the
elasticsearch major version target.
To build it faster (without tests), run:
mvn clean package -DskipTests
When running from the command line with
mvn integration tests are ran against all supported versions.
This is done by running a Docker instance of elasticsearch using the expected version.
A HTTP server is also started on port 8080 during the integration tests, alternatively the assigned port can be set with -Dtests.rest.port=8090 argument.
Run tests from your IDE¶
To run integration tests from your IDE, you need to start tests in
But you need first to specify the Maven profile to use and rebuild the project.
es-7xfor Elasticsearch 7.x
es-6xfor Elasticsearch 6.x
es-5xfor Elasticsearch 5.x
Run tests with an external cluster¶
To run the test suite against an elasticsearch instance running locally, just run:
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v7
If you want to run against a version 5 or 6, run:
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v5 mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v6
If elasticsearch is not running yet on
http://localhost:9200, FSCrawler project will run a Docker instance before
the tests start.
If you are using a secured instance, use
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v7 \ -Dtests.cluster.user=elastic \ -Dtests.cluster.pass=changeme \ -Dtests.cluster.url=https://127.0.0.1:9200 \
To run tests against another instance (ie. running on
Elasticsearch service by Elastic,
you can also use
tests.cluster.url to set where elasticsearch is running:
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v7 \ -Dtests.cluster.user=elastic \ -Dtests.cluster.pass=changeme \ -Dtests.cluster.url=https://XYZ.es.io:9243
Or even easier, you can use the
Cloud ID available on you Cloud Console:
mvn verify -pl fr.pilato.elasticsearch.crawler:fscrawler-it-v7 \ -Dtests.cluster.user=elastic \ -Dtests.cluster.pass=changeme \ -Dtests.cluster.cloud_id=fscrawler:ZXVyb3BlLXdlc3QxLmdjcC5jbG91ZC5lcy5pbyQxZDFlYTk5Njg4Nzc0NWE2YTJiN2NiNzkzMTUzNDhhMyQyOTk1MDI3MzZmZGQ0OTI5OTE5M2UzNjdlOTk3ZmU3Nw==
Using security feature¶
Integration tests are run by default against a standard Elasticsearch cluster, which means with no security feature activated.
New in version 2.7.
You can run all the integration tests against a secured cluster by using the
mvn verify -Psecurity
Note that secured tests are using by default
changeme as the password.
You can change this by using
mvn verify -Psecurity -Dtests.cluster.pass=mystrongpassword
Some options are available from the command line when running the tests:
tests.leaveTemporaryleaves temporary files after tests.
tests.parallelismhow many JVM to launch in parallel for tests. Set to
- which means that it depends on the number of processors you have.
tests.outputwhat should be displayed to the console while running tests. By default it is set to
onErrorbut can be set to
tests.seedif you need to reproduce a specific failure using the exact same random seed
tests.timeoutSuitehow long a single can run. It’s set by default to
600000which means 5 minutes.
tests.localeby default it’s set to
randombut you can force the locale to use.
tests.timezoneby default it’s set to
randombut you can force the timezone to use.
mvn install -rf :fscrawler-it -Dtests.output=always
Check for vulnerabilities (CVE)¶
The project is using OSS Sonatype service to check for known
vulnerabilities. This is ran during the
Sonatype provides this service but with a anonymous account, you might be limited by the number of tests you can run during a given period.
If you have an existing account, you can use it to bypass this limit for anonymous users by
mvn verify -DskipTests \ -Dsonatype.firstname.lastname@example.org \ -Dsonatype.password=yourverysecuredpassword
If you want to skip the check, you can run with
mvn clean install -Dossindex.fail=false