Installation & Setup
The RO-Crate Validation Service works in two ways: a metadata-only mode, in which the contents of an ro-crate-metadata.json file are assessed, and storage-backed validation, where complete RO-Crates (zip or directory) are evaluated.
Quick start
You will need Docker with Docker Compose.
To start, clone the repository, copy the example environment file (example.env), and start the stack:
git clone https://github.com/eScienceLab/RO-Crate-Validation-Service.git
cd RO-Crate-Validation-Service
cp example.env .env
docker compose up
Warning
Remember to update the default .env values when running the object store in production.
The API is served at http://localhost:5001. Redis and a Celery worker are also started, but these are only used once storage is enabled.
To check the service is up, run:
curl http://localhost:5001/healthz
This returns {"status": "ok"}.
To validate the contents of an ro-crate-metadata.json file, post to the metadata endpoint. The running example from the RO-Crate specification is a good test document. Create an ro-crate-metadata.json file and copy the text from the link above into this, for use in the example below.
The file contents need to be sent to the API as an escaped JSON string, identified using the crate_json tag, within a JSON object. The command-line JSON processor, jq can be used to do this, as shown below.
jq -Rs '{crate_json: .}' ro-crate-metadata.json | curl -X POST http://localhost:5001/v1/ro_crates/validate_metadata -H 'Content-Type: application/json' -d @-
The returned response will contain a status of valid, invalid or error, along with the detailed validation. For more information, the API reference describes the endpoints and result format in full.
Enabling object storage
To enable the validation of complete RO-Crates (zip or directory) that are held in an object store, set STORAGE_ENABLED=true in .env.
The storage-backed validation mode requires six environmental variables to be set in the .env file, described below: S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, S3_BUCKET, CELERY_BROKER_URL and CELERY_RESULT_BACKEND. The service will fail at startup if any are missing.
The CELERY_BROKER_URL and CELERY_RESULT_BACKEND are already configured to the bundled Redis service within the docker compose stack, so the values for these within the example.env file can be left as they are. For any purpose other than an initial demonstration of the service the S3 settings given in the example.env file, S3_ENDPOINT, S3_ACCESS_KEY, S3_SECRET_KEY, and S3_BUCKET, should be changed for both security reasons and to match your own local setup.
To start the stack with the bundled development object store (RustFS), run:
docker compose --profile objectstore up
RustFS serves the S3 API on port 9000 and a web console at http://localhost:9001.
Warning
The service does not create the bucket itself. Create the bucket in the console or with an S3 client. The bucket name needs to match the value in the S3_BUCKET environmental variable (ro-crates by default).
Upload an RO-Crate to this S3 bucket, using the prefix crates. This should give the uploaded RO-Crate a path of either crates/<id>.zip for a zipped RO-Crate, or crates/<id>/ for a directory. Note that for a zipped RO-Crate, ro-crate-metadata.json must be at the root of the archive.
The readiness endpoint can be used to check the object store and broker connections:
curl http://localhost:5001/readyz
Using your own object store
Any S3-compatible store can be used in place of RustFS, including AWS S3, MinIO and Ceph. To do this set the S3_ENDPOINT and S3_BUCKET environment variables to match the location of your store, and provide your store credentials in the S3_ACCESS_KEY and S3_SECRET_KEY environment variables. Make sure that the STORAGE_ENABLED environment variable is still set to true, but do not include the objectstore profile flag in your docker compose command:
docker compose up
If you already use a 1.x release against MinIO, the upgrade guide maps the old settings to the new ones.
Configuration reference
The following can all be set as environment variables for the service using an .env file.
| Variable | Default | Description |
|---|---|---|
STORAGE_ENABLED |
false |
Enable the stored-crate endpoints and storage checks |
S3_ENDPOINT |
— | Object store endpoint, e.g. objectstore:9000 (required in storage mode) |
S3_ACCESS_KEY |
— | Object store access key (required in storage mode) |
S3_SECRET_KEY |
— | Object store secret key (required in storage mode) |
S3_BUCKET |
— | Bucket holding RO-Crates and results (required in storage mode) |
S3_USE_SSL |
false |
Use HTTPS for connecting to the object store |
S3_REGION |
— | Region; needed when using AWS S3 object stores |
S3_CRATE_PREFIX |
crates |
Prefix key from which RO-Crates are read |
S3_RESULTS_PREFIX |
validation-results |
Prefix key to which results are written |
CELERY_BROKER_URL |
— | Redis broker URL (required in storage mode; preset in the Compose stack) |
CELERY_RESULT_BACKEND |
— | Celery result backend URL (required in storage mode; preset in the Compose stack) |
PROFILES_PATH |
— | Profiles directory for replacing the bundled profiles |
EXTRA_PROFILES_PATH |
— | Profiles directory for adding extra profiles |
CACHE_PATH |
/app/.rocrate-cache |
Validator HTTP cache location |
VALIDATION_OFFLINE |
false |
Validate using only the cache, with no network access |
FLASK_ENV |
development |
Set to production to disable debug behaviour |
Custom profiles
The validator comes with several RO-Crate profiles, and for the Five Safes RO-Crate, the prebuilt ghcr.io/esciencelab/ro-crate-validation-service-fivesafes-profile image has the five-safes-crate profile already included; see Five Safes validation.
Other profiles can be provided by mounting the directory containing these profiles as a volume for the flask container. Mount the same directory as a volume for the celery_worker container as well if you have enabled stored-crate validation. Then set either the EXTRA_PROFILES_PATH or PROFILES_PATH environment variable to match the volume path. There is a working example in docker-compose-develop.yml.
Note
EXTRA_PROFILES_PATH adds the directory to the bundled profiles, whereas PROFILES_PATH replaces them entirely. The two can be set together, in which case the validator takes profiles from both locations.
Offline validation
The validator fetches profile and context resources over HTTP and caches them. The published v2.* images pre-populate this cache at build time, so setting VALIDATION_OFFLINE=true runs validation entirely from the cache, with no network access at runtime. This is useful inside TREs and other networks with restricted internet access.
Online validation (the default) also uses and refreshes the same cache. Offline validation requires rocrate-validator at 0.10.0 or later, which the published v2.* ro-crate validation service images include.