Improving computational workflows in Galaxy by maturing Pulsar Network and publishing computational workflows as FAIR Digital Objects.

Motivation

In the past decade, many scientific domains have been transformed into data-driven disciplines relying on the exchange and integration of internationally distributed data. Exploiting this data is still a laborious and largely manual task, prone to losses and errors, and increasingly specialised beyond most users technical capabilities. FAIR practices are encouraged but their adoption curve is steep. The needs for compute and data resources, tools, and application platforms are often domain-specific.

Many scientists struggle to navigate this intricate ecosystem. Generally, researchers do not possess the computing skills to effectively use the HPC or Cloud platforms they need. Thus, new approaches are needed to enable all researchers, with widely ranging digital skills, to efficiently use the diverse computational infrastructures available across Europe, for asynchronous and for interactive applications.

EuroScienceGateway

EuroScienceGateway will leverage a distributed computing network across 13 European countries, accessible via 6 national, user-friendly web portals, facilitating access to compute and storage infrastructures across Europe as well as to data, tools, workflows and services that can be customized to suit researchers’ needs.

EuroScienceGateway will deliver a robust, scalable, seamlessly integrated open infrastructure for data-driven research, contributing an innovative and customizable service for EOSC that enables operational open and FAIR data and data processing, empowering European researchers to embrace the new digital age of science.

EOSC integration

At the heart of the proposal workflows will integrate with the EOSC-Core. Adoption, development and implementation of technologies to interoperate across services, will allow researchers to produce high-quality FAIR data, available to all in EOSC. Communities across disciplines – Life Sciences, Climate and Biodiversity, Astrophysics, Materials science - will demonstrate the bridge from EOSC’s technical services to scientific analysis.

Funded by Horizon Europe call HORIZON-INFRA-2021-EOSC-01-04 (Enabling an operational, open and FAIR EOSC ecosystem), EuroScienceGateway will work closely with EOSC, EOSC projects including EOSC-Life, as well as established e-Infrastructures and life science communities like ELIXIR.

Galaxy and Pulsar Network

EuroScienceGateway will leverage the Galaxy platform (an open, web-based platform for data-intensive research), the Pulsar Network (a wide job execution system distributed to scale computing power over heterogeneous resources) and FAIR workflow services pioneered in the EOSC-Life cluster (WorkflowHub, Workflow RO-Crate and metadata standards like schema.org).

EuroScienceGateway will lift Pulsar from TRL-7 to TRL-9 by expanding the APIs, hardening deployments and adding support for different usage patterns, e.g. data localisation during job scheduling.

National Galaxy instances across Europe and other workflow management systems will be enabled to submit jobs to this distributed compute network. Pulsar will become a production-ready interface for European computing resources, including EGI and EuroHPC as Pulsar providers.

WP2: Stimulate FAIR and reusable research

The eScience Lab at The University of Manchester is leading the work package WP2 Stimulate FAIR and reusable research, contributing with our expertise on FAIR Computational Workflows).

This work package adds sufficient measures to EuroScienceGateway to support FAIR practices for and by workflows.

FAIR Digital Objects (FDOs), for exchanging, publishing, archiving and citing workflows and their companion data, provenance logs and associated resources, will be realized as RO-Crates.

The FDOs will be published through EOSC and in EOSC catalogues (e.g. OpenAIRE) and further communicated through outreach plans to increase uptake of the EuroScienceGateway in affiliated and newly targeted computational researcher communities.

WorkflowHub, a FAIR registry for workflows that is RO-Crate compliant, will be lifted to TRL-9 status (from TRL-7) and made the registry of choice for all workflow system types and for all disciplines in EOSC.

By collaborating with the project CSA INFRA-01-EOSC-01-05 (Enabling discovery and interoperability of federated research objects across scientific communities), both the WorkflowHub and Workflow Digital Objects will be further aligned with the PID and metadata schema frameworks.

WP2 Objectives

  • O2.1 Bringing FAIR workflows into EOSC through the EuroScienceGateway
  • O2.2 Support reusable and reproducible workflows
  • O2.3 Establish FAIR Digital Objects as citable exchange format for workflows for all EOSC services
  • O2.4 Establish FAIR Workflow Digital Objects as publishable scholarly objects

eScienceLab contributions

UK partners in Horizon Europe projects are funded through Innovate UK (#10038963) from the UKRI Horizon Europe guarantee.

  • WP2: Stimulate FAIR and reusable research (lead: UNIMAN)
    • Task T2.1: Integration of EuroScienceGateway in EOSC (lead: UNIMAN)
    • Task T2.2: Reproducible and reusable FAIR Digital Objects (lead: UNIMAN)
    • Task T2.3: Using and enriching workflow FDOs (lead: UNIMAN)
    • Task T2.4: FAIR workflows as scholarly objects in scientific publishing (lead: EPFL)
    • Deliverable D2.1 Reproducible FAIR Digital Objects for workflows (lead: UNIMAN)
    • Deliverable D2.2 Publishing workflow enriched FDOs to EOSC (lead: UNIMAN)
  • WP3: Pulsar Network: Distributed heterogeneous compute (lead: CNR)
    • Task T3.4 Add TES support to WfExS (Workflow Execution Service) (lead: BSC)-

Relevant Deliverables

Stian Soiland-Reyes, Björn Grüning, Paul De Geest (2025):
EuroScienceGateway MS3: Initial EuroScienceGateway workflows registered.
Zenodo
https://doi.org/10.5281/zenodo.10728922

Stian Soiland-Reyes, Eli Chadwick, Finn Bacall, José M Fernández, Björn Grüning, Hakan Bayındır (2024):
EuroScienceGateway D2.1: Reproducible FAIR Digital Objects for Workflows Creators.
Zenodo
https://doi.org/10.5281/zenodo.13225792

Stian Soiland-Reyes, Eli Chadwick, Armin Dadras, Björn Grüning, Catalin Condurache, Sebastian Luna-Valero, Volodymy Savchenko (2025):
EuroScienceGateway D2.2: Publishing workflow enriched FDOs to EOSC.
Zenodo
https://doi.org/10.5281/zenodo.15094824

Relevant Publications

Sean R. Wilkinson, Johan Gustafsson, Finn Bacall, Khalid Belhajjame, Salvador Capella, Jose Maria Fernandez Gonzalez, Jacob Fosso Tande, Luiz Gadelha, Daniel Garijo, Patricia Grubel, Bjorn Grüning, Farah Zaib Khan, Sehrish Kanwal, Simone Leo, Stuart Owen, Luca Pireddu, Line Pouchard, Laura Rodríguez-Navas, Beatriz Serrano-Solano, Stian Soiland-Reyes, Baiba Vilne, Alan Williams, Merridee Ann Wouters, Frederik Coppens, Carole Goble (2025):
An Ecosystem of Services for FAIR Computational Workflows.
Book chapter, draft
arXiv 2505.15988 [cs.DC]
https://doi.org/10.48550/arXiv.2505.15988

Frédéric Suter, Tainã Coleman, İlkay Altintaş, Rosa M. Badia, Bartosz Balis, Kyle Chard, Iacopo Colonnelli, Ewa Deelman, Paolo Di Tommaso, Thomas Fahringer, Carole Goble, Shantenu Jha, Daniel S. Katz, Johannes Köster, Ulf Leser, Kshitij Mehta, Hilary Oliver, J.-Luc Peterson, Giovanni Pizzi, Loïc Pottier, Raül Sirvent, Eric Suchyta, Douglas Thain, Sean R. Wilkinson, Justin M. Wozniak, Rafael Ferreira da Silva (2025):
A Terminology for Scientific Workflow Systems.
Future Generation Computer Systems 174:107974
https://doi.org/10.1016/j.future.2025.107974

Ove Johan Ragnar Gustafsson, Sean R. Wilkinson, Finn Bacall, Stian Soiland-Reyes, Simone Leo, Luca Pireddu, Stuart Owen, Nick Juty, José Mª Fernández, Tom Brown, Hervé Ménager, Björn Grüning, Salvador Capella-Gutierrez, Frederik Coppens, Carole Goble (2025):
WorkflowHub: a registry for computational workflows.
Scientific Data 12:837
https://doi.org/10.1038/s41597-025-04786-3

Stian Soiland-Reyes, Peter Sefton, Simone Leo, Leyla Jael Castro, Claus Weiland, Herbert Van de Sompel (2025):
Practical webby FDOs with RO-Crate and FAIR Signposting: Experiences and lessons learned.
International FAIR Digital Objects Implementation Summit 2024, Berlin, Germany, 2024-03-20/–21.
Open Conference Proceedings 5
https://doi.org/10.52825/ocp.v5i.1273

Sean Wilkinson, Meznah Aloqalaa, Khalid Belhajjame, Michael R. Crusoe, Bruno de Paula Kinoshita, Luiz Gadelha, Daniel Garijo, Ove Johan Ragnar Gustafsson, Nick Juty, Sehrish Kanwal, Farah Zaib Khan, Johannes Köster, Karsten Peters-von Gehlen, Line Pouchard, Randy K. Rannow, Stian Soiland-Reyes, Nicola Soranzo, Shoaib Sufi, Ziheng Sun, Baiba Vilne, Merridee A. Wouters, Denis Yuen, Carole Goble (2025):
Applying the FAIR Principles to Computational Workflows.
Scientific Data 12:328
https://doi.org/10.1038/s41597-025-04451-9

Alexander Hambley, Eli Chadwick, Oliver Woolland, Stian Soiland-Reyes, Volodymyr Savchenko (2024):
WorkflowHub Knowledge Graph. (Dataset)
Zenodo
https://doi.org/10.5281/zenodo.13362051

Eli Chadwick, Stian Soiland-Reyes, José María Fernández, Björn Grüning, Carole Goble (2025):
RO-Crate – Capturing FAIR research outputs throughout the ELIXIR landscape.
ELIXIR All Hands Meeting (AHM2025), Thessaloniki, Greece, 2025-06-02/–05.
F1000Research 14(ELIXIR):543 (poster)
https://doi.org/10.7490/f1000research.1120199.1

Paula Iborra, José M. Fernández, Laia Codó, Eugenio Gonzalo, Stian Soiland-Reyes, Alexander Hambley, Jonathan Couldridge, Carole Goble, Philip Quinlan, Salvador Capella-Gutierrez (2025):
ENTRUSTing WfExS 1.0 with WES and TES.
ELIXIR All Hands Meeting (AHM2025), Thessaloniki, Greece, 2025-06-02/–05.
F1000Research 14(ELIXIR):518 (poster)
https://doi.org/10.7490/f1000research.1120178.1

Finn Bacall, Stian Soiland-Reyes, Carole Goble, Stuart Owen, Johan Gustafsson, Frederik Coppens (2024):
WorkflowHub.
ELIXIR All Hands Meeting (AHM2024), Uppsala, Sweden, 2024-06-10/–12.
F1000Research 13(ELIXIR):1027 (poster)
https://doi.org/10.7490/f1000research.1119852.1

Eli Chadwick, Stian Soiland-Reyes (2024):
RO-Crate: package your research outputs with their metadata.
Research Software Engineering Conference (RSECon 2024), Newcastle, UK, 2024-09-03/–05.
https://research.manchester.ac.uk/en/publications/ro-crate-package-your-research-outputs-with-their-metadata

Carole Goble, Finn Bacall, Stian Soiland-Reyes, Stuart Owen, Ignacio Eguinoa, Bert Droesbeke, Hervé Ménager, Laura Rodriguez-Navas, José M. Fernández, Salvador Capella-Gutierrez, Michael R. Crusoe, Björn Grüning, Simone Leo, Luca Pireddu, Marco Enrico Piras, Johan Gustafsson, Phil Ewels, WorkflowHub Community, Frederik Coppens (2023):
WorkflowHub – a FAIR registry for workflows.
ELIXIR All Hands Meeting (AHM2023), Dublin, Ireland, 2023-06-05/–08.
https://doi.org/10.7490/f1000research.1119430.1

Simone Leo, Laura Rodríguez-Navas, José M. Fernández, Paul De Geest, Luca Pireddu, Michael R. Crusoe, Daniel Garijo, Iacopo Colonnelli, Raül Sirvent, Stian Soiland-Reyes (2023):
Making workflow provenance FAIR across workflow systems with Workflow Run RO-Crate.
Elixir All Hands 2023, 2023-06-06, Dublin, Ireland.
F1000Research 12(ELIXIR):592 (poster)
https://doi.org/10.7490/f1000research.1119445.1

Simone Leo, Michael R. Crusoe, Laura Rodríguez-Navas, Raül Sirvent, Alexander Kanitz, Paul De Geest, Rudolf Wittner, Luca Pireddu, Daniel Garijo, José M. Fernández, Iacopo Colonnelli, Matej Gallo, Tazro Ohta, Hirotaka Suetake, Salvador Capella-Gutierrez, Renske de Wit, Bruno de Paula Kinoshita, Stian Soiland-Reyes (2024):
Recording provenance of workflow runs with RO-Crate.
PLOS One 19(9):e0309210
https://doi.org/10.1371/journal.pone.0309210

Stian Soiland-Reyes, Carole Goble, Paul Groth (2024):
Evaluating FAIR Digital Object and Linked Data as distributed object systems.
PeerJ Computer Science 10:e1781
https://doi.org/10.7717/peerj-cs.1781 [RO-Crate]

Stian Soiland-Reyes, Leyla Jael Castro, Rohitha Ravinder, Claus Weiland, Jonas Grieb, Alexander Rogers, Christophe Blanchi, Herbert Van de Sompel (2024):
BioHackEU23 report: Enabling FAIR Digital Objects with RO-Crate, Signposting and Bioschemas.
BioHackrXiv
https://doi.org/10.37044/osf.io/gmk2h

José M. Fernández, Paula Iborra, Sébastien Moretti, Arun Isaac, Paul De Geest, Stian Soiland-Reyes (2024): BioHackEU23: FAIR Workflow Execution with WfExS and Workflow Run Crate.
BioHackrXiv https://doi.org/10.37044/osf.io/7f94w

Ignacio Eguinoa, Marek Suchánek, Vojtěch Knaisl, Jan Slifka, Paul De Geest, David López, Bjorn Gruning, Simone Leo, Stian Soiland-Reyes (2023):
BioHackEU22 Report: Enhancing Research Data Management in Galaxy and Data Stewardship Wizard by utilising RO-Crates.
BioHackrXiv
https://doi.org/10.37044/osf.io/24jst