Set up automatic software publishing

Note

This tutorial works for repositories hosted on GitHub or GitLab, and shows how to automatically publish to Zenodo Sandbox. Zenodo Sandbox is a “toy” repository that can be used to try things out. This tutorial also works with the “real” Zenodo.

Automated Setup using hermes init

Following the automated initialization process will take you about 5 minutes start to finish. If you need to set up accounts as well, add +5 minutes. If you need to create a project-specific CITATIONS.cff file via cffinit, add another +5 to +15 minutes, depending on the level of detail you require. Installing Python and Git might take you another +15 minutes. You should be done in about 5 to 40 minutes.

Note

This also works with many Jupyter Hubs, no need for a Linux computer! (Try using one hosted by HIFIS, Google Colab or within your home institution)

1. To use the automated setup you need Python to be installed on your device. If it is not installed, you can download Python here.

2. Open a terminal or your favourite shell. Don’t worry, we don’t do anything fancy here.

3. Use Git to clone a working copy of your project onto your computer or other machine with access to a shell. (You can skip this if you have done this already, or you can use a GUI Git client.) For example:

     git clone https://github.com/my/project myproject
   

4. Now use pip to install HERMES:

     python -m pip install hermes
   

5. Once installed, navigate to the main directory of your project.

     cd myproject/
   

6. From here you can start the automated setup by running:

     hermes init
   

   Now simply follow the steps through the setup.

7. When you’re done, don’t forget to add, commit and push the changes and new files you just made!

Important

Be advised: during the setup, you can choose to grant access to all of your projects on your social coding platform. The HERMES setup requires this to speed up the automated setup process for you. It is perfectly safe to revoke this access once you’re done. You can still opt out and do some of the steps manually.

Manual setup

If using Python is not an option, you can set up the HERMES workflow manually for your git project by following the steps below.

Configure your .gitignore

The HERMES workflow (hermes) uses temporary caches in .hermes/. Ignore this directory in your git repository.

Add .hermes/ to your .gitignore file:

.gitignore

.hermes/

Provide additional metadata using CITATION.cff

To provide high-quality citation metadata for your project and your publication, provide a CITATION.cff file in the Citation File Format.

If you don’t have one yet, use the cffinit website to create a CITATION.cff file.

Important

Make sure that you define the license for your repository in the CITATION.cff file! Otherwise, Zenodo Sandbox will not accept your publication as open access.

Save the CITATION.cff file to the root directory of your repository, and add it to version control.

git add CITATION.cff
git commit -m "Add citation file"
git push

HERMES configuration

The HERMES workflow is configured in a TOML file: hermes.toml. Each step in the publication workflow has its own section.

Configure HERMES to:

• harvest metadata from Git and CITATION.cff

• deposit on Zenodo Sandbox (which is built on the InvenioRDM)

• use Zenodo Sandbox as the target publication repository

hermes.toml

[harvest]
sources = [ "cff" ]

[deposit]
target = "invenio_rdm"

[deposit.invenio_rdm]
site_url = "https://sandbox.zenodo.org"
access_right = "open"

Copy this file to the root directory of your repository and add it to version control:

git add hermes.toml
git commit -m "Configure HERMES to harvest git and CFF, and deposit on Zenodo Sandbox"
git push

Note

If you decide to start from an existing hermes.toml (e.g., the one found in this repository), be sure that there is no record_id value defined in the deposit.invenio_rdm section. Otherwise, your deposition will fail as hermes would try to deposit a new version for the given record.

Get a personal access token for Zenodo (Sandbox)

To allow GitHub Actions to publish your repository to Zenodo (Sandbox) for you, you need a personal access token from Zenodo (Sandbox).

Log in at Zenodo Sandbox or Zenodo (you may have to register first).

Then create a new personal access token in your Zenodo Sandbox user profile or in your Zenodo user profile with the scopes deposit:actions and deposit:write.

Keep the generated token somewhere from where you can easily retrieve it later on (e.g., a password safe).

Configure continuous integration build to use hermes

The following instructions differ depending on whether you are using GitHub actions or GitLab CI for your development workflow.

To support this, the HERMES project provides templates for continuous integration systems in a dedicated repository: hermes-hmc/ci-templates.

Configure a GitHub Action to automate publication

Copy the Zenodo sandbox token you just created into a new GitHub Secret called ZENODO_SANDBOX in your repository.

Copy the template file for GitHub to Zenodo Sandbox publication into the .github/workflows/ directory in your repository, and rename it as you like (e.g. hermes_github_to_zenodo.yml).

Go through the file, and look for comments marked with # ADAPT. Adapt the file to your needs.

If you need help with how GitHub Action workflows work in general, have a look at the Github Actions documentation.

Add the workflow file to version control and push it:

git add .github/workflows/hermes_github_to_zenodo.yml
git commit -m "Configure automatic publication with HERMES"
git push

Note

If you haven’t adapted the workflow file and push it to the branch main, the HERMES workflow will run for the first time. This will create a new pull request with compiled metadata for curation. You can safely close it at this point.

Allow GitHub Actions to create pull requests in your repository

The automated HERMES workflow will not publish any metadata without your approval. It will instead create a pull request for you to approve or change the metadata that is being deposited. To do this, GitHub Actions needs to be able to create pull requests in your repository.

To enable this, go to Settings > Actions > General in your repository, and activate the option “Allow GitHub Actions to create and approve pull requests” in the section Workflow permissions.

Configure GitLab CI to automate publication

Copy the Zenodo sandbox token you just created into a new GitLab CI variable called ZENODO_TOKEN.

For Gitlab you also need the HERMES Push Token. That Token gives access to the project in order for HERMES to create Merge Requests. Therefore, you create an access token in your project. The Token needs to have at least the developer role and write access (e.g. write_repository scope). Then you create a Gitlab CI variable with the token called HERMES_PUSH_TOKEN.

Note

The two Gitlab CI Variables include sensitive and powerful information. Therefore you should at least select the flag Mask variable when creating. If possible you should also select the flag Protect variable and define all branches hermes/* as protected branch.

Copy the template file for GitLab to Zenodo Sandbox publication into your project to .gitlab/hermes-ci.yml.

To integrate the workflows into your .gitlab-ci.yml, you need to add the following:

.gitlab-ci.yml

# Use jobs defined by the HERMES GitLab template  
include:
  - .gitlab/hermes-ci.yml

# Run the first half of the pipeline, resulting in a Merge Request for curation
hermes_curate:
  extends:
    - .hermes_curate

# Run the second half, resulting in a new deposition and a Merge Request with post-processing results
hermes_deposit:
  extends:
    - .hermes_deposit

If you need help with how GitLab CI works in general, have a look at the GitLab CI documentation.

Add the template file and changes in the gitlab-ci.yml to version control and push it:

git add .gitlab/hermes-ci.yml
git add .gitlab-ci.yml
git commit -m "Configure automatic publication with HERMES"
git push

By default, the first part of the workflow (i.e., .hermes_curate) will run on every new push during the deploy stage.

Note

If you haven’t adapted the workflow file and push it to the branch main, the HERMES workflow will run for the first time. This will create a new merge request with compiled metadata for curation. You can safely close it at this point.

Congratulations

Now the HERMES workflow is ready to run. If you haven’t adapted the workflow file, it will be triggered whenever you push to your main branch.

Automatic Publication Workflow Details

flowchart TD t(("Trigger\nGitHub Action\n(e.g. push)")) rh("Run hermes") subgraph "hermes (harvest, process, curate)" ci1("Harvest & process metadata") pr1("Create curation pull request") end d{"Merge?"} subgraph "hermes (deposit, postprocess)" ci2("Publish software") pp("Postprocessing") pr2("Create pull request\nfrom postprocessing") ci3("Cleanup") end e((("End"))) t --> rh --> ci1 --> pr1 --> d -->|Merge| ci2 --> pp --> pr2 --> e d -->|Close| ci3 --> e

The diagram to the right shows the different steps that will happen each time.

When the workflow runs, it harvests and processes the metadata from Git and your CITATION.cff file, and creates a new pull request in your repository. You then have the chance to curate the metadata, i.e., make sure that it looks the way you want. If you merge this pull request, a new publication is created on Zenodo Sandbox. In postprocessing, a new pull request is opened to update the HERMES configuration file in your repository with the ID of the publication. This is needed so that future published versions are collected under the same concept DOI.