You might want to interact with a software database via Python.

Why would I want to do that?

You might want to contribute repositories to a database, or just use it for your own analysis. In this walkthrough, we will clone the the database, checkout a new branch, and then make changes that we would intend to do a pull request to contribute back.

Get the Database

While you could start from scratch and create your own database with rse init and then rse add, it’s more likely you’ll want to start with an already existing base. Let’s do that by cloning:

git clone
cd software

You will notice an rse.ini file that has a basic configuration for the filesystem. While other formats are supported, because we want to keep this database in version control, a filesystem format is optimal. Once we have our repository cloned, we can start a shell to interact with it:

$ rse shell
Out[1]: <rse.main.Encyclopedia at 0x7fb543b25590>

And remember that you can always export the full path to an rse.ini config file, in the case that you want to be able to run rse shell from anywhere:

export RSE_CONFIG_FILE=/home/vanessa/Desktop/Code/rseng/software/rse.ini


All of the commands that are available on the command line (and a few more)! are available to you.


Does a particular software repository exists in your database?

$ client.exists("")

$ client.exists("")


Let’s say that we tested if our software of interest existed in the repository, and then we found that it did not, and want to add it. We can use rse add to do this.

$ client.add("")
INFO:rse.main.database.relational:github/sci-f/scif-go was added to the the database.

Ah! But don’t forget that you need to export your RSE_GITHUB_TOKEN. If the entry already exists, you will be told that!

$ client.add("") already exists in the database.

And if the entry doesn’t exist on the remote (e.g., GitHub) you’ll see:

$ client.add("")
ERROR:rse.main.parsers.github:Cannot find repository singularityhub/pancakes.

We can also add in bulk from a text file with one repository per line. For example, here is a small one:

We would add in bulk as follows:

$ client.bulk_add("repos.txt")

By default, repos that are already added will be skipped over.


A get will retrieve a known identifier. Unlike exists, if it doesn’t exist, it will parse and retrieve it for you.

$ repo = client.get("")
# <SoftwareRepository 'github/singularityhub/sregistry'>

You can then inspect, export, or otherwise interact with the repository instance.

If you run get without an argument, it will retrieve the last modified entry for you:

$ client.get()
# <SoftwareRepository 'github/sci-f/scif-go'>


Updating an entry coincides with retriving updated metadata. You can do this for an existing software repository:

> client.update("")
INFO:rse.main:github/singularityhub/sregistry has been updated.
# <SoftwareRepository 'github/singularityhub/sregistry'>

And of course you cannot update an entry that doesn’t exist.

$ client.update("") does not exist.

We can also update in bulk from a text file with one repository per line. For example, here is a small one:

We would update in bulk as follows:

$ client.bulk_update("repos.txt")

By default, repos that are not present will be skipped over.


We can easily list repos with list.

> client.list()

You can also list a particular parser:

> client.list("github")


If you want to delete a software entry, just use clear with it’s unique id:

> client.clear("")
This will delete software, are you sure? [n]|y: y

If you don’t want the prompt, add noprompt=True

> client.clear("", noprompt=True)

You can also remove an entire parser:

> client.clear("github")
This will delete all github software in the database, are you sure? [n]|y: y

or all software repositories in the database:

> client.clear()
This will delete all software in the database, are you sure? [n]|y: y

Unless you set noprompt to True, each time you’ll be asked for a confirmation first, in case the command was run in error.

We can easily search across our software repos with search. For a filesystem database, this means only the filenames.

[['github/singularityhub/sregistry', 'github', '2020-06-09 17:59:41'],
 ['github/hpcng/singularity', 'github', '2020-06-09 19:44:04']]

Depending on your database backend, you might retrieve more metadata (the above is for the sqlite backend).


The criteria that we use to populate the client is the present version available from []. If you need an earlier verison, you can interact with the rseng library directly. To do this with the client here, you can simply list criteria:

> client.list_criteria()
[{'uid': 'RSE-absence',
  'name': 'Would taking away the software be a detriment to research?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'},
 {'uid': 'RSE-citation',
  'name': 'Has the software been cited?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'},
 {'uid': 'RSE-domain-intention',
  'name': 'Is the software intended for a particular domain?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'},
 {'uid': 'RSE-question-intention',
  'name': 'Was the software created with intention to solve a research question?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'},
 {'uid': 'RSE-research-intention',
  'name': 'Is the software intended for research?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'},
 {'uid': 'RSE-usage',
  'name': 'Has the software been used by researchers?',
  'options': ['yes', 'no'],
  'date': '2020-06-13 15:04:25 +0000'}]


You can also list a flattened version of the taxonomy from from [].

> client.list_taxonomy()
[{'uid': 'RSE-taxonomy-analysis',
  'name': 'Domain-specific analysis software',
  'example': 'SPM, fsl, afni for neuroscience',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-application-programming-interfaces',
  'name': 'Application Programming Interfaces',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-communication-tools',
  'name': 'Communication tools or platforms',
  'example': 'email, slack, etc.',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-data-collection',
  'name': 'Data collection',
  'example': 'web-based experiments or portals',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-databases',
  'name': 'Databases',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-domain-hardware',
  'name': 'Domain-specific hardware',
  'example': 'software for physics to control lab equipment',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-frameworks',
  'name': 'Frameworks',
  'example': 'to generate documentation, content management systems',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-ide-research',
  'name': 'Interactive development environments for research',
  'example': 'Matlab, Jupyter',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-numerical libraries',
  'name': 'Numerical libraries',
  'example': 'includes optimization, statistics, simulation, e.g., numpy',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-operating-systems',
  'name': 'Operating systems',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-optimized',
  'name': 'Domain-specific optimized software',
  'example': 'neuroscience software optimized for GPU',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-personal-scheduling-task-management',
  'name': 'Personal scheduling and task management',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-provenance-metadata-tools',
  'name': 'Provenance and metadata collection tools',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-text-editors-ides',
  'name': 'Text editors and integrated development environments',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-version-control',
  'name': 'Version control',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-visualization',
  'name': 'Visualization',
  'example': 'interfaces to interact with, understand, and see data, plotting tools',
  'date': '2020-06-13 14:48:53 +0000'},
 {'uid': 'RSE-taxonomy-workflow-managers',
  'name': 'Workflow managers',
  'date': '2020-06-13 14:48:53 +0000'}]