rse.main.database package

Submodules

rse.main.database.base module

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.base.Database[source]

Bases: object

A rse database holds common functions to represent repos, and metadata.

add(uid)[source]: Add a new software repository to the database.

clear()[source]: clear (delete) all software repositories.

database = 'notimplemented'

delete_parser(name)[source]: delete all repos for a parser based on the parser name.

delete_repo(uid)[source]: delete a repo based on a specific identifier.

exists()[source]: determine if a software repository exists in the database.

get()[source]: get a software repository from the database.

get_or_create(uid)[source]: Determine if a repo exists.

list_repos(name=None)[source]: list software repositories, either under a particular parser name or just under all parsers.

search(query)[source]: search is only available to non-filesystem databases

update(uid)[source]: update a software repository.

rse.main.database.filesystem module

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.filesystem.FileSystemDatabase(config_dir, config=None, **kwargs)[source]

Bases: rse.main.database.base.Database

A FileSystemDatabase writes raw json to files to a database.

add(uid)[source]: Add a new software repository to the database.

clear()[source]: clear (delete) all software repositories.

create_database(config_dir)[source]: Create the database. The parent folder must exist.

database = 'filesystem'

delete_parser(name)[source]: delete all repos for a parser, based on executor’s name (str).

delete_repo(uid)[source]: delete a repo based on a specific identifier.

exists(uid)[source]: Determine if a repo exists.

get(uid=None, exact=False)[source]: Get a software repo based on a uid. If exact is not needed, we can search for a match based on the partial uid. If exact is False, and a uid is not provided, get the last repository created.

get_or_create(uid)[source]: Determine if a repo exists.

iter_parsers(fullpath=False)[source]: list executors based on the subfolders in the base database folder.

label(repo, key, value, force=False)[source]: Update a repository with a specific key/value pair.

list_repos(name=None)[source]: list software repositories, either under a particular parser name or just under all parsers. This returns repos in rows to be printed (or otherwise parsed).

search(query, taxonomy=None, criteria=None)[source]: A filesystem search can only support returning results with filenames. For taxonomy and criteria items, we load them and search. We organize results based on the query, taxonomy, and criteria The results returned are separate (e.g., a single repo can appear in more than one list).

update(repo, rewrite=False)[source]: Update a repository by retrieving metadata, and then calling update on the software repository to save it.

class rse.main.database.filesystem.SoftwareRepository(parser, data_base, exists=False)[source]

Bases: object

A software repository is a filesystem representation of a repo. It can take a uid, determine if the repo exists, and then interact with the metadata for it. If the repo is instantiated without a unique id it is assumed to not exist yet, otherwise it must already exist.

property avatar

create(should_exist=False)[source]: create the filename if it doesn’t exist, otherwise if it should (and does not) exit on error.

property description

export()[source]: wrapper to expose the executor.export function

property filename

get_criteria()[source]: Get loaded criteria

get_taxonomy()[source]: Get loaded taxonomy

has_criteria_annotation(uid, username)[source]: Determine if a repository has been annotated by a user.

has_taxonomy_annotation(username)[source]: Determine if a repository has been annotated by a user.

load()[source]: Given a software uid, load data from filename.

load_criteria()[source]: Given a repository directory, load criteria files if they exist

load_taxonomy()[source]: Given a repository directory, load taxonomy annotations if they exist The taxonomy.tsv file should be a tab separated file with: username category-unique-id. This means that we keep a record of who has categorized what, and load this information into the taxonomy dictionary (organized by the category-unique-id which then has a total count and list of users).

property parser_dir

save()[source]: Save a json object (metadata.json) for the software repository.

save_criteria()[source]: Save criteria to file. Each file is named based on the criteria id, and is a tab separated file that includes the username and response.

save_taxonomy()[source]: Save taxonomy to file. Each file is named taxonomy.tsv, and is a tab separated file that includes the username and response.

summary()[source]

update(updates=None)[source]: Update a data file. This means reading, updating, and writing.

update_criteria(uid, username, response)[source]: Update a criteria, meaning adding a True/False answer to the unique id for the user. We are currently assuming that criteria have yes/no responses, and True == yes, False == no.

property url

rse.main.database.models module

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.models.SoftwareRepository(uid=None, parser=None, data={})[source]

Bases: sqlalchemy.orm.decl_api.Base

A software repository.

property avatar

criteria

data

property description

export()[source]: Export removes the outer wrapper, and just returns the data

get_criteria()[source]: load criteria into a dictionary

get_taxonomy()[source]: load taxonomy into a dictionary

has_criteria_annotation(uid, username)[source]: Determine if a repository has been annotated by a user.

has_taxonomy_annotation(username)[source]: Determine if a repository has been annotated by a user.

load()[source]: loading a software repository means exporting as json

parser_name

summary()[source]

taxonomy

timestamp

uid

update_criteria(uid, username, response)[source]: Given a username and unique id update criteria

update_taxonomy(username, uids)[source]: Given a username and unique id update taxonomy items

property url

rse.main.database.relational module

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.relational.RelationalDatabase(config_dir, config=None, **kwargs)[source]

Bases: rse.main.database.base.Database

A RelationalDatabase is a more robust relational datbase (to sqlite). Since the global database property can be any of postgresql, mysql+pysq;, it is defined on init. The sqlite database also uses this class, but defines a custom init function to handle the rse.db file.

add(uid)[source]: Create a new repo based on a uid that matches to a parser.

clear()[source]: clear (delete) all repos. This could be improved to cascade instead.

create_database()[source]: create the databsae based on the string, whether it’s relational or sqlite. self.db must be defined.

delete_parser(name)[source]: delete all repos for a parser, based on parser’s name (str).

delete_repo(uid)[source]: delete a repo based on a specific repo id.

exists(uid)[source]: Determine if a repo exists.

get(uid=None)[source]: Get a repo based on a uid. Exits on error if doesn’t exist. If a uid is not provided, get the last updated repository.

get_or_create(uid)[source]: Determine if a repo exists.

label(repo, key, value, force=False)[source]: Update a repository with a specific key/value pair.

list_repos(name=None)[source]: list repos, either under a particular parser name (if provided) or just the parsers.

search(query, taxonomy=None, criteria=None)[source]: Search across the database for a particular query.

update(repo, updates=None, rewrite=False)[source]: update a repo with a json dictionary.

rse.main.database.sqlite module

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

class rse.main.database.sqlite.SqliteDatabase(config_dir, config=None, **kwargs)[source]

Bases: rse.main.database.relational.RelationalDatabase

A SqliteDatabase writes to a rse.db file in $HOME/.rse. This is the suggested database backend for QueueMe, as it doesn’t require anything beyond a filesystem and still allows for relational type queries.

database = 'sqlite'

Module contents

This Source Code Form is subject to the terms of the Mozilla Public License, v. 2.0. If a copy of the MPL was not distributed with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

rse.main.database.init_db(database, config_dir=None, database_string='', config=None)[source]: Initialize the database, meaning a base client and appropriate functions to save, or generate a unique ID based on the backend being used. Each client has it’s own init to check for a connection (or filesystem path existence) and then functions to interact with entities.