Welcome to AMP-SCZ Lochness’s documentation!

Introduction

Lochness is a data management tool designed to periodically pull and download data from various data archives into a local directory. This is often referred to as building a “data lake” (hence the name). This repository is a fork from harvard-nrg lochness, updated and maintained by DPACC for AMP-SCZ project.

AMP-SCZ Lochness has a number of extra functions, many of which are specific to PRONET and PRESCIENT research networks within the AMP-SCZ. However, extra functions to pull from the additional data sources such as Box, Mediaflux, RPMS, and Mindlamp could be useful for other studies as well.

The main difference between the AMP-SCZ Lochness verses the original harvard-nrg Lochness, is that AMP-SCZ Lochness requies a database of unique subject IDs in either REDCap or RPMS, for lochness to automatically create a list of subject IDs to be used with further down mechanisms of Lochness (metadata.csv, which will be used in searching for any data to download). So, if your project also has either REDCap or RPMS as the main database for keeping the list of subjects involved in the study, AMP-SCZ Lochness could also be useful and linked to your project!

List of supported data sources are

• REDCap or

• RPMS

• XNAT

• Box

• Mediaflux

• Mindlamp

Note

Currently, REDCap or RPMS data sources are the requirements for using lochness, but future update will allow AMP-SCZ Lochness to pull data without them, by allowing manual creation of the metadata.csv file.

Please report any bug or issue to our github repository. Thanks!

Contents

• Installation
  • Requirement
  • AMP-SCZ Lochness
  • Amazon Web Service (AWS) commandline tool
  • mailx for sending out emails
• Setting up Lochness
  • 1. Create a template directory
  • 2. Edit credentials in the template keyring file
  • 3. Encrypt lochness.json to make a keyring file
  • 4. Edit configuration file
• Running lochness
  • Run sync.py
• Example PHOENIX-BIDS structure
  • Summary of the structure
  • Different levels of the structure
• Lochness sync.py function in detail
  • Loads configuration file, then keyring file
  • Creates and updates metadata for each site
  • Pull data for each subject in metadata.csv
  • Transfer selected data to s3 bucket
• Other shell functions available in lochness
  • lochness_check_config.py
  • phoenix_generator.py
  • listen_to_redcap.py
  • lochness_create_template.py
  • send_email_update.py
• Configuration file
  • keyring_file
  • phoenix_root
  • stdout
  • stderr
  • poll_interval
  • ssh_user
  • ssh_host
  • sender
  • mindlamp_days_to_pull
  • pii_table
  • lochness_sync_history_csv
  • redcap_id_colname and redcap_consent_colname
  • RPMS_PATH, RPMS_id_colname, and RPMS_consent_colname
  • AWS_BUCKET_NAME and AWS_BUCKET_ROOT
  • s3_selective_sync
  • redcap
  • beiwe
  • dropbox
  • box
  • mediaflux
  • redcap
  • admins
  • notify
• Data sources
  • REDCap
  • RPMS
  • XNAT
  • Box
  • Mediaflux
  • Mindlamp

Indices and tables

• Index

• Module Index

• Search Page