Configuration file¶
The folling page will go over all of the configuration file sections, fields, and what each one does
keyring_file¶
This field specifies the location of your keyring file. This should be
a simple filesystem location
keyring_file: ~/.keyring.enc
phoenix_root¶
This field determines the root location of your PHOENIX filesystem. This should be a simple filesystem location
phoenix_root: /data/PHOENIX
stdout¶
This field determines the location of the Lochness process standard output
stdout: /logs/lochness.out
stderr¶
This field determines the location of the Lochness process standard error
stderr: /logs/lochness.err
poll_interval¶
This field determines the frequency at which Lochness will poll external data sources for incoming data (in seconds)
poll_interval: 43200
pii_table¶
This field determines the location of the csv file that has the mappings for each personally identifiable information (PII) to how to process them. It is used to process the PII field values in both REDCap and RPMS sources.
poll_interval: ~/pii_convert_table.csv
lochness_sync_history_csv¶
This field determines the location of the csv file that has the history of lochness to lochness data transfer timestamp. If the csv file is missing, the timestamp of the next lochness to lochness will be stored to a csv file in the given location.
lochness_sync_history_csv: /data/lochness_sync_history.csv
beiwe¶
The beiwe section is used to configure how Lochness will behave while downloading
data from the Beiwe.
backfill_start¶
The backfill_start field should be an ISO 8601 formatted timestamp. If you do not
add a backfill_start date, the start date will fall back to the date that Beiwe
was initially released
2015-10-01T00:00:00
If you set the backfill_start field to the string consent, Lochness will use
the subject Consent Date from the PHOENIX metadata file
as the backfill starting point.
A valid backfill_start field should look like this
beiwe:
backfill_start: consent
or like this
beiwe:
backfill_start: 2020-01-01
dropbox¶
The dropbox section is used to configure how Lochness will behave when
downloading data from Dropbox.
delete on success¶
You can add a delete_on_success: True field to indicate that any data successfully
downloaded from a specific Dropbox account should be subsequently deleted from Dropbox
to save space. You can configure delete_on_success for each Dropbox account defined
in your keyring.
The resulting section should look as follows
dropbox:
example:
delete_on_success: True
dropbox base¶
For each Dropbox account, you may add a base field to the configuration file to
indicate that Lochness should begin searching Dropbox starting at that location.
The resulting section should look as follows
dropbox:
example:
base: /PHOENIX
box¶
The box section is used to configure how Lochness will behave when
downloading data from Box.
delete on success¶
You can add a delete_on_success: True field to indicate that any data successfully
downloaded from a specific Box account should be subsequently deleted from Box
to save space. You can configure delete_on_success for each Box account defined
in your config.yml.
The resulting section should look as follows
box:
xxxxx:
delete_on_success: True
box base¶
For each Box account, you may add a base field to the configuration file to
indicate that Lochness should begin searching Box starting at that location.
file_patterns field will have the name of directory under the base
directory, with subfields.
The subfields are
- vendor, product: currently not used by lochness.
- pattern: string pattern of the files in interest. The files with matching
names will be pulled.
compress: if True, the matching files will be downloaded and compressed.protect: if True, the files are downloaded under the PROTECTED directory.
The resulting section should look as follows
box:
xxxxx:
base: xxxxx_dir
delete_on_success: False
file_patterns:
actigraphy:
- vendor: Philips
product: Actiwatch 2
pattern: '.*\.csv'
- vendor: Activinsights
product: GENEActiv
pattern: 'GENEActiv/.*(\.csv|\.bin)'
compress: True
mri_eye:
- vendor: SR Research
product: EyeLink 1000
pattern: '.*\.mov'
mediaflux¶
A standalone documentation for the interaction between Mediaflux and lochness is available here. Specifically, you can take a look at mediaflux#configuration-file.
redcap¶
For each PHOENIX study, you may add an entry to the redcap section indicating
that data should be de-identified before being downloaded and saved to PHOENIX.
data_entry_trigger_csv determines the location of the database created
by the listen_to_recap.py by storying the Data Entry Trigger post signals
from REDCap.
Assuming your PHOENIX study is named StudyA this field would look like so
redcap:
data_entry_trigger_csv: ~/data_entry_trigger_database.csv
StudyA:
deidentify: True
admins¶
All email addresses defined in the admins section will be notified on all emails
sent out by Lochness
admins:
username@email.com
notify¶
The notify section allows you to configure more detailed notification behavior.
You can use this section to set different groups of email addresses to be notified
in the event of an error downloading files on a per study basis
notify:
StudyA:
- username1@email.com
- username2@email.com
StudyB:
- username3@email.com
You can also use a __global__ field to add email addresses that should be
notified on any error for any study, similar to the admins
section
notify:
__global__:
- admin1@email.com
sender¶
Whenever an email is sent by Lochness, use this field to determine the sender address
sender: lochness@host.example.org
ssh_user¶
Occasionally, you may receive data on an external hard drive or flash drive.
If you want to use Lochness to transfer this data to your PHOENIX filesystem,
you can do this over rsync+ssh. The ssh_user field determines the
username that will be used for this
ssh_user: example
ssh_host¶
Occasionally, you may receive data on an external hard drive or flash drive.
If you want to use Lochness to transfer this data to your PHOENIXfilesystem,
you can do this over rsync+ssh. The ssh_host field determines the
destination host you will connect to for this
ssh_host: host.example.org