Local developer environment setup

Background

The stack consists of three services, running as containers: -

  • a MySQL database

  • a neo4j graph database

  • the fraglaysis stack

  • a transient data loader container

The stack is formed from code resident in a number of repositories. Begin by forking repositories you anticipate editing (although you really want to consider forking all the repositories as this is a relatively low-cost operation).

The repositories are:

Prerequisites

  • Docker

  • Docker-compose

  • Git

  • NodeJS (v12)

  • Yarn

  • Some target data

Setup

1. Create project directory, e.g.

mkdir fragalysis

2. Clone repositories inside your project’s directory

You can clone original xchem repositories or your forked e.g. m2ms and checkout any branch if necessary

git clone https://github.com/xchem/fragalysis-backend.git
git clone https://github.com/xchem/fragalysis-frontend.git
git clone https://github.com/xchem/fragalysis-loader.git
git clone https://github.com/InformaticsMatters/dls-fragalysis-stack-openshift.git

3. Create some key data directories

In the fragalysis/ directory run script to create data directory structure

mkdir -p data/input/django_data/EXAMPLE
mkdir -p data/neo4j/data
mkdir -p data/neo4j/logs
mkdir -p data/stack/media
mkdir -p data/stack/logs
mkdir -p data/media/compound_sets
mkdir -p data/postgre/data

4. Build images locally

Optional

pushd fragalysis-backend || exit
docker build . -t xchem/fragalysis-backend:latest
popd || exit

pushd fragalysis-loader || exit
docker build . -t xchem/fragalysis-loader:latest
popd || exit

pushd fragalysis-stack || exit
docker build . -t xchem/fragalysis-stack:latest
popd || exit

Mandatory

pushd dls-fragalysis-stack-openshift/images/loader || exit
docker build . -f Dockerfile-local -t loader:latest
popd || exit

Optional

pushd dls-fragalysis-stack-openshift/images/graph || exit
docker build . -t xchem/graph:latest
popd || exit

4. Populating the database

Copy your example data to fragalysis/data/input/django_data/EXAMPLE, before you can launch the application.

Launch the stack

  1. To build the Fragalysis stack (All infrastructure - databases + populating data):

cd fragalysis-backend
docker-compose -f docker-compose.dev.yml up -d

Note: sometimes the database does not finish building before the data loader starts. If this happens, allow the build to finish, kill it, and then run the build again

  1. Navigate to localhost:8080 to check that the front-end is running, and that the data has successfully loaded into the back-end

  2. If needed, stop containers

docker-compose -f docker-compose.dev.yml down

Note: If you want to connect to get into the stack container (web_dock) run:

docker exec -it web_dock /bin/bash

Troubleshooting

Problem: The database keeps throwing errors when I try to build the stack

Problem: The data in my local stack has disappeared.

Solution: Sometimes the postgres database becomes corrupted, or there are problems with migrations. To fix:

  • Delete the ‘mysql_data’ folder located in the data folder (Step 3)

  • Delete the migrations by running the commands below in the project directory (Step 1)

find . -path "*/migrations/*.py" -not -name "__init__.py" -delete
find . -path "*/migrations/*.pyc"  -delete

  • In the ‘fragalysis-frontend’ folder rerun the docker compose command another two times

docker-compose -f docker-compose.dev.yml up

Problem: I keep getting 500 codes when I make changes to the backend code, but I don’t know why

Solution: Turn on Django’s debug mode

  • change the value of DEBUG to True in fragalysis-backend/fragalysis/settings.py

Note: please don’t push this change into git

Problem: No matter what I try, the stack won’t build anymore (it did before)

Solution: Clean up and rebuild:

# remove all old images
docker system prune -a

# rebuild the loader image (and any other images needed)
pushd dls-fragalysis-stack-openshift/images/loader || exit
docker build . -f Dockerfile-local -t loader:latest
popd || exit #etc

# re-run docker compose
docker-compose -f docker-compose.dev.yml up