Automatic data pipeline (Old Version)

Steps to get access the data pipeline

1. SSH to the BioHPC: ssh yournetid@cbsujohnson.biohpc.cornell.edu
2. cd /workdir/xl847/get_colors, the files are the same as you see in the Jupyter notebook (see below) since they are synchronized (any change in the Jupyter notebook in the biohpc server will sync to the files under the path: /workdir/xl847/get_colors)
3. /workdir/xl847/get_colors/extract_colors_pipelines.py is the data pipeline python file, including 3 phases: 
1. data extraction and recovery from the production mongodb database, images saved in input_photos; 
2. image color identification, save the color images in output_photos; 
3. save the model color into the test mongodb database collection: test.colors;
4. Scheduling process using cron

Currently, the cv model is run every week for all the images and you can adjust accordingly. If you want to change the scheduling time, you can use crontab -e; If you want to see current scheduling time, you can use crontab -l;

1. Under the /workdir/xl847/get_colors , enter   cat mylog  to view latest execution status and results of extract_colors_pipelines.py 

1. Currently, there is one unsolved problems in the data extraction and recovery phase in the automatic data pipeline:

If one barcode corresponds to multiple images as the below picture shows, the get_image_ids function will have error. The issue needs to be rectified in the barcode_id generation process;

1. Before the above issue solved, one feasible way to execute the automatic data pipeline is to first execute the first phase - data extraction and recovery in the Jupyter notebook (data_preprocessing.ipynb file) on the biohpc server before the scheduling time and identify the problematic barcode_id and temporarily remove the problematic barcode_id from the dataframe in main1() function. This will ensure that the following two phases in the datapipeline can be executed successfully on time. Admittedly, this is not the best solution, the root cause of multiple barcode_id should be found and rectified from a long-run perspective.

Jupyter notebook run in a docker file on Biohpc server

The notebook in the server can be connected any time you want and this serves as an interactive platform where for you to change and test the data pipeline. 

Steps to log in to the Jupyter notebook

1. SSH to the biohpc: ssh yournetid@cbsujohnson.biohpc.cornell.edu
2. Enter docker1 ps  then you will see the docker container list, container id c3e5fc34710f is the data pipeline; 
3. Enter docker1 exec -it c3e5fc34710f /bin/bash , the docker container will be executed and you then can enter the container path;
4. Enter jupyter server list, you will see an address to log into the Jupyter notebook, for example http://cbsujohnson.biohpc.cornell.edu:8031/lab?token=ea1ddefabeea6d41d9b221c70b41cc6415b1adc9d311ced3
5. Copy paste the web address to a browser, then you can get access to the jupyter notebook in the Biohpc server. The token may be updated across time so remember to update the above address.

Structure of the Jupyter notebook

1. Core files:
   1. Input_photos
   2. Output_photos
   3. U-2-Net
   4. extract_colors_pipeline.py
   5. Requirements.txt
2. Additional files:
   1. Extract_colors.py: the core cv model, which corresponds to the 2nd phase of the data pipeline; (also in the github)
   2. data_preprocessing.ipynb: if the multiple barcode_id issue is not solved, you can execute this file to extract the images and save them into the input_photos file; 

Next steps

1. The 2nd phase of the datapipeline currently will be executed for all the images in the input_photos folder sequentially, which we could change to only read the new images added to avoid repeatedly color extraction for the existed images. Possible way to avoid repeatedly run color extraction over existed images in phase2: clean the input_photos folder after execution the extraction_colors_pipeline.py. For example, one line of code using shutil.remtree() can be added at the end of the extraction_colors_pipeline.py file.
2. Error in the image extraction phase due to multiple barcode_id needs to be solved.