Page History

...

1. Before the above issue solved, one feasible way to execute the automatic data pipeline is to first execute the first phase - data extraction and recovery in the Jupyter notebook (data_preprocessing.ipynb file) on the biohpc server before the scheduling time and identify the problematic barcode_id and temporarily remove the problematic barcode_id from the dataframe in main1() function. This will ensure that the following two phases in the datapipeline can be executed successfully on time. Admittedly, this is not the best solution, the root cause of multiple barcode_id should be found and rectified from a long-run perspective.

Jupyter notebook run in a docker file on Biohpc server

The notebook in the server can be connected any time you want and this serves as an interactive platform where for you to change and test the data pipeline. 

...

1. Core files:
   1. Input_photos
   2. Output_photos
   3. U-2-Net
   4. extract_colors_pipeline.py
   5. Requirements.txt
2. Additional files:
   1. Extract_colors.py: the core cv model, which corresponds to the 2nd phase of the data pipeline; (also in the github)
   2. data_preprocessing.ipynb: if the multiple barcode_id issue is not solved, you can execute this file to extract the images and save them into the input_photos file; 

Next steps

1. The 2nd phase of the datapipeline currently will be executed for all the images in the input_photos folder sequentially, which we could change to only read the new images added to avoid repeatedly color extraction for the existed images. Possible way to avoid repeatedly run color extraction over existed images in phase2: clean the input_photos folder after execution the extraction_colors_pipeline.py. For example, one line of code using shutil.remtree() can be added at the end of the extraction_colors_pipeline.py file.
2. Error in the image extraction phase due to multiple barcode_id needs to be solved.