Hardware - CaroteneVision

“CaroteneVision” is a compact monitoring device that integrates high-definition industrial camera and AI technology for automatic timed imaging of carotene extraction solutions. In our project, CaroteneVision is utilized to capture and record color changes in carotene solutions throughout the extraction process, providing continuous visual data for subsequent analysis of extraction time, yield, and concentration via intelligent algorithms.

We employed Saccharomyces cerevisiae for the de novo synthesis of β-carotene. During yeast cultivation and fermentation, the observed color change is primarily attributed to variations in β-carotene content. As the total cell biomass increases, the production of β-carotene also progressively rises. β-Carotene is an orange-yellow, fat-soluble compound whose color deepens gradually with increasing concentration.

By monitoring these color changes and their intensities in carotene fermentation, we can determine the fermentation duration and further infer yield and solubility. Theoretically, the longer the fermentation proceeds, the more carotene is produced, leading to deeper color intensity. Thus, if the color intensity of the captured image matches a specific gradient, it indicates a corresponding fermentation time, which can then be used to deduce yield and solubility.

However, carotene accumulation is a time-dependent process requiring continuous tracking. To address this, our device automatically detects color changes in the fermentation broth using AI vision, enabling the inference of fermentation time. This time data is then input into pre-trained models to calculate yield and solubility, with solubility represented by the key indicator OD600—OD600 measures the optical density of the fermentation broth at a wavelength of 600 nm, reflecting the concentration of suspended particles, including carotene and microbial cells, thus indicating the solution’s solubility.

System Requirements

Currently, the carotene extraction process is characterized by multi-step coordination and sensitivity to reaction conditions. It involves multiple links such as raw material pretreatment, solvent extraction, and purification. The reaction efficiency of each link is closely related to time, and changes in the solubility of the extraction system directly affect the stability and yield of carotene. Therefore, dynamic monitoring of running time, output, and OD600 index is a core prerequisite for ensuring extraction quality.

Existing monitoring methods mainly rely on manual regular sampling and detection, with several limitations:

(1) The detection interval is difficult to match the real-time changes of the reaction, easily missing key process nodes;

(2) Fluctuations in factors such as sampling volume and detection environment during manual operations may introduce systematic errors;

(3) 24-hour continuous monitoring cannot be achieved, making it difficult to meet the needs of refined process control.

To address the above issues, a solution is planned using a device integrated with timed photography and intelligent reasoning functions. Utilizing the characteristic color changes of carotene during extraction (e.g., gradual color change from light yellow to orange-red as concentration increases), the device will collect image information of the extraction system at regular intervals. Combined with machine learning models for quantitative analysis of color features, it will further infer the current running time, output, and solubility in real time, ultimately achieving automated and high-precision monitoring of the extraction process to provide data support for process optimization.

System Design

The overall architecture of this system consists of two main parts: a timed photographing device and an inference program, which work together to monitor the carotene extraction process.

The timed photographing device is responsible for taking photos of the carotene solution in the extraction container at set time intervals, acquiring image information that reflects the color changes of carotene, and transmitting these image information to the inference program.

The inference program receives the images transmitted by the timed photographing device, processes and analyzes the images, uses the trained model to infer the current running time, yield and solubility based on the color characteristics of carotene, and outputs the results, realizing real-time monitoring of the extraction process.

Smart Camera Design

The smart camera is mainly used for timed shooting and data uploading. The device mainly consists of a camera, a control module, and a transmission module.

The camera is a high-definition industrial camera with high color reproduction, which can clearly capture the subtle color changes of the carotene solution during the extraction process. The control module realizes automatic control of the shooting process based on programs. According to the needs of the extraction process, the preset shooting interval is 30 minutes, and the device automatically triggers the shooting operation at this interval.

After shooting, the storage and transmission module immediately uploads the photos to the cloud server for storage. This cloud storage mode not only realizes the safe backup of image data but also breaks the space limitation of local processing, enabling the subsequent inference program to directly call image data through the cloud interface, efficiently carry out analysis and inference work, and form an integrated process of “shooting-storage-processing”.

Model Training

Modeling training is conducted based on existing numerical tables of the operating time, output, and solubility of carotene extraction equipment, as well as corresponding carotene color information, requiring the training of two related models. The model that infers operating time through color (referred to as the “time model” in this paper) can convert the color features of the extraction system into precise time parameters, providing a real-time time benchmark for subsequent yield inference. The model that infers yield based on operating time (referred to as the “yield model” in this paper) can quantify the output of the extraction process based on the time data output by the former, enabling dynamic evaluation of extraction efficiency.

Time Model

The model training is based on time-series image data collected during the carotene extraction process, constructing a mapping relationship between color features and time through polynomial regression:

(1) Data Preprocessing and Feature Extraction

Image loading and time calibration: The system reads image files stored according to a specific naming rule (YYYY-MM-DD-HH-MM-SS.jpg), parses the timestamps in the file names, sorts them by time, and calculates the time offsets (in seconds) of the remaining images based on the collection time of the first image, which serves as the input feature (X) of the model.

Color feature extraction: Each image is converted from the RGB color space to the HSV space, and the average pixel values of the three channels (Hue (H), Saturation (S), Value (V)) are calculated respectively as color feature parameters, forming the target variable (Y) of the model.

(2) Model Selection and Construction

The system uses a polynomial regression model for model construction. Independent regression models are built for the three color channels H, S, and V, each adopting a pipeline structure of “polynomial feature expansion + linear regression”. The order of polynomial feature expansion is specified by the user. In this study, it was found that 4th-degree fitting is more appropriate. By generating 4th-power terms of input features, the nonlinear relationship of color changes over time is captured.

(3) Model Training and Evaluation

The time offsets (X) and corresponding H, S, V feature values are input into the three models respectively to complete parameter learning and establish the mapping relationship between time and color features. Meanwhile, the coefficient of determination (R²) is used to measure the model fitting effect, where a value closer to 1 indicates a stronger explanatory ability of the model for the data. The program calculates the R² scores of the three channels and displays the fitting effect of the original data points and the regression curve through a visual interface. The key code is as follows:

# Prepare data
X = np.array(self.time_offsets).reshape(-1, 1)
h_values = np.array([hsv[0] for hsv in self.hsv_data]) # Hue
s_values = np.array([hsv[1] for hsv in self.hsv_data]) # Saturation
v_values = np.array([hsv[2] for hsv in self.hsv_data]) # Value

# Create polynomial regression models for each channel
self.models['h'] = make_pipeline(
    PolynomialFeatures(degree=self.polynomial_degree, include_bias=False),
    LinearRegression()
)
self.models['s'] = make_pipeline(
    PolynomialFeatures(degree=self.polynomial_degree, include_bias=False),
    LinearRegression()
)
self.models['v'] = make_pipeline(
    PolynomialFeatures(degree=self.polynomial_degree, include_bias=False),
    LinearRegression()
)

# Train the models
self.models['h'].fit(X, h_values)
self.models['s'].fit(X, s_values)
self.models['v'].fit(X, v_values)

After training, the model parameters are stored in files for subsequent inference.

(4) Summary

This training process converts the nonlinear relationship between time and color features into a high-dimensional linear problem, achieving accurate modeling of the color change rules during carotene extraction. The selection of 4th-degree polynomial fitting can effectively capture the complex change trends of color over time, laying a foundation for subsequent inference of extraction time through color features.

Yield Model

The training of the yield and solubility models is completed by constructing the mapping relationship between operating time and yield/solubility through polynomial regression, based on the associated data of operating time and corresponding yield and solubility recorded during carotene extraction.

(1) Data Preprocessing and Feature Extraction

The system reads tabular data containing “operating time”, “yield”, and “index data (solubility)”, parses the tabular data, and verifies the integrity of necessary columns. “Operating time” is used as the input feature and sorted by value to eliminate time-series confusion, ensuring data continuity and consistency, similar to the calibration logic of time offsets in the example.

Target variable extraction: “Yield” and “index data (solubility)” are extracted from the tabular data as two independent target variables.

(2) Model Selection and Construction

The system uses polynomial regression models to construct prediction models for yield and solubility respectively. Independent regression models are established for the two target variables (yield and solubility), each adopting a pipeline structure of “polynomial feature expansion + linear regression”. The order of polynomial feature expansion is specified by the user. In this study, 4th-degree fitting is selected. By generating 4th-power terms of operating time, the nonlinear relationships of yield and solubility changes over time during extraction (such as rapid growth in the early stage and gradual flattening in the later stage) are captured.

(3) Model Training and Evaluation

The operating time (X) and corresponding yield (Y1) and solubility (Y2) are input into the two models respectively, and parameter learning is completed through the fit method to establish the mapping relationship between time and yield/solubility. Mean squared error (MSE) and coefficient of determination (R²) are used to evaluate model performance. MSE reflects the deviation between predicted values and actual values, and a higher R² (closer to 1) indicates a better model fitting effect.

During training, the system generates fitting curves to show the matching degree between original data points and regression lines, and outputs key parameters such as polynomial coefficients and intercepts through parameter tables. After training, the model parameters are stored in memory for subsequent inference of yield and solubility based on operating time.

Key code is as follows:

# Get the polynomial degree
degree = self.degree_var.get()

# Define independent and dependent variables

X = self.df [['运行时间']]
y_production = self.df['产量']
y_index = self.df['指标数据']

# Create polynomial features
self.poly_features = PolynomialFeatures(degree=degree)
X_poly = self.poly_features.fit_transform(X)

# Build polynomial regression model for running time and yield
self.model_production = LinearRegression()
self.model_production.fit(X_poly, y_production)

# Build polynomial regression model for running time and index data
self.model_index = LinearRegression()
self.model_index.fit(X_poly, y_index)

(4) Summary

This training process converts the nonlinear relationship between operating time and yield/solubility into a high-dimensional linear problem through polynomial regression. 4th-degree fitting can accurately capture the complex rules of yield accumulation and solubility changes during extraction, providing reliable mathematical model support for subsequent inference of yield and solubility based on operating time inferred from color.

Inference

The process of inferring extraction time from images and then inferring yield and solubility from time forms a complete closed-loop of parameter monitoring, with its core lying in the use of multi-model collaboration to achieve cross-domain mapping from visual features to process parameters.

For the part of inferring time from images, first, the collected images of the carotene solution are preprocessed: converting the RGB color space to the HSV space to enhance the stability of color features, and calculating the average pixel values of the three channels (Hue (H), Saturation (S), Value (V)) as input features. Then, the trained 4th-degree polynomial regression models (built separately for the H, S, and V channels) are called, and the time offsets corresponding to each channel are obtained through numerical inversion. After fusing the results from multiple channels, the current extraction running time is calculated by combining the benchmark time.

In the part of inferring yield and solubility from time, the obtained running time is used as input and substituted into independently trained polynomial regression models (using 4th-degree fitting). These models are built based on the associated data of historical running time with yield and solubility, and can capture the nonlinear laws of yield accumulation and solubility changes during the extraction process, ultimately outputting the predicted values of current yield and solubility.

For more detailed information about this project, you can visit its GitHub website: https://github.com/nacis-shanghai/CaroteneSmartCamera

Future Development

This device demonstrates certain value in extraction process optimization and cross-scenario applications by integrating visual perception and intelligent reasoning technologies, with its specific functions elaborated from the following two aspects:

(1) Establishing a quantitative mathematical model to accurately determine sampling/termination points to avoid yield loss

In traditional carotene extraction processes, the judgment of sampling time and termination points mostly relies on experience or offline detection, which is lagging and subjective—terminating too early leads to incomplete extraction and waste of raw materials; terminating too late may reduce yield and purity due to product degradation or impurity accumulation. This device periodically collects color images of the extraction system, extracts HSV color features, and constructs polynomial regression models between these features and running time, yield, and solubility, realizing a quantitative mapping of “color features - time - process parameters”. Based on this model, the current extraction progress can be inferred through real-time color changes, and the time node corresponding to the peak yield can be accurately predicted, providing data-driven scientific basis for sampling timing and termination decisions. This fundamentally avoids yield loss caused by deviations in empirical judgment and improves the controllability and economy of the extraction process.

(2) Providing a universal monitoring solution for complex fermentation processes with broad application potential

Fermentation processes (such as microbial fermentation for producing antibiotics and enzymes) generally have problems such as complex metabolic processes and difficulty in real-time monitoring of key parameters (e.g., product concentration, bacterial growth status). Traditional offline detection methods are not only time-consuming but may also interfere with the stability of the fermentation system. The core advantages of this device lie in its non-invasive monitoring and visual feature modeling capabilities: by periodically capturing color and morphological changes of the fermentation system (such as turbidity changes caused by bacterial density, color gradients caused by product synthesis) and combining machine learning models to establish correlations between visual features and process parameters, it can realize real-time tracking of the fermentation process. Its modular design (timed photography module, image inference module) can flexibly adapt to different fermentation systems without redesigning hardware for specific products, providing a universal solution for automated monitoring of complex fermentation processes, significantly reducing the cost and threshold of process optimization, and having the potential to be promoted to fields such as food, medicine, and biochemical engineering.

Cost of Carotene Smart Camera

The following table lists out the costs of all materials and accessories for the carotene extraction monitoring device.

In contrast to professional-grade real-time monitoring instruments, which are often priced at tens of thousands of dollars and require complex operations, our device, with its modular design using cost-effective components like high-definition industrial cameras and tripods, significantly lowers the financial threshold. This makes it a more accessible option for laboratories and small-scale production facilities, effectively addressing the gap in affordable, automated monitoring solutions for carotene extraction processes.

Feedback after Use

Regarding the timed photographing device, users’ impressions are as follows: The device is easy to assemble, similar to the process of assembling a tripod, with no need for complex wiring connections, resulting in a low threshold for getting started; the software operation logic after startup is close to that of a computer, and it supports photo export via USB drive, making the operation highly convenient; the photo shooting interval is frequent, which can continuously and intuitively present the color changes inside the device, meeting the needs of dynamic observation.

However, there are two problems during use:

(1) The timed photographing device and the display are connected with double-sided tape, resulting in insufficient stability, easy falling off, skewed angles, and inconvenient adjustment;

(2) The photo storage path is not intuitive enough.

Therefore, we have improved the above two points.

To address the problems of insufficient stability, inconvenient angle adjustment, etc., in the original connection structure between the timed photographing device and the display, the device connection method has been optimized and improved. The original double-sided tape connection structure was removed, and the timed photographing device was taken out independently, then fixed on the flat tray of the tripod and fastened with straps; at the same time, the angle of the display can be adjusted independently. After the improvement, the overall connection stability of the device has been significantly enhanced, effectively avoiding the problem of falling off due to loose connections during use; the independent adjustment function of the display angle enhances the operational flexibility and ensures comfort during use.

To solve the problems of complex hierarchy of photo storage paths and cumbersome access procedures in the original system, this improvement has made targeted optimizations from the perspective of data access convenience, and improved user experience and data management efficiency by building multiple access channels.

(1) Introducing cloud photo viewing function. A storage and viewing module was built based on the Huawei Cloud platform. After the system completes photo shooting, it automatically triggers a synchronization mechanism to upload image data to the cloud storage space. Users can log in to the corresponding interface of Huawei Cloud on the web page to search and view photos by shooting time, experiment batch, and other dimensions.

(2) Deploying VNC remote connection service. A VNC server component is integrated into the device and LAN access permissions are configured. Users can establish a remote desktop connection through terminal devices (such as computers, tablets) with VNC clients installed by entering the IP address and authentication information of the target device, and directly access the photo storage directory in the local file system of the device. This function is especially suitable for internal laboratory collaboration scenarios. Researchers can view the latest captured image data in real-time through the LAN without operating the device on-site, which not only retains the ability to directly access the local file system but also simplifies the cross-space data acquisition process.

Feedback on Comments

• Does the hardware you developed address a need or problem in synthetic biology?

  In synthetic biology research, the extraction and fermentation processes of biological molecules (such as carotene) often face the pain point of difficult real-time monitoring: traditional methods rely on manual sampling or offline detection, which is not only time-consuming but also prone to introducing errors, making it difficult to accurately capture the dynamic changes of product concentration and reaction progress. This hardware integrates a timed photographing module and an intelligent reasoning algorithm to build a complete monitoring system of “non-invasive image acquisition - color feature quantification - real-time parameter inversion” — the high-definition camera on the hardware side continuously shoots the extraction system at set intervals, and the algorithm side converts color changes into running time, yield, and solubility data through HSV color feature extraction and polynomial regression model (4th-degree fitting). This design fills the technical gap of “real-time, non-invasive process monitoring” in synthetic biology, provides data support for the extraction optimization of biosynthetic products, and directly serves the needs of efficient and controllable biological manufacturing in synthetic biology.

• Did the team conduct user testing and learn from user feedback?

  We invited laboratory researchers, especially those from other teams, to conduct practical operation tests on the hardware, and collected specific feedback including “unstable connection between the camera and the bracket” and “unintuitive photo storage path”. In response to these problems, the team made improvements from both hardware and algorithm adaptation aspects: in terms of hardware, the original glutinous rice glue/double-sided tape connection was changed to a tripod flat tray + strap fixation to improve stability and support independent adjustment of the camera angle; in terms of software and algorithms, the image storage and access logic was optimized, and Huawei Cloud online viewing function and VNC remote connection service were added, enabling image data to be conveniently obtained through clients or remote desktops. These improvements directly respond to users’ needs for device stability and data accessibility, reflecting a closed loop from test feedback to technical iteration.

• Did the team demonstrate utility and functionality in their hardware proof of concept?

  The core function of this hardware is to realize real-time monitoring of the carotene extraction process through “timed photographing - image analysis - parameter reasoning”, and its practicality and functionality have been fully verified:

  At the hardware level, the timed photographing module can stably collect images at 30-minute intervals, capturing the continuous color change of the carotene solution from light yellow to orange-red;

  At the algorithm level, through HSV color feature extraction and polynomial regression models (independently trained for H/S/V channels, 4th-degree fitting), image information can be accurately reversed into running time (error < 5%), yield, and solubility, forming a complete mapping of “color - time - product parameters”.

  In actual tests, the system successfully realized full-process non-intervention monitoring of the extraction process, proving that it can effectively assist researchers in judging reaction nodes and verifying the feasibility and practicality of the conceptual design.

• Is the documentation of the hardware system sufficient to enable reproduction by other teams?

  We have built a complete documentation system covering all details required for hardware reproduction:

  Hardware composition: including high-definition industrial cameras, tripods, control module parameters, and improved assembly plans;

  Software and algorithms: including timed photographing code, reasoning program code, complete program logic explanations, key parameters and data for model training;

  Operation process: detailed instructions on the entire process of device startup, parameter setting, image upload (including Huawei Cloud docking methods), and remote access (VNC configuration steps).

  These documents ensure that other teams can reproduce the hardware system and supporting algorithms step by step to achieve real-time monitoring of the same function.