Diabetes Screening and Monitoring Using Tongue Images and Self-reported Symptoms: a Machine Learning Approach

Diabetes Screening and Monitoring Using Tongue Images and Self-reported Symptoms: a Machine Learning Approach

Study Background: Diabetes mellitus (DM) is a major non-communicable disease. Diagnosis and self-management of DM is important. Currently, detection of diabetes requires blood tests, which is costly and inconvenient, especially for elderlies.Tongue diagnosis has been used in Chinese medicine as a routine diagnostic method, and it has recently been studied for detection of DM and diabetic retinopathy (DR). We have developed a method for taking tongue images using smartphone, which can reveal more detailed features than conventional clinical tongue inspection. There are many limitations of the preliminary study. Therefore, it is our plan in this study to address these specific limitations with the following objectives. The results of this study will enable us to develop a practical App for diabetes screening and monitoring.

Study Objective

The aim of the study is to develop an algorithm for diabetes screening, with the following objectives:

1. . To determine the sensitivity and specificity of tongue images taken with smartphone in predicting abnormal HbA1c (≥6.5%);
2. To determine tongue image features responsible for the classification of normal and abnormal levels of HbA1c (≥6.5%);
3. To determine the sensitivity and specificity of tongues image in predicting four different levels of HbA1c: <6% (normal), 6-6.4% (prediabetes), 6.5-8.9% (diabetes) and ≥ 9% (diabetes with high HbA1c);
4. To determine the sensitivity and specificity of combining image analysis results with the results from a TCM symptom questionnaire in predicting the four levels of HbA1c.

Hypothesis: Our working hypothesis is that different tongue coating features may be associated with different stages of diabetes, as indicated by different levels of HbA1c；and different combination of symptoms from a TCM point of view may also be associated with different levels of HbA1c. Thus, combining tongue image with TCM symptoms may allow a machine learning model to build an algorithm for HbA1c prediction with reasonable accuracy.

Inclusion criteria: The inclusion criteria is adult subjects with HbA1c test results from a laboratory that meets the ISO 15189 standard, such as those laboratories used by the Hospital Authority of Hong Kong.

Exclusion criteria: Subjects who are unable to give consent, unable to answer the questionnaire or to cooperate in tongue image collection will be excluded. We will not include subject who are unable to understand written Chinese or English.

Study design: This is a cross-sectional design looking at the relationship between tongue image pattern and HbA1c reading. Age, gender, weight, height , duration of diabetes, family history of diabetes and any comorbid disease will be recorded. The level of hemoglobin and blood lipid profile will also be recorded if the information is available. Any acute repertory or digestive illness, as well as smoking habits will also be noted. An electronic questionnaire (using Qualtrics survey software) based on published TCM symptoms of diabetes and the abovementioned information will be used for data collection.

Data processing and analysis

1. Tongue segmentation The images containing the tongue and its surrounding area will be processed for segmentation of the tongue area. This segmentation is carried out by a computer algorithm developed in-house by machine learning.
2. Machine learning Two approaches will be used in machine learning. In the first approach, we will first perform image classification of either normal or abnormal HbA1c and generate the probabilities for the classification using convolutional neural networks (CNNs) (Anwar et al., 2016; Ødegaard et al., 2016). Then we will try to classify the images into four different classes according to their HbA1c level: <6% (normal), 6-6.4% (prediabetes), 6.5-8.9% (diabetes) and ≥ 9% (diabetes with high HbA1c).

Primary Outcome: Tongue image features: We will extract tongue image features and perform image classification of either normal or abnormal HbA1c and generate the probabilities for the classification using convolutional neural networks (CNNs). Then we will try to classify the images into four different classes according to their HbA1c level: <6% (normal), 6-6.4% (prediabetes), 6.5-8.9% (diabetes) and ≥ 9% (diabetes with high HbA1c).

Secondary Outcome: Symptom patterns: Questionnaire data will be combined with image data for regression analysis

Tongue image collection Tongue image collection will be carried out by trained research assistants based on our published protocol (Wang et al., submitted), which consisted of the following steps. Step 1, start the phone's camera function without using any filter or artificial intelligence (A.I.) function; Step 2, turn on the flash; Step 3, aim at the lips of the subject by holding the phone directly in front of, and 15-20 cm away from, the subject with a 45 degree angle; Step 4, tap on the screen to adjust focus when the subject protrudes the tongue and then take a picture; Step 5, upload the image in full resolution to the online questionnaire system (Qualtrics survey software).

Chinese medicine questionnaire collection A questionnaire in electronic/paper form will be used to collect symptoms relevant to Chinese medicine pattern diagnosis of diabetes. The questions have been selected from previous studies on the most commonly appearing symptoms of diabetes from Chinese medicine perspectives (Hsu et al., 2016; Zhou et al., 2012; Zhao et al., 2017). Age, gender, weight, height , duration of diabetes, family history of diabetes and any comorbid disease, Hb reading (if available) will also be taken into consideration. Recent history of smoking is also recorded as a potential confounding factor for tongue coating features (Tomooka et al., 2017)

Data entry and verification To ensure accuracy, image, questionnaire answers and HbA1c (and Hb, if available) readings will be uploaded to the server at the same time. A patient identification number for the trial will also be generated to facilitate data verification. Such identification number consists of information on data collection date and outpatient number, etc. However, it will not contain any information related to patient's HKID, full patient name, HN, MRN, DOB, address or phone number.

The data will be collected by an online questionnaire service using Qualtrics and then store at a local computer at SCM/HKBU. A paper version of the questionnaire may be used when necessary, and the filled questionnaire will be kept for 3 years after the study. The collected image will contain no eye features on the photo, and the questionnaire will not include any information for identification of the subject, except a subject ID number for the trial. After completion of the project, data on Qualtrics will be deleted, whereas data on the local computer will be kept for seven years after the study has been published, as per institution requirement.

Machine learning Two approaches will be used in machine learning. In the first approach, we will first perform image classification of either normal or abnormal HbA1c and generate the probabilities for the classification using convolutional neural networks (CNNs) (Anwar et al., 2016; Ødegaard et al., 2016). Then we will try to classify the images into four different classes according to their HbA1c level: <6% (normal), 6-6.4% (prediabetes), 6.5-8.9% (diabetes) and ≥ 9% (diabetes with high HbA1c). The probability data of image classification will be combined with data from the questionnaire as variables using fully connected CNNs layers to determine which one of the four HbA1c levels the subject belongs to (Osia et al., 2018). The advantage of this approach is that it allow us to determine the sensitivity and specificity of the classifications using tongue image alone. However, a drawback of this approach is that it may lose some important features of the tongue in classification with questionnaire data. Alternatively, we will extract features from each tongue image using CNNs, and use one hot encoding to transfer the features into one dimensional matrix, which enable the blending of image data with questionnaire data for regression analysis using neural network.

A separate testing data set, which contains at least 200 negative and 200 positive images, will be used to evaluated sensitivity and specificity of the algorithms obtained. In the final algorithm, sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) will be reported.

Potential problems and experimental alternatives

Potential problem 1:

It is highly likely that the tongue image features are not exclusive to abnormal level of HbA1c, and they are also present in smokers and some acute and chronic diseases, such as COPD, upper respiratory infection and chronic kidney or liver diseases.

Experimental alternatives:

We have now included a section for smoking history and known acute or chronic diseases in our questionnaire. If we found a significant correlation between certain confounding variables with abnormal HbA1c tongue diagnosis, we would warn the future user about the unreliability of the algorithm when used in a population with the confounding condition. Furthermore, we would adjust the weighting of parameters in the questionnaire to achieve a prediction model with reasonable specificity and sensitivity, even for populations with the said condition.

Potential problem 2:

Machine classification results from the collected tongue image data set show poor specificity and sensitivity in predicting the four HbA1c levels, even after the inclusion of parameters in TCM questionnaire.

Experimental alternatives:

As we will collect blood lipid profile of subjects from the hospital when available, we can analyze our results as we go. When our total sample population reaches 2,000, and we still see a poor specificity and sensitivity, we will include blood lipid profile test to our community subjects. By adding blood lipid profile as parameters in our regression model, we should be able to enhance the specificity and sensitivity of the prediction, as previous studies have found a close correlation between high level of HbA1c and dyslipidemia (Khan et al., 2007). Of course, we do not wish this to happen because it will lower the value of our prediction model, as it requires an invasive procedure to obtain blood lipid profile. Nevertheless, we are confident that at least a non-invasive model for detection of abnormal level of HbA1c (≥ 6.5%) can be produced, based on the results from our proof-of-concept study.

Potential problem 3:

The image features responsible for the classification of normal and abnormal levels of HbA1c (≥6.5%) could not be readily identified.

Experimental alternatives:

Extraction of visual features that can provide a semantic and robust representation of tongue images with abnormal levels of HbA1c is highly desirable. The greasy and thick coating, which we have identified as features for the classification, needs to be further defined. To this end, we will try different techniques proposed in the literature (Ching et al., 2018), including assigning example-specific importance scores, matching or exaggerating the hidden representation, activation maximization and latent space manipulation, etc.

Inclusion Criteria:

Adult subjects With HbA1c test results from a laboratory meeting the ISO 15189 standard within two weeks before or after the tongue images and questionnaire collection.

Exclusion Criteria:

Unable to give consent Unable to answer the questionnaire or to cooperate in tongue image collection Unable to understand written Chinese or English.