Content » Vol 95, Issue 1

Investigative Report

Concordance and Time Estimation of Store-and-forward Mobile Teledermatology Compared to Classical Face-to-face Consultation

Niccolò Nami1, Cesare Massone2, Pietro Rubegni1, Gabriele Cevenini3, Michele Fimiani1 and Rainer Hofmann-Wellenhof2

Departments of 1Medical, Surgical and Neuro Sciences, Dermatology Section and 3Surgery and Bioengineering, University of Siena, Siena, Italy, and 2Department of Dermatology, Medical University of Graz, Graz, Austria

Smartphones have overcome the limitations of image quality seen in older devices and opened a new field of telemedicine called “mobile teledermatology”. Technological advances and the need to reduce health service costs will strongly promote the development of telemedicine. For this reason, we evaluated the concordance between store-and-forward mobile teledermatology and the classical face-to-face dermatological visit. We also measured the time taken to submit a teleconsultation using a smartphone. Before conventional face-to-face visit, a final-year resident of the 3-year course for general practitioners collected medical history, took digital images of skin diseases with a smartphone and, measuring the time required to complete this operation, transmitted them to an expert teledermatologist. In 391 patients we obtained a concordance between face-to-face and store-and-forward diagnosis of 91.05% (Cohen κ coefficient = 0.906). On average only few minutes needs to be added to a normal visit to transmit the cases to an expert teledermatologist. Key words: telemedicine; mobile teledermatology; store-and-forward teledermatology; smartphones; time.

Accepted Apr 24, 2014: Epub ahead of print Apr 25, 2014

Acta Derm Venereol

Dr. Niccolò Nami, MD, Department of Medical, Surgical and Neuro Sciences; Dermatology Section. University of Siena, Policlinico “Le Scotte”, Viale Bracci, IT-53100 Siena, Italy. E-mail: niccolo.nami@libero.it

The growing interest in telemedicine for dermatology depends on the nature of dermatology itself. Dermatology, together with radiology and pathology, is one of the most visual specialties in medicine (1–3). Several recent studies demonstrate the feasibility and reliability of patient assessment by teledermatology (TD) (4–12). However, these studies also show that accurate medical history, clinical data and suitable equipment to reproduce visual perception are essential for successful teleconsultations (13, 14).

Today there are many instruments for acquiring high-resolution digital images, ranging from professional and compact cameras to new generation smartphones. The latter have overcome the limitations of image quality seen in older devices and opened a new field of “mobile teledermatology” (15–18). Mobile phone technology offers portability and the convenience of capturing clinical images and medical history on a single mobile device and transmitting them by email or specific web applications to consulting dermatologists (16, 17).

The aim of our study was to evaluate the efficacy and reliability (concordance of diagnosis and prescribed therapy) of a web-based application system developed for mobile phones, by comparing the classical face-to-face dermatological examination with store-and-forward TD. Another aim was to investigate whether it is time-saving to transmit the cases to an expert teledermatologist rather than to have a conventional face-to-face dermatological consultation.

Materials and Methods

Patient data sample and population

The study was conducted at the Department of Dermatology, Siena University, Italy and the Department of Dermatology, Medical University of Graz, Austria. Once written informed consent had been obtained, patients, self-referred (17%) or referred by a general practitioner (GP) (83%) for a first evaluation of skin disorders, were selected prospectively from the general outpatient clinics of the 2 departments (one day per week for 3 h; about 10 patients every day). Pigmented skin lesions (PSL) were excluded from the study. Between October 2011 and October 2012, 391 patients were examined (187 males (47.8%) and 204 females (52.2%)), age 1 month to 100 years (mean age 54 years); 386/391 were Caucasian and 5 were North African; 297 were enrolled in Siena (74%) and 94 in Graz (26%). In each case, 1–6 digital images (mean 2.34) were taken. Fig. S11 shows the mean number of images and the respective 95% confidence interval for each site.

Digital image acquisition

Before examination, a final-year resident of the 3-year course for GPs collected medical history, took on his own choice 1–6 digital images of skin lesions using a new generation mobile phone (iPhone 4s, Apple Inc., Cupertino, CA, USA), with wireless internet connection and built-in 8-megapixel autofocus camera. To acquire clinical images, he followed American Telemedicine Association (ATA) guidelines (14), using a standard procedure consisting of maintaining the mobile phone camera angle perpendicular to the lesion, multiple views (close-up straight on the lesion, middle field with affected area in centre, whole body or obvious region, distant if needed) after removal of distracting jewellery and clothing. We also standardised the background and lighting using solid neutral colour perpendicular to the camera and diffuse indirect lighting. The mobile telephone was equipped with MugDerma (e-derm-consult GmbH, Graz, Austria), a web-based mobile-phone application to facilitate acquisition of digital images and patient data (identification number, age, gender, site of lesion, description of skin lesion, salient medical history) and their transmission to a secure website with personal user password. The mobile platform was configured to ensure encryption and authentication of data and secure transmission according to the regulations of the Health Insurance Portability and Accountability Act (HIPAA) (19).

Face-to-face examination and tele-evaluation

Three dermatologists of the Department of Dermatology of Siena and 3 from the Department of Dermatology of Graz, with similar experience in clinical dermatology, took turns in face-to-face examination, while an expert teledermatologist from the Department of Dermatology of Graz evaluated the cases by TD (store-and-forward) on a 20 HP L2045w LCD widescreen (Hewlett-Packard Development Company, Houston, USA).

According to ethical principles and the standards of routine practice, the clinical face-to-face diagnosis was taken as benchmark. Dermatologists doing face-to-face examination and the teledermatologist formulated a probable diagnosis and prescribed therapy, even in cases requiring further study, such as biopsy, mycological examination, cytology or ultrasonography.

A standardised form, developed by the dermatologists prior to the study, was used to record diagnosis and therapies. The form was used for conventional and remote examination. Diagnosis and the therapy prescribed were recorded as a pre-definied list of dermatological diseases (>200) and a list of systemic, topical and surgical treatments.

Teledermatology time

Face-to-face examinations were divided into parts, each of which was timed. The parts were: (i) Medical history time = time taken to record medical history, examine the patient and formulate a diagnostic hypothesis. (ii) Face-to-face prescription time = time taken to write letter of referral to GP with diagnosis and prescription, and explaining them to patient. (iii) Total face-to-face time = medical history time + face-to-face prescription time. (iv) TD consent, photo and uploading time = time taken to explain the study to the patient, obtain written informed consent, open the application on the device, take pictures, complete the patient record (identification number, age, gender, site of lesion, description of lesion, prominent medical history) and send them by mobile phone. (v) Total TD submission time: medical history time + TD consent, photo and uploading time. (vi) Consultant TD interpretation and response time = the time taken by the expert teledermatologist to fill in the standardised form with his diagnosis and suggested therapy.

Reliability analysis

Diagnostic accuracy was defined as agreement between the primary diagnosis by remote consultation and that reached by face-to-face examination. The concordance was evaluated by comparing all prescribed therapies together and all local, systemic or surgical therapies separately. The reliability of TD was assessed statistically using Cohen’s kappa coefficient (κ) of concordance which objectively evaluates agreement between 2 raters or methods of measurement (20, 21). The statistical significance of sample-estimated κ was evaluated by 95% confidence interval (CI). The Landis and Koch (LK) scale is usually used for qualitative evaluation of the type of agreement associated with estimated κ values and their respective 95% CIs (22). The LK scale considers 5 equally spaced intervals of κ values ranging from 0 to 1, which correspond to no, slight, fair, moderate, substantial and almost perfect agreement. Statistical calculations were performed with a self-made code written in the Matlab programming language (23).

Results

Between October 2011 and October 2012, 391 patients were enrolled in the study. A total of 982 images were sent to an expert teledermatologist, 24 of these photos were considered out of focus or overexposed. Despite some photos were discarded because of their poor quality (out of focus and/or overexposed), no patient could not be examined and then excluded from the study; in most of the cases, infact, more than one picture were submitted for evaluation and the patient was examinated through the other photos sent (Fig. S11). Three patients, a 17-year-old girl with acne, a 47-year-old woman with rosacea and a 22-year-old man with genital warts, declined enrolment in the study.

Diagnostic agreement

Table I lists the correlation between diagnoses reached by face-to-face examination and those reached by telediagnosis. Details of diagnostic discordances are also given. A total of 356 out of 391 patients were identically diagnosed, making a total observed agree­ment of 91.1%. Cohen’s κ was 0.906, interpreted as almost perfect agreement by the LK scale. By virtue of our relatively large sample size, the 95% CI of κ was rather narrow, covering a range of values (0.876–0.936) that was included completely in the highest LK class of almost perfect agreement (Table II). Because the κ value declines with increasing number of diagnostic categories and decreasing random agreement, and we had a large number (n = 65) of distinct types of diagnoses, the teledermatological diagnosis can be considered very reliable (20, 21). A more detailed analysis of Table I showed that the most frequent diagnostic disagreements were for psoriasis, with only 10/14 cases correctly diagnosed by telemedicine (71.43%), and for other skin neoplasms such as fibroma molle, lipomas and dermatofibromas, which together reached a correct classification in 8/11 cases (72.7%).

Table I. Correlation between face-to-face and tele-diagnoses

Principal group and percentage of total

Face-to-face diagnostic category and percentage of totala

Concordant diagnosis of cases/total and correct classification n (%)

Wrong diagnosis proposed in teledermatology

Skin neoplasms (27.6%)

Carcinoma of skin and lips (12.0%)

45/47 (95.7)

Actinic keratosis, seborrhoeic keratosis

Seborrhoeic keratosis (5.4%)

21/25 (84)

Carcinoma of skin and lips (× 4)

Sebaceous cyst (3.3%)

12/13 (92.3)

Lipoma

Actinic keratosis (3.1%)

12/12 (100)

Others (2.8%)

8/11 (72.7)

Inflammatory skin diseases (27.1%)

Contact dermatitis (9.2%)

34/36 (94.4)

Psoriasis, parapsoriasis

Psoriasis (3.6%)

10/14 (71.4)

Contact dermatitis (× 3), varicella

Seborrhoeic dermatitis (3.3%)

13/13 (100)

Prurigo nodularis (2.1%)

7/8 (87.5)

Scabies

Others (9.0%)

32/35 (91.4)

Viral, bacterial and fungal infections (23.0%)

Viral wart (4.9%)

16/19 (84.2)

Carcinoma, and callosity (× 2)

Erysipelas (2.6%)

10/10 (100)

Tinea (2.3%)

8/9 (88.9)

Contact dermatitis

Onychomycoses (2.1%)

8/8 (100)

Others (13.0%)

48/51 (94.1)

Other dermatological conditions (20.5%)

72/80 (90)

a Listed diagnoses were observed more than 3 times.

Therapy agreement

Table II shows the concordance of telemedicine and face-to-face examination in terms of the therapy prescribed. Specifically, systemic and local therapy were prescribed in 31% and surgical treatment in 38% of face-to-face examinations. The LK agreement was always substantial agreement, even considering the different therapies, i.e. systemic, local or surgical. κ values ranged from 0.652 to 0.862 and the corresponding 95% CIs were sufficiently narrow (Table II for details).

Table II. Concordance of diagnosis and therapy between face-to-face examination and telemedicine (store-and-forward). CI = confidence interval

Observed agreement

Cohen’s κ

%

Estimate

95% CI

Diagnosis

91.05%

0.906

0.876–0.936

Systemic therapy

72.95%

0.652

0.617–0.687

Local therapy

73.77%

0.655

0.618–0.692

Surgical therapy

90.48%

0.862

0.832–0.892

Overall therapy

79.80%

0.701

0.663–0.739

Teledermatology time

Table III shows the descriptive statistics of TD times. Face-to-face and teledermatological examinations were divided into parts, each of which was timed. Min, max, mean time of each part, expressed in minutes and seconds, are reported. Times did not differ between lesion sites. An increment of a few minutes was sometimes observed for total body lesions.

Table III. Descriptive statistics of teledermatology (TD) times in minutes and seconds

Min–Max

Mean ± SD

Medical history time

0:30–17:45

4:30 ± 3:30

Face-to-face prescription time

3:20–38

10:30 ± 5:20

Total face-to-face time (medical history time + face-to-face prescription time)

6–40:30

15 ± 6:20

TD consent, photo and uploading time

2:30–10

4 ± 1:40

Total TD submission time (medical history time + TD consent, photo and uploading time)

3–22

8:30 ± 2:20

Consultant TD interpretation and response time

1:30–6

2:30 ± 1:50

Discussion

Many studies have demonstrated the feasibility and reliability of TD and shown that it reduces face-to-face examinations by 70–80%, with clear advantages for patients and healthcare systems (1, 4, 5, 7, 8). The results of these studies are so encouraging that new techniques and methods have recently been developed, e.g. the idea of using mobile phone technology in TD (3, 24–32). Mobile phone technology offers portability and the convenience of capturing clinical images and history on a single mobile device and transmitting them to the consulting dermatologist without the need of a computer. This prompted us to conduct the present clinical trial to compare conventional dermatological examination with store-and-forward telemedicine for patients with skin disorders, using new generation smartphone. Our results showed an almost perfect agreement between face-to-face and store-and-forward diagnosis (Tables I and II); indeed, 356/391 patients were diagnosed identically, giving a concordance of 91% (Cohen κ coefficient = 0.906).

The most frequent skin disorders encountered in our patients were benign and malignant skin growths (27.6%), which on the whole showed very high percentages of correct diagnosis (90.7%; 98/108) despite the fact that our study did not make use of epiluminescence images. Also for that reason 2 cases of malignant skin tumours were diagnosed as benign by TD (Table I). This underlines the absolute importance of dermoscopic image for skin tumour evaluations. We recall that in our study the doctor who sent images in TD was the prototype of a GP who usually does not perform dermoscopy. For the same reason we excluded all PSLs from our evaluations, which could be half of the referrals, since in our opinion it would seem anachronistic to evaluate PSLs without epiluminescence images.

The second most frequent group of skin disorders observed was inflammatory skin diseases such as papulo-squamous lesions, eczema, rashes, hives and vesicular/bullous lesions (27.1%). Table I shows that correct classification percentages were always above 90%, except for prurigo nodularis (7/8; 87.5%) and psoriasis (10/14; 71.4%). For prurigo nodularis, the correct classification percentage was just below 90% due to the small total number of cases observed (8). For psoriasis, however, the total number of cases was quite large (14) and out of 4 telemedicine errors, 3 were wrongly classified as contact dermatitis. These results again underline that in certain cases, such as palmoplantar and inverse forms, psoriasis is complex to diagnose. The third most frequent group of skin diseases was infections and parasites, the correct classification percentages of which were always around 90%, except for viral warts. In these cases the percentages were only slightly lower (84.2%; 16/19) and in 2 cases were wrongly diagnosed as “calluses and callosity” and in one case as “carcinoma of the skin and lips”.

As far as therapy is concerned, agreement between all the therapies prescribed by the 2 methods (face-to-face and TD) was slightly less than just seen for diagnosis (79.8%; Cohen’s κ = 0.701) (Table II). This may be influenced by subjectivity and by the variety of the prescriptions of the different dermatologists who alternated for assessment. However, substantial agreement was always found for skin diseases requiring systemic, surgical and local therapy. Specifically, surgery showed the highest concordance of the 3 types of therapy (133/147). This result, combined with the fact that the most frequent group of skin disorders was neoplasms, confirms that TD is a useful auxiliary and could be a valid alternative to conventional face-to-face examination for benign and other epithelial growths, which often require surgery (8, 33–35).

Our study is one of the first to use new generation smartphones that capture high quality (8 megapixel) digital images. The only similar studies are those of Lim et al. (36), Thind et al. (37) and Lasierra et al. (38), which obtained lower concordance than ours. This prompts us to assert that modern smartphones offer a valid alternative to digital cameras, with the advantage of being easier, more convenient and faster to use.

One of the main concerns with the implementation of TD in daily routine practice is the time needed to acquire images and patient data and to send them to the teledermatologist. We measured this time using a specially designed application for a mobile phone and we found that on average only 4 min needs to be added to a normal visit, with a minimum of 2.5 min and a maximum of 10 min (Table III). This means that a visit may last on average 19 min instead of 15 min. However, although TD prolongs a visit by approximately 25%, it can in some cases spare a dermatologic consultation, thus saving time for both patients and dermatologists. In countries like Holland, where TD services already exist, this has led to reducing the cost of a teleconsultation compared to conventional face-to-face visit by half (7). TD centres could also attract patients from distant regions and this could lead to a reduction in the cost of a consultation and save patients’ time. However, it is important to underline that the time it takes to recall the patient and for the GP to recount the expert opinion to the patient must be added to the total teleconsultation time. Another issue to take into consideration is that in approximately 9% of the patients telemedicine diagnosis was incorrect. For this reason the patient could need another face-to-face visit and the time needed for this second examination should be considered. This also applies for those who were diagnosed by telemedicine with serious or dangerous diseases, or whenever further investigations or a direct therapeutic approach was required.

This is the first study that measured the “teledermatology time” for mobile TD, which resulted much faster in a diagnosis than a digital camera connected to a computer. Studies that assessed the time invested by GP during store-and-forward TD consultation recorded times ranging from 6 to 20 min using digital cameras (39, 40). Also, the mean time it took for the teledermatologist to view the cases and write an answer was 2 min and 30 s, showing that a “teledermatologic consultation” is shorter than a face-to-face visit and that a teledermatologist can see more patients/h than an ordinary dermatologist.

The saying “time is money” is also valid for patients and health costs. Technological advances and the need to reduce health service costs will strongly promote the development of telemedicine applications. Telemedicine will probably become part of national health services just as smartphones have become important in our lives. Since health systems must reduce costs while maintaining good service, the dermatological community should be aware of these new technologies, seek sectors where they can be useful and make them part of daily clinical practice.

ACKNOWLEDGEMENTS

Funding sources: The study was carried out with financial support obtained through the Fellowship SIDeMaST (Società Italiana di Dermatologia medica, chirurgica, estetica e delle Malattie Sessualmente Trasmesse) 2009.

The authors declare no conflict of interest.

10054.png

1http://www.medicaljournals.se/acta/content/?doi=10.2340/00015555-1876

References

1. High WA, Houston MS, Calobrisi SD, Drage LA, McEvoy MT. Assessment of the accuracy of low-cost store and forward teledermatology consultation. J Am Acad Dermatol 2000; 42: 776–783.

2. Armstrong AW, Wu J, Kovarik CL, Goldyne ME, Oh DH, McKoy KC, et al. State of teledermatology programs in the United States. J Am Acad Dermatol 2012; 67: 939–944.

3. Nami N, Giannini E, Burroni M, Fimiani M, Rubegni P. Teledermatology: state of the art and future perspectives. Expert Rev Dermatol 2012; 7: 1–3.

4. Pak H, Triplett CA, Lindquist JH, Grambow SC, Whited JD. Store-and-forward teledermatology results in similar clinical outcomes to conventional clinic-based care. J Telemed Telecare 2007; 13: 26–30.

5. Edison KE, Ward DS, Dyer JA, Lane W, Chance L, Hicks LL. Diagnosis, diagnostic confidence, and management concordance in live-interactive and store-and-forward teledermatology compared to in-person examination. Telemed J E Health 2008; 14: 889–895.

6. Levin YS, Warshaw EM. Teledermatology: a review of reliability and accuracy of diagnosis and management. Dermatol Clin 2009; 27: 163–176.

7. van der Heijden JP, de Keizer NF, Bos JD, Spuls PI, Witkamp L. Teledermatology applied following patient selection by general practitioners in daily practice improves efficiency and quality of care at lower cost. Br J Dermatol 2011; 165: 1058–1065.

8. Rubegni P, Nami N, Cevenini G, Poggiali S, Hofmann-Wellenhof R, Massone C, et al. Geriatric teledermatology: store-and-forward vs. face-to-face examination. J Eur Acad Dermatol Venereol 2011; 25: 1334–1339.

9. Romero G, Sánchez P, García M, Cortina P, Vera E, Garrido JA. Randomized controlled trial comparing store-and-forward teledermatology alone and in combination with web-camera videoconferencing. Clin Exp Dermatol 2010; 35: 311–317.

10. Romero G, Sánchez P, García M, Cortina P, Vera E, Garrido JA. Evaluation and comparison of store-and-forward teledermatology applications. Telemed J E Health 2010; 16: 424–438.

11. Vañó-Galván S, Hidalgo A, Aguayo-Leiva I, Gil-Mosquera M, Ríos-Buceta L, Plana MN, et al. Store-and-forward teledermatology: assessment of validity in a series of 2000 observations. Actas Dermosifiliogr 2011; 102: 277–283.

12. Burdick AE, Simmons SC. The teledermatology train is coming: get on board, get out of the way, or get run over. Cutis 2011; 88: 213–214.

13. Krupinski EA, LeSueur B, Ellsworth L, Levine N, Hansen R, Silvis N, et al. Diagnostic accuracy and image quality using a digital camera for teledermatology. Telemed J 1999; 5: 257–263.

14. Krupinski E, Burdick A, Pak H, Bocachica J, Earles L, Edison K, et al. American Telemedicine Association’s Practice Guidelines for Teledermatology. Telemed J E Health 2008; 14: 289–302.

15. Massone C, Wurm EM, Hofmann-Wellenhof R, Soyer HP. Teledermatology: an update. Semin Cutan Med Surg 2008; 27: 101–105.

16. Ebner C, Wurm EM, Binder B, Kittler H, Lozzi GP, Massone C, et al. Mobile teledermatology: a feasibility study of 58 subjects using mobile phones. J Telemed Telecare 2008; 14: 2–7.

17. Farshidi D, Craft N, Ochoa MT. Mobile teledermatology: As doctors and patients are increasingly mobile, technology keeps us connected. Skinmed 2011; 9: 231–238.

18. Berndt RD, Takenga MC, Kuehn S, Preik P, Dubbermann D, Juenger M. Development of a mobile teledermatology system. Telemed J E Health 2012; 18: 668–673.

19. Luxton DD, Kayl RA, Mishkind MC. mHealth data security: the need for HIPAA-compliant standardization. Telemed J E Health 2012; 18: 284–288.

20. Thompson WD, Walter SD. A reappraisal of the kappa coefficient. J Clin Epidemiol 1988; 41: 949–958.

21. Sim J, Wright CC. The Kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther 2005; 85: 257–268.

22. Landis JR, Koch GG. The measurement of observer agree­ment for categorical data. Biometrics 1977; 33: 159–174.

23. Cai JJ, Smith DK, Xia X, Yuen KY. MBEToolbox: a MATLAB toolbox for sequence data analysis in molecular biology and evolution. BMC Bioinformatics 2005; 6: 64.

24. Tran K, Ayad M, Weinberg J, Cherng A, Chowdhury M, Monir S, et al. Mobile teledermatology in the developing world: implications of a feasibility study on 30 Egyptian patients with common skin diseases. J Am Acad Dermatol 2011; 64: 302–309.

25. Massone C, Hofmann-Wellenhof R, Ahlgrimm-Siess V, Gabler G, Ebner C, Soyer HP. Melanoma screening with cellular phones. PLoS One 2007; 2: e483.

26. Massone C, Lozzi GP, Wurm E, Hofmann-Wellenhof R, Schoellnast R, Zalaudek I, et al. Personal digital assistants in teledermatology. Br J Dermatol 2006; 154: 801–802.

27. Frühauf J, Hofman-Wellenhof R, Kovarik C, Mulyowa G, Alitwala C, Soyer HP, et al. Mobile teledermatology in Sub-Saharan Africa: A uUseful tool in supporting health workers in low-resource centres. Acta Derm Venereol 2012; 93: 122–123.

28. Frühauf J, Schwantzer G, Ambros-Rudolph CM, Weger W, Ahlgrimm-Siess V, Salmhofer W, et al. Pilot study on the acceptance of mobile teledermatology for the home monitoring of high-need patients with psoriasis. Australas J Dermatol 2012; 53: 41–46.

29. Massone C, Lozzi GP, Wurm E, Hofmann-Wellenhof R, Schoellnast R, Zalaudek I, et al. Cellular phones in clinical teledermatology. Arch Dermatol 2005; 141: 1319–1320.

30. Massone C, Brunasso AM, Campbell TM, Soyer HP. Mobile teledermoscopy – melanoma diagnosis by one click? Semin Cutan Med Surg 2009; 28: 203–205.

31. Lamel SA, Haldeman KM, Ely H, Kovarik CL, Pak H, Armstrong AW. Application of mobile teledermatology for skin cancer screening. J Am Acad Dermatol 2012; 67: 576–581.

32. Kroemer S, Frühauf J, Campbell TM, Massone C, Schwantzer G, Soyer HP, et al. Mobile teledermatology for skin tumour screening: diagnostic accuracy of clinical and dermoscopic image tele-evaluation using cellular phones. Br J Dermatol 2011; 164: 973–979.

33. Hsiao JL, Oh DH. The impact of store-and-forward teledermatology on skin cancer diagnosis and treatment. J Am Acad Dermatol 2008; 59: 260–267.

34. Kanthraj GR. Teledermatology: its role in dermatosurgery. J Cutan Aesthet Surg 2008; 1: 68–74.

35. Moreno-Ramirez D, Ferrandiz L, Ruiz-de-Casas A, Nieto-Garcia A, Moreno-Alvarez P, Galdeano R, et al. Economic evaluation of a store-and-forward teledermatology system for skin cancer patients. J Telemed Telecare 2009; 15: 40–45.

36. Lim AC, Egerton IB, See A, Shumack SP. Accuracy and reliability of store-and-forward teledermatology: preliminary results from the St George Teledermatology Project. Australas J Dermatol 2001; 42: 247–251.

37. Thind CK, Brooker I, Ormerod AD. Teledermatology: a tool for remote supervision of a general practitioner with special interest in dermatology. Clin Exp Dermatol 2011; 36: 489–494.

38. Lasierra N, Alesanco A, Gilaberte Y, Magallón R, García J. Lessons learned after a three-year store and forward teledermatology experience using internet: Strengths and limitations. Int J Med Inform 2012; 81: 332–343.

39. van den Akker TW, Reker CH, Knol A, Post J, Wilbrink J, van der Veen JP. Teledermatology as a tool for communication between general practitioners and dermatologists. J Telemed Telecare 2001; 7: 193–198.

40. Berghout RM, Eminović N, de Keizer NF, Birnie E. Evaluation of general practitioner’s time investment during a store-and-forward teledermatology consultation. Int J Med Inform 2007; 76 Suppl 3: 384–391.

Supplementary content
Figure S1