Content » Vol 53, Issue 1

Original report

A TEAM APPROACH TO APPLYING THE INTERNATIONAL CLASSIFICATION OF FUNCTIONING, DISABILITY AND HEALTH REHABILITATION SET IN CLINICAL EVALUATION

Malan Zhang, MD, PhD1,2*, Yun Zhang, BMS3*, Yun Xiang, MD, PhD4, Ziling Lin, MMSc5, Wei Shen, MMSc6, Yingmin Wang, MSN1, Liyin Wang, BSc7, Jiani Yu, MD, PhD1, Tiebin Yan, MD, PhD2,8

From the 1Department of Exercise Rehabilitation, College of Exercise and Health, Guangzhou Sport University, 2Department of Rehabilitation Medicine, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, Guangzhou, 3Department of Rehabilitation Medicine, Xiamen Fifth Hospital, Xiamen, 4Department of Rehabilitation Medicine, Shenzhen Nanshan People’s Hospital, Shenzhen, 5Department of Rehabilitation Medicine, The Fifth Affiliated Hospital of Sun Yat-sen University, Zhuhai, 6Department of Rehabilitation Medicine, Guangdong 999 Brain Hospital, 7Department of Rehabilitation Medicine, Guangdong Clifford Hospital and 8Engineering Technology Research Center for Rehabilitation and Elderly Care, Sun Yat-sen University, Guangzhou, China

*These authors contributed equally to this article.

Abstract

Objective: To develop a team approach to applying the International Classification of Functioning, Disability and Health Rehabilitation Set (ICF-RS) in clinical evaluation.

Design: A Delphi study.

Subjects: Experts from rehabilitation institutions in China including physicians, nurses, physiotherapists and occupational therapists.

Methods: A 2-round Delphi survey and expert panel discussion were used to generate the team approach. Firstly, the candidate types of professionals for team rating were chosen through expert panel discussion. A carefully selected group of participants was then asked to score the suitability of physicians, nurses, or other candidate therapists for each category’s rating, applying the International Classification of Functioning, Disability and Health Rehabilitation Set in clinical evaluation. After initial assignment of category to types of professionals, a second round Delphi survey was conducted to quantify the professionals’ agreement with the category assignments and generate a final team evaluation approach.

Results: Thirty of the category assignments achieved consensus. The final team evaluation approach assigned 6 categories to physicians to evaluate, 7 categories to nurses, 9 categories to physiotherapists, and 8 to occupational therapists.

Conclusion: Such a team evaluation approach could facilitate implementation of the ICF-RS in clinical settings and provide a more convenient assessment tool for professionals.

Key words: International Classification of Functioning, Disability and Health Rehabilitation Set; Delphi studies; rehabilitation, team evaluation

Accepted Oct 6, 2020; Epub ahead of print Oct 19, 2020

J Rehabil Med 2021; 53: jrm00147

Correspondence address: Tiebin Yan, Department of Rehabilitation Medicine, Sun Yat-sen Memorial Hospital, Sun Yat-sen University, 107 Yan Jiang West Road, Guangzhou 510120, China. E-mail: yantb@mail.sysu.edu.cn

DOI: 10.2340/16501977-2756

Lay Abstract

As the Chinese International Classification of Functioning, Disability and Health Rehabilitation Set became widely used in rehabilitation institutions in China, it was found to be time-consuming and inefficient for a single rater to complete the entire evaluation in a single setting. Team evaluation provides an alternative team approach for busy professionals, especially when used to evaluate patients with complex problems or poor communication ability. The whole set of 30 evaluation categories was divided among a hypothetical team consisting of a physician, a nurse, a physiotherapist and an occupational therapist, with 6 categories assigned to the physician, 7 to the nurse, 9 to the physiotherapist, and 8 to the occupational therapist. Each professional in the team rated categories closely related to their daily work. The team evaluation approach promises to better share the evaluation workload, perhaps improving the accuracy of evaluations and strengthening interdisciplinary collaboration in the clinic.

Introduction

The International Classification of Functioning, Disability and Health (ICF) is a standardized framework for describing and understanding health, functioning and disability, which was endorsed by the World Health Assembly in 2001 (1). An individual’s functioning and level of disability may be recorded by selecting the appropriate ICF category and using its corresponding code and qualifiers (2). With the increasing number of people with disability, there have been calls to strengthen rehabilitation services in the world’s health systems (3). The ICF can be used as a unifying model for the conceptual description of rehabilitation strategies and as a useful tool for quantifying the effectiveness of rehabilitation services (4, 5). In order to facilitate clinical practice, the ICF Core Sets specify shortlists of ICF categories relevant to a particular health condition or setting. They have been rigorously developed to enhance the utility of the ICF in clinical practice and research (6, 7).

The ICF Rehabilitation Set (ICF-RS), with 30 categories, is a minimal set for reporting and assessing functioning and disability in clinical rehabilitation and in research (8–10). Its domains cover body functions, activities and participation. As the ICF Rehabilitation Set is essentially a list of categories, it needs to be considered from a measurement perspective (11, 12). A Chinese version of the ICF-RS has been developed, describing how to use it to measure aspects of functioning (13). The Chinese psychometric properties have shown good reliability and validity in multi-centre research, and they have been widely used as a functional evaluation tool in China (14).

However, as the Chinese ICF-RS became widely used, some problems were observed in its application. Using it involved clinical examination and interviews. That was time-consuming. A single rater would have difficulty finishing the entire evaluation in a single sitting, especially with a patient with complex complaints or poor language expression. Some raters reported that even 30 min to complete one evaluation was too long in a busy rehabilitation department. That undoubtedly limited the instrument’s application. In addition, the categories of the ICF-RS cover a wide range of functioning, and some categories were not closely related to the job responsibilities of some of the rehabilitation professionals using it. This may have resulted in a further decrease in overall attention and enthusiasm in its use.

A team approach may be a way to increase the evaluation efficiency and facilitate the process. In such an approach the assessment would be completed by a team of professionals, with each rater handling some categories consistent with his or her routine job responsibilities in the clinic. Ottiger has reported a new multidisciplinary observation scale for stroke patients, in which neurologists, nurses, physiotherapists, occupational therapists and speech therapists each rate assigned domains (15). A group led by Catz has reported a revised version of the Spinal Cord Independence Measure evaluated by a team including an occupational therapist, a nurse and a physiotherapist (16). Both of those scales were found easier and more convenient to implement, as each professional in the team rated their own part of the scale. As yet, however, there have been no reports about a team version of the ICF Rehabilitation Core Set.

This study aimed to develop a team evaluation approach to using the Chinese version of the ICF-RS. It was designed to be applied in rehabilitation institutions in China, with the professionals in the team rating categories closely related to their daily work. The team approach was supported by a mobile app, which the team members could use to record and share their evaluation results (17).

METHODS

Overview

The objective of the team approach was to divide the original 30 categories into sections, and to have each section evaluated by a different rehabilitation professional in the team: one professional for each category. The study was conducted using a modified Delphi technique and an expert panel (18, 19). The process had 3 steps, as follows. (i) To establish a panel of ICF experts, which would screen candidate professionals for participating in the rating team through discussion. (ii) A first-round Delphi survey was conducted in which the participants were asked to score the candidate professionals using a 5-point Likert Scale in terms of their suitability for evaluating particular ICF-RS categories (20, 21). Table I presents the list of categories. A first draft of the team roster was then prepared according to the results combined with discussion among the experts. (iii) A second-round survey was then conducted in which the participants were asked to rate their agreement with the category assignments. Ethics approval for this procedure was obtained from the ethics committee of Sun Yat-sen Memorial Hospital (Guangzhou, China). Written informed consent was obtained from all participants.


Table I. International Classification of Functioning, Disability and Health Rehabilitation Set (ICF-RS) categories

Screening basis

An expert panel was set up to screen professionals for the rating team. The experts were all from the Rehabilitation Department of Sun Yat-sen Memorial Hospital, Sun Yat-sen University (Guangzhou, China). They were a senior physician, a senior nurse, a senior physiotherapist, and a senior occupational therapist. All of the experts had at least 5 years of experience working in a rehabilitation department and had worked with the ICF for more than 3 years.

The candidate professionals were evaluated through item-by-item discussion of each category. Two main questions were proposed to help the screening: (i) What kinds of professionals are in contact with or pay attention to this category’s content in their daily work? (ii) What kinds of professionals are involved with this category using other scales in a rehabilitation clinic? The discussion was based on the definition and evaluation content of each category, but constrained, to some extent, by the availability of professionals in rehabilitation departments in China.

First-round Delphi survey

Participants. Purposive snowball sampling was used to enrol participants (22). The sample size of participants was not clearly defined for the Delphi survey, and many studies reported the participant numbers ranged from 10 to 50 (23–25). This study aimed to enrol 25–30 experts for the Delphi survey. The initial recruiting exploited an online expert group specialized in ICF research in China. Group members were asked to volunteer. Those who volunteered were then asked to recommend other relevant ICF experts. Some experts who had participated in a previous Delphi survey about the psychometric properties of Chinese ICF-RS (13) were also selected. The members of the expert panel did not participate in the 2 rounds of Delphi surveys.

Geographical representation was taken into account in the recruitment process. Those included were all registered senior physicians, senior nurses, attending or senior therapists. All had at least 10 years of clinical experience in hospitals, including at least 5 years of clinical experience in rehabilitation and also 2 years of experience related to the ICF. One nurse who was not at the senior level was included because the ICF-RS was the topic of her doctoral research.

Design

An invitation letter, describing the survey’s purpose, method, content and matters needing attention, was sent to the participants by e-mail or WeChat (Weixin, Beijing, Tencent Technology Co., LTD), inviting them to participate. A questionnaire was then sent to those who agreed to participate. The questionnaire had 2 parts: expert consultation for team evaluation; and personal information about the respondent.

In the expert consultation part, the respondents were asked to evaluate if a physician, a nurse, or a therapist was suitable for assessing each ICF-RS category. The sub-questions for each category were designed based on the types of candidate professionals. The respondents used a 5-point response format from “strongly unsuitable” (quantified as 1) to “strongly suitable” (quantified as 5) for each question. Only one option could be selected in response to each question. The evaluation content and evaluation options for each category of the ICF-RS were provided for the respondents’ reference (Fig. 1).


Fig. 1. An example of the questionnare in the first-round survey. Four sub-questions for the category “b280 Sensation of pain”.

The personal information collected about the participants included their name, age, gender, education, profession, title, years using the ICF and (self-assessed) degree of familiarity with the ICF.

It took approximately 10–15 min to complete the questionnaire. The respondents were asked to return it within 2 weeks. If no reply had been received by 3 days before the deadline, the respondent was reminded by the coordinator. All of the responses collected were checked, and if there were any omissions the respondent was contacted and asked to complete the responses. Eventually, a questionnaire with missing responses was deemed invalid.

If only one type of professional was rated as “basically suitable”, “suitable” or “strongly suitable” (a score of 3, 4 or 5) for assessing a particular category by more than 75% of the respondents, and if the mean suitability score was ≥ 3.6, that type of professional was considered appropriate for that category without any further discussion (13, 26, 27). In most cases, however, more than one type of professional met that standard. In those cases, the category assignment was considered through discussion among the expert panel. The mean suitability scores, the balance of workload and the relationship with a profession’s routine work were major factors to be considered in assignment. If no profession met that standard, then the profession rated as “basically suitable”, “suitable” or “strongly suitable” (a score of 3, 4 or 5) was chosen by the majority of the respondents.

Second-round Delphi survey

The respondents who participated in the first-round survey received a second questionnaire by email or WeChat. That questionnaire reported the results of the first-round survey and the initial category assignments. The respondents were asked to consider to what extent they agreed with the assignments. They replied using a 5-point Likert scale: 1, “strongly disagree”; 2, “disagree”; 3, “doesn’t matter”; 4, “agree”; and 5, “strongly agree”. Two weeks were allowed to return their responses, and a reminder was sent 3 days before the deadline by a coordinator. According the standard for consensus generally used in such research (13, 26, 27), an assignment was accepted if it had a mean agreement score ≥ 3.6 and more than 75% of the respondents chose “agree” or “strongly agree” (Likert scores of 4 and 5). Any category with a mean agreement score below 3.6 or total agreement among the respondents of less than 75% was re-assigned in discussion among the expert panel considering the results from the first Delphi round and based on their clinical experience.

Statistical analysis

Version 20.0 of the SPSS software suite (IBM, Armonk, NY, USA) was used for statistical analysis. The suitability and expert agreement scores for each category were calculated and expressed as mean (standard deviation (SD)). The higher the mean, the better the agreement about a category assignment.

RESULTS

Screening of professionals

The Chinese version of the ICF-RS was generated to be used in rehabilitation institutions in China, but many of these institutions do not employ personnel across the full set of professional categories. As a result, some of the work of speech therapists, psychotherapists and social workers is, in fact, done by other types of professionals. Through discussion, it was decided that a speech therapist was only related to category “d550 Eating”. A psychotherapist was related to the category “b152 Emotional functions”, and a social worker was related to category “d850 Remunerative employment”. All of those categories could, it was decided, also be evaluated by other sorts of professionals instead. Thus, speech therapists, psychotherapists and social workers were excluded from the evaluations. Physicians, nurses, physiotherapists and occupation therapists were eventually selected to participate in the rating team.

Results of the first-round survey

Of the 31 experienced rehabilitation professionals initially recruited for the Delphi survey, 30 (96.8%) completed the questionnaire adequately (11 physicians, 9 nurses and 10 therapists). The demographics and qualifications of these professionals are shown in Table II.


Table II. Demographic characteristics and experience of the professionals

Among the 30 categories, there were 22 in which only one type of professional got a suitability percentage ≥ 75% and a mean suitability score ≥ 3.6. Those selections were accepted without further discussion. Of the 22, there were 4 categories for which a physician was clearly considered the most suitable. 2 was assigned to a nurse, 9 were assigned to a physiotherapist, and 7 to an occupational therapist. There were 8 categories in which at least 2 types of professionals got a suitability percentage ≥ 75% and a suitability score ≥ 3.6. Through discussion, the expert panel assigned them to be evaluated by a physician or a nurse in an attempt to share the workload. “d240 Handling stress and other psychological demands” was assigned to the physicians. “d510 Washing oneself”, “d520 Caring for body parts”, “d530 Toileting” and “d550 Eating” were assigned to nurses. The nurses also received “d230 Carrying out daily work” and “d570 Looking after one’s health”, although 3 types of professionals were considered suitable. The category “b130 Energy and drive functions” met that standard with 4 types of professionals, but it was assigned to the physicians (Fig. 2). The details of the initial category assignments, the suitability rate and the mean suitability scores are listed in Table III.


Fig. 2. Results of the first round of the Delphi survey. The 8 categories in which 2 or more professions met the standard were assigned through expert panel discussion


Table III. First round suitability scores using a 5-point response scale

Results of the second-round survey

The 30 respondents who participated in the first expert survey also received the second-round questionnaire. Of these, 29 (96.7%) responded adequately, including 10 physicians, 9 nurses and 10 therapists. In the second round, 29 categories received more than 75% agreement, with the mean agreement scores ranging from 3.79 to 4.72 (Table IV). Category “d230 Carrying out daily routine” received less than 70% agreement, so it was assigned to the occupational therapists in view of the fact that it was the assignment that received the highest mean score in the first-round survey.

Results of the second-round survey

The 30 respondents who participated in the first expert survey also received the second-round questionnaire. Of these, 29 (96.7%) responded adequately, including 10 physicians, 9 nurses and 10 therapists. In the second round, 29 categories received more than 75% agreement, with the mean agreement scores ranging from 3.79 to 4.72 (Table IV). Category “d230 Carrying out daily routine” received less than 70% agreement, so it was assigned to the occupational therapists in view of the fact that it was the assignment that received the highest mean score in the first-round survey.


Table IV. Final assignments after the second round of Delphi survey

DISCUSSION

This study attempted to develop a team approach to facilitate use of the Chinese version of the ICF-RS in evaluations. Team evaluation provides an alternative approach for busy professionals, especially when used to evaluate patients with complex problems or poor communication ability. The professional analysis assigned 6 categories to physicians, 7 to nurses, 9 to physiotherapists and 8 to occupational therapists.

The Delphi technique used in this study successfully developed a consensus among the experts contributing through repeated information exchange and feedback (28). Unlike with the classic Delphi technique, the modified Delphi method used here did not apply the conventional 4 rounds (29–31). The consultation ended after 2 rounds, as the experts had reached a consensus. A review of 30 studies using the Delphi technique reveals they reached consensus after 1–5 rounds. Among them are 14 studies which reported requiring only 2 Delphi rounds (32). They were mostly generating standards, guidelines and scales (33–35). Some reports described using intermediate face-to-face meetings between Delphi survey rounds (36) or the involvement of different expert panels in the consensus process (37). As there were no reports about how to divide the same scale into sections to generate another rating version, in this study expert panel discussion was combined with 2 rounds of Delphi survey to successfully develop the team approach.

Finally, relatively few categories were assigned to the physicians and nurses. However, as the goal of the team approach was to share out the evaluation tasks to facilitate the process while ensuring the relevance and the accuracy of the results, which was considered satisfactory. All of the assignments met the consensus standard and seemed relevant to actual job responsibilities in a clinic.

The category “d240 Handling stress and other psychological demands” refers to regulating and controlling one’s mental state in order to accomplish tasks that may be complicated by stress and/or distraction. It emphasizes completing tasks under stress rather than simple physical movement and coordination. Physicians pay more attention to patients’ psychological adjustment than therapists normally would, so assigning this category to physicians received good support in the second round of the survey. The category “b130 Energy and the drive function” refers to physical fitness and the executive initiative required to achieve general goals and meet special needs. It too emphasizes the patient’s mental state, but also physical strength and energy. Because low energy and fatigue in the rehabilitation process are often related to inconsistent training intensity, or perhaps to heart failure or other factors, physicians can better grasp the patient’s basic difficulty and its causes and carry out more comprehensive and accurate assessments. The categories “d510 Washing oneself”, “d520 Caring for body parts”, “d530 Toileting” and “d550 Eating” were developed based on the content of the Modified Barthel Index. That index is a rating scale for ability in the activities of daily living, which is commonly used in rehabilitation in China. Research has shown that Barthel Index scoring by nurses agrees well with that by physicians (38).

In the second round of the survey the assignment of category “d230 Carrying out daily routine” generated less than 70% agreement. It was eventually assigned to the occupational therapists. The category refers to planning, scheduling and completing the daily routine, including controlling one’s activity level. The goal of occupational therapy is to create meaningful and satisfying lives through targeted and well-designed activities. Much of occupational therapy relates to patients’ daily lives and work, improving their quality of life and returning them to their families and society (39). Hence, d230 was considered consistent with the scope of practice and professional responsibilities of occupational therapists.

While this study related entirely to China’s rehabilitation context, medical professionals and their skills and patients and their problems have many similarities worldwide. This study’s assignments may therefore be helpful for rehabilitation personnel attempting to apply the ICF-RS in many different settings. The professionals who completed the questionnaire all had rich experience and deep understanding of rehabilitation practice. It might, however, have been better to have provided a job responsibility guide for their reference. That might have helped them to make easier and more consistent choices. In addition, although consensus was reached in 2 rounds, a third round of Delphi surveying would perhaps have been useful for research purposes. It is possible that some deviation from the consensus might have emerged in a third round. More research is needed to demonstrate that the team evaluation approach developed here has validity consistent with that of the conventional single-rater procedure. This will be the next stage of our research.

CONCLUSION

A new team approach to applying the ICF-RS in evaluation was generated in this study. The whole set of 30 evaluation categories was divided among a hypothetical team consisting of a physician, a nurse, a physiotherapist and an occupational therapist, with 6 categories assigned to the physician, 7 to the nurse, 9 to the physiotherapist and 8 to the occupational therapist. A limited number of categories can be finished easily during a professional’s work day without undue pressure. Improving the evaluation efficiency of the individual team members inevitably facilitates the implementation of the ICF-RS in the clinic. The team evaluation approach promises to better share the evaluation workload, perhaps improving the accuracy of evaluations and strengthening interdisciplinary collaboration in the clinic.

ACKNOWLEDGEMENTS

The authors thank all the participants in the Delphi study. Funding. This study was funded as a Ministry of Education Tian Cheng Hui Zhi Innovative Education Project (number 2018A01026) and as a Guangzhou Science and Technology Plan Project (number 201704020140).

The authors have no conflicts of interest to declare.

REFERENCES
1. World Health Organization (WHO). International Classification of Functioning, Disability and Health. Geneva: WHO; 2001.

2. World Health Organization (WHO). How to use the ICF: a practical manual for using the International Classification of Functioning, Disability and Health (ICF). Geneva: WHO; 2013.

3. Gimigliano F, Negrini S. The World Health Organization "Rehabilitation 2030-a call for action". Eur J Phys Rehabil Med 2017; 53: 155-168.

4. Stucki G, Cieza A, Melvin J. The International Classification of Functioning, Disability and Health: a unifying model for the conceptual description of the rehabilitation strategy. J Rehabil Med 2007; 39: 279-285.
DOI: https://doi.org/10.2340/16501977-0041

5. Ponomarenko GN. The International Classification of Functioning, Disability and Health as a tool for the scientifically grounded estimation of the effectiveness of medical rehabilitation. Vopr Kurortol Fizioter Lech Fiz Kult 2013; 2: 57-62.

6. EwertT, Üstün TB, Chatterji S, Kostanjsek N, Stucki G, Cieza A. Development of ICF core sets for patients with chronic conditions. J Rehabil Med 2004; 36: 9-11.
DOI: https://doi.org/10.1080/16501960410015353

7. Selb M, Escorpizo R, Kostanjsek N, Cieza A, Stucki G, Üstün B. A guide on how to develop an International Classification of Functioning, Disability and Health core set. Eur J Phys Rehabil Med 2015; 51: 105-117.

8. Prodinger B, Reinhardt JD, Selb M, Stucki G, Yan T, Zhang X, et al. Towards system-wide implementation of the International Classification of Functioning, Disability and Health (ICF) in routine practice: developing simple, intuitive descriptions of ICF categories in the ICF generic and rehabilitation set. J Rehabil Med 2016; 48: 508-514.
DOI: https://doi.org/10.2340/16501977-2066

9. Prodinger B, Cieza A, Oberhauser C, Bickenbach J, Üstün TB, Chatterji S, et al. Toward the International Classification of Functioning, Disability and Health (ICF) rehabilitation set: a minimal generic set of domains for rehabilitation as a health strategy. Arch Phys Med Rehabil 2016; 97: 875-884.
DOI: https://doi.org/10.1016/j.apmr.2015.12.030

10. Gimigliano F, Gastaldo M, Maghini I, Paoletta M, Pasquini A, Boldrini P, et al. Use of the International Classification of Functioning, Disability and Health generic-30 set for the characterization of outpatients: Italian Society of Physical and Rehabilitative Medicine residents section project. Eur J Phys Rehabil Med 2019; 55: 258-264.
DOI: https://doi.org/10.23736/S1973-9087.18.05324-8

11. Li J, Prodinger B, Reinhardt J, Stucki G. Towards the system-wide implementation of the International Classification of Functioning, Disability and Health in routine practice: Lessons from a pilot study in China. J Rehabil Med 2016; 48: 502-507.
DOI: https://doi.org/10.2340/16501977-2067

12. Stucki G, Prodinger B, Bickenbach J. Four steps to follow when documenting functioning with the International Classification of Functioning, Disability and Health. Eur J Phys Rehabil Med 2017; 53: 144-149.

13. Gao Y, Yan T, You L, Li K, Zhang L. Developing operational items for the International Classification of Functioning, Disability and Health Rehabilitation Set: the experience from China. Int J Rehab Res 2018; 41: 20-27.
DOI: https://doi.org/10.1097/MRR.0000000000000254

14. Gao Y, Yan T, You L, Li K, Zhang L. [The reliability and validity analysis of International Classification of Functioning, Disability and Health Rehabilitation Set in non-acute patients.] Chin J Rehab Med 2019; 34: 1193-1198 (in Chinese).

15. Ottiger B, Vanbellingen T, Gabriel C, Huberle E, Koenig-Bruhin M, Pflugshaupt T, et al. Validation of the new Lucerne ICF based Multidisciplinary Observation Scale (LIMOS) for stroke patients. PLoS One 2015; 10: e0130925.
DOI: https://doi.org/10.1371/journal.pone.0130925

16. Catz A, Itzkovich M, Steinberg F, Philo O, Ring H, Ronen J, et al. The Catz-Itzkovich SCIM: a revised version of the Spinal Cord Independence Measure. Disabil Rehabil 2001; 23: 263-268.
DOI: https://doi.org/10.1080/096382801750110919

17. Zhang M, Yu J, Shen W, Zhang Y, Xiang Y, Zhang X, et al. A mobile app implementing the international classification of functioning, disability and health rehabilitation set. BMC Med Inform Decision Making 2020; 20: 12-22.
DOI: https://doi.org/10.1186/s12911-020-1019-1

18. Skinner EH, Thomas P, Reeve JC. Minimum standards of clinical practice for physiotherapists working in critical care settings in Australia and New Zealand: a modified Delphi technique. Physiother Theory Pract 2016; 32: 468-482.
DOI: https://doi.org/10.3109/09593985.2016.1145311

19. Matsuoka YJ, Okubo R, Shimizu Y, Tsuji K, Narisawa T, Sasaki J, et al. Developing the structure of Japan's cancer survivorship guidelines using an expert panel and modified Delphi method. Cancer Surviv 2020; 14: 273-283.
DOI: https://doi.org/10.1007/s11764-019-00840-3

20. Amgarth-Duff I, Hosie A, Caplan G, Agar M. Toward best practice methods for delirium biomarker studies: an international modified Delphi study. Int J Geriatr Psychiatry 2020; 35: 737-748.
DOI: https://doi.org/10.1002/gps.5292

21. Short HL, Taylor N, Piper K, Raval MV. Appropriateness of a pediatric-specific enhanced recovery protocol using a modified Delphi process and multidisciplinary expert panel. J Pediatr Surg 2018; 53: 592-598.
DOI: https://doi.org/10.1016/j.jpedsurg.2017.09.008

22. Li K, Yan T, You L, Xie S, Li Y, Tang J, et al. The inter-rater reliability of the International Classification of Functioning, Disability and Health set for spinal cord injury nursing. Int J Rehabil Res 2016; 39: 240-248.
DOI: https://doi.org/10.1097/MRR.0000000000000176

23. Diamond IR, Grant RC, Feldman BM, Pencharz PB, Ling SC, Moore AM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting of Delphi studies. J Clin Epidemiol 2014; 67: 401-409.
DOI: https://doi.org/10.1016/j.jclinepi.2013.12.002

24. Sakashita A, Kizawa Y, Kato M. Development of a standard for hospital-based palliative care consultation teams in Japan using a modified Delphi method. J Pain Symptom Manage 2018; 56: 746-751.
DOI: https://doi.org/10.1016/j.jpainsymman.2018.08.013

25. Blumberg JB, Cena H, Barr SI. The use of multivitamin/multimineral supplements: a modified Delphi consensus panel report. Clin Ther 2018; 40: 640-657.
DOI: https://doi.org/10.1016/j.clinthera.2018.02.014

26. Guan L, Gao P, Liu S, Liu Y, Li X, Liu F, et al. Development of a global health bachelor curriculum in China: a Delphi study. BMJ Open 2019; 9: e023893.
DOI: https://doi.org/10.1136/bmjopen-2018-023893

27. van de Ven-Stevens LA, Graff MJ, Selles RW, Schreuders TA, van der Linde H, Spauwen PH, et al. Instruments for assessment of impairments and activity limitations in patients with hand conditions: a European Delphi study. J Rehabil Med.2015; 47: 948-956.
DOI: https://doi.org/10.2340/16501977-2015

28. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi survey technique. J Adv Nurs 2000; 32: 1008-1015.
DOI: https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x

29. van Vliet DC, van der Meij E, Bouwsma EV, Vonk Noordegraaf A, van den Heuvel B, Meijerink WJ, et al. A modified Delphi method toward multidisciplinary consensus on functional convalescence recommendations after abdominal surgery. Surg Endosc 2016; 30: 5583-5595.
DOI: https://doi.org/10.1007/s00464-016-4931-9

30. Short HL, Taylor N, Piper K. Appropriateness of a pediatric-specific enhanced recovery protocol using a modified Delphi process and multidisciplinary expert panel. J Pediatr Surg 2018; 53: 592-598.
DOI: https://doi.org/10.1016/j.jpedsurg.2017.09.008

31. Kotecha A, Longstaff S, Azuara-Blanco A. Developing standards for the development of glaucoma virtual clinics using a modified Delphi approach. Br J Ophthalmol 2018; 102: 531-534.
DOI: https://doi.org/10.1136/bjophthalmol-2017-310504

32. Jünger S, Payne SA, Brine J, Radbruch L, Brearley SG. Guidance on conducting and reporting Delphi studies (CREDES) in palliative care: recommendations based on a methodological systematic review. Palliat Med 2017; 31: 684-706.
DOI: https://doi.org/10.1177/0269216317690685

33. Dreesen M, Foulon V, Hiele M. Quality of care for cancer patients on home parenteral nutrition: development of key interventions and outcome indicators using a two-round Delphi approach. Support Care Cancer 2013; 21: 1373-1381.
DOI: https://doi.org/10.1007/s00520-012-1679-1

34. Dreesen M, Foulon V, Vanhaecht K. Development of quality of care interventions for adult patients on home parenteral nutrition (HPN) with a benign underlying disease using a two-round Delphi approach. Clin Nutr 2013; 32: 59-64.
DOI: https://doi.org/10.1016/j.clnu.2012.05.006

35. Madsen MM, Eiset AH, Mackenhauer J, Odby A, Christiansen CF, Kurland L, et al. Selection of quality indicators for hospital-based emergency care in Denmark, informed by a modified-Delphi process. Scand J Trauma Resusc Emerg Med 2016; 24: 11.
DOI: https://doi.org/10.1186/s13049-016-0203-x

36. Morita T, Bito S, Koyama H, Uchitomi Y, Adachi I. Development of a national clinical guideline for artificial hydration therapy for terminally ill patients with cancer. J Palliat Med 2007; 10: 770-780.
DOI: https://doi.org/10.1089/jpm.2006.0254

37. Hawryluck LA, Harvey WRC, Lemieux-Charles L, Singer PA. Consensus guidelines on analgesia and sedation in dying intensive care unit patients. BMC Med Ethics 2002; 3: E3.
DOI: https://doi.org/10.1186/1472-6939-3-3

38. Klemenc-Ketis Z, Makivić I, Poplas Susic A. The development and validation of a new interprofessional team approach evaluation scale. PLOS One 2018; 13: e0201385.
DOI: https://doi.org/10.1371/journal.pone.0201385

39. Katz N, Bar-Haim Erez A, Livni L, Averbuch S. Dynamic Lowenstein occupational therapy cognitive assessment: evaluation of potential to change in cognitive performance. Am J Occup Ther 2012; 66: 207-214.
DOI: https://doi.org/10.5014/ajot.2012.002469

Comments

Do you want to comment on this paper? The comments will show up here and if appropriate the comments will also separately be forwarded to the authors. You need to login/create an account to comment on articles. Click here to login/create an account.
Advertisement