|Year : 2017 | Volume
| Issue : 5 | Page : 8-14
Hybrid tool for assessment of professionalism among dental undergraduate students
Eswara Uma1, Abdul Hj Ismail Rashid1, Adinegara Lutfi Abas2, Sowmya Nettem3, Sumanth Kumbargere Nagraj4, Noorliza Mastura5
1 Department of Pediatric Dentistry, Community Dentistry and Dean, FOD, Melaka Manipal Medical College, Melaka, Malaysia
2 Department of Community Medicine and Deputy Dean, FOM, Melaka Manipal Medical College, Melaka, Malaysia
3 Department of Periodontics, FOD, Melaka Manipal Medical College, Melaka, Malaysia
4 Department of Oral Diagnosis and Radiology, FOD, Melaka Manipal Medical College, Melaka, Malaysia
5 Department of Community Dentistry, FOD, Melaka Manipal Medical College, Melaka, Malaysia
|Date of Submission||24-Apr-2017|
|Date of Acceptance||24-Oct-2017|
|Date of Web Publication||20-Dec-2017|
Dr. Eswara Uma
Department of Pediatric Dentistry, FOD, Melaka Manipal Medical College, Jalan Batu Hampar, Bukit Baru, Melaka 75150
Source of Support: None, Conflict of Interest: None
| Abstract|| |
Context: Of the several methods available for assessment of professionalism, there is still no consensus on an ideal tool for dental undergraduate (UG) students. Aims: The study aims to use a hybrid tool for assessment of professionalism among dental undergraduate students. Settings and Design: Cross-sectional design with purposive sampling. Subjects and Methods: All final year UG dental students participated in this study. Evaluation of knowledge about professionalism was through written test. Professional behavior of each final year student in a clinical setting was assessed with a prevalidated questionnaire of multisource feedback (MSF). The scores of written test and the MSF were calculated for each student. Data were analyzed to evaluate scores of knowledge and MSF scores as per assessor category. Correlation between knowledge scores and MSF was evaluated. Student perceptions were taken toward assessment of professionalism. Statistical Analysis Used: Statistical analysis was done using descriptive statistics. Pearson's coefficient was used to determine the correlation between average knowledge scores and the MSF scores. Results: Knowledge scores were significantly more for female students (P < 0.05, t-test). Patients rated the students highest. Correlation between knowledge and MSF scores was found to be statistically significant (Pearson's correlation, P < 0.01). Students gave feedback that assessment of professionalism should be done from the beginning of the clinical years. Conclusions: Evaluation revealed that knowledge toward professionalism correlated with the professional behavior implying association between knowledge and reasons for a particular action.
Keywords: Assessment, multisource feedback, professionalism
|How to cite this article:|
Uma E, Ismail Rashid AH, Abas AL, Nettem S, Nagraj SK, Mastura N. Hybrid tool for assessment of professionalism among dental undergraduate students. Int J App Basic Med Res 2017;7, Suppl S1:8-14
|How to cite this URL:|
Uma E, Ismail Rashid AH, Abas AL, Nettem S, Nagraj SK, Mastura N. Hybrid tool for assessment of professionalism among dental undergraduate students. Int J App Basic Med Res [serial online] 2017 [cited 2018 Apr 22];7, Suppl S1:8-14. Available from: http://www.ijabmr.org/text.asp?2017/7/5/8/221296
| Introduction|| |
The word professionalism comprises of the goals, behaviors, and attributes that characterize a profession. It is an abstract concept that includes a high degree of skill and knowledge related to work.,, However, the level of professionalism exhibited by any health-care professional is determined by the personality of the professional and factors related to his culture and perceived wishes of the patients.,
To evaluate professionalism several assessment systems have been proposed. The types of assessment fall into four main categories, which are; written assessment, competency-based assessment, performance-based assessment and portfolios. For the assessment of professionalism, it has been advocated that multiple approaches that test knowledge and skills should be employed. Knowledge can be assessed by written assessments in the format of selected response (multiple-choice question or questionnaire) or constructed response (essays, simulation formats)., Skills can be assessed by 360° degree reviews, objective structured clinical examinations, review of patient complaints, and many other methods. While written assessment is a good way of assessing the knowledge, it is too restricted to be useful as the only method of assessment of professionalism, which is a complex construct. Further, written assessment alone may encourage students to memorize the “correct” answer, and will not influence their actual behavior. Most of the existing methods assess either “Knows” or “Does” components only. It has been suggested that assessment of professional behavior alone is not enough, what is also needed is assessment of knowledge for the reason behind the action taken.
Combining written assessment with skills assessment can overcome the limitations of written assessment. Also by combining different tools and by engaging multiple assessors in a variety of settings, validity of assessment can be improved. Several tools have been tested for the assessment of professionalism, but there has been no consensus as to which tools are validated and reliable in dental curriculum.
This study aimed to combine evaluation of knowledge regarding professionalism with multisource feedback as a hybrid tool for assessment of professionalism among dental undergraduate students. We hypothesized that there is no correlation between knowledge and demonstration of professionalism by final year dental undergraduate students as assessed by multisource feedback.
The following were the specific objectives of the study:
- To assess the knowledge (knows) regarding professionalism among the male and female final year dental undergraduate students
- To obtain multisource feedback (does) for the professionalism followed/demonstrated by the final year dental undergraduate students
- To assess correlation between the knowledge scores and multisource feedback scores
- To assess the scores of multisource feedback given by different evaluators and assess the correlation with the knowledge component
- To get feedback on this tool from the students.
| Subjects and Methods|| |
Permission to conduct the study and ethical waiver was granted by the institutional ethical committee (IRB#.MMMC/FOD/AR/E C -2016[F-01]). An institutional core committee comprising of seven faculties was formed to look into the logistics and implementation of the assessment tool. The core committee also looked into finalizing the domains of professionalism to be assessed for the students. Since professionalism is a construct with several components, our core committee decided to include honesty, integrity, confidentiality, communication skills, teamwork, and respect for patients as the components of professionalism to be evaluated as part of the knowledge assessment. For assessment of the skills related to professionalism, multisource feedback (MSF) was preferred. Individuals who interact with the students on a daily basis were chosen to be the assessors for MSF. Assessors for MSF therefore comprised of faculty, peers, patients, dental surgery assistants (DSA), laboratory assistants, and office staff.
Selection of multisource feedback assessment form
Of the various validated assessment forms.,, MSF assessment form by Chandler et al., which has been found to be user-friendly as well as been used in an outpatient setting, was adopted for the study. The form was a modified version of the one developed by Joshi et al. It was a ten-item questionnaire with responses on a five-point Likert scale with the highest score of 5. Lowest score (0) was “never,” and the highest score (5) was “always.” Each assessor had to complete each question on the 10-item assessment form. The form was available for faculty, nurses/support staff/peer, patients, and self-assessment. For the student assessment by the patient, the original form was translated into the local language, i.e., Bahasa Melayu. Backward translation of the translated form was done to ensure that the validity of the questionnaire did not change due to translation. Once translated the questionnaire was tested on patients to check the validity.
Written assessment was conducted through constructed response to vignettes posed as questions related to professionalism. A question paper for one hour with five questions comprising of 5 marks each with maximum marks of 25 was prepared with the help of content experts. The question paper comprised of 5 vignettes, each vignettes on a domain related to professionalism (honesty, integrity, confidentiality, respect for patients, and communication skills). The content experts validated the question paper. The students gave responses to the vignette given. Three assessors evaluated the answer sheets to prevent any bias.
Sensitization for assessment of professionalism
Since assessment of professionalism was a new concept for all the faculty and support staff, a sensitization program for the faculty and support staff about assessment using MSF was carried out on different days. Thirty-six faculties and support staff (dental surgery assistants, laboratory assistants, and office staff) participated as assessors for multisource feedback. Students were sensitized toward assessment of professionalism through written exam and MSF on a different day. All the seventy-four students of the final year gave consent to participate in the study.
Feedback from students on assessment of professionalism
Feedback questionnaires for students were designed and modified after pretesting with colleagues of core committee.
Conducting multisource feedback assessment
The batch of 74 students was divided into 6 groups for their clinical postings. All the groups moved to a different department on a daily basis. Considering the size of the batch and number of staff available, it was decided that it would be prudent to collect the MSF during clinical postings so that all the faculty and support staff participate and we get a general feedback about the batch [Figure 1]. On the day of assessment, the packets containing the forms bearing the students name were handed over to each department DSA. The DSA further handed over the forms to the faculty, peers, and patient undergoing treatment by the student on that day. The packets containing forms for laboratory staff and office staff were handed over the same day. The faculty, DSA, and peers assessed the students on the same day and handed over the filled forms on the same day. The laboratory staff and office staff were given 2 weeks' time to assess the students as and when they interacted with them in either the laboratory or the office. While it was planned to collect patient feedback on the same day, unfortunately, due to last minute cancellation of appointment, it could not be done for some students only. The assessment of student by the patient was then rescheduled for the next appointment of the patient.
Assessment of students
First, the students were asked to do self-assessment as per the prescribed form. Each group of students posted in a particular department was assessed on two consecutive days by the faculty of that department. The whole batch was evaluated by 16 DSA who are attached to six departments. It was planned in such a way that each DSA evaluated between 4-5 students, on the day, the student was posted in their department. It was done for two days so that each student got evaluated twice by the DSA [Figure 2]. Each of the three dental laboratory technician evaluated 25 students, who were working in the laboratory during the duration of assessment. The three office staff including the receptionist, evaluated 25 students each. Patient who was treated by a student on his/her repeat visit, assessed the treating student on that visit. Each student assessed two of his/her peers. The peers to be assessed were allocated by randomizing the class list.
|Figure 2: Total number of feedbacks for multisource feedback per student|
Click here to view
In total twice faculty, peers and DSA assessed each student, whereas, the office staff, laboratory assistant assessed the student once. Patient assessment was conducted once, and there was one self-assessment. Ten assessments per student were done. For the statistical analysis, however, the mean score of two assessments done by faculty, DSA and peers respectively was taken.
Students' feedback - On the assessment method for professionalism was taken after 3 weeks of assessment.
The scores of each assessor for each student was entered into Microsoft Excel. The total score for each 10 item in the assessor form was added. Combining all 10 items, each with a score from 0 to 5, created a variable that could be analyzed using means and nonparametric analysis. The average score by each assessor category was calculated for the whole batch. Statistical analysis was done using descriptive statistics. Student's t-test (two-tailed) was used to compare the average scores of female and male students. Pearson's coefficient was used to determine the correlation between average knowledge scores and the MSF scores.
| Results|| |
Seventy-four final year students participated in this study. The MSF was conducted over a period of 1 month. A total of 10 evaluations per student (740) were done in addition to a written test. A total of 50 female and 24 male students with a mean age of 24.7 years formed the participant group [Table 1]. Female students scored significantly more than male students in the assessment of knowledge [Table 2]. Patients scored the students the highest while the office staff scored the students the least [Table 3]. Pearson's coefficient was determined between average knowledge scores and the MSF scores [Table 4] which showed significant fair correlation. The knowledge scores were statistically significant in the correlation with the self-assessment score and faculty score [Table 4]. Student feedback revealed that majority of the students agreed that assessment of professionalism is needed and that it helps in improvement of professional behavior. It was also observed that majority agreed that they would prefer assessment from multiple assessors than one assessor and that professionalism should be taught from third year onward when they enter the clinics. Most of the students agreed that faculty are role models for professionalism [Table 5].
|Table 3: Average score of multi-source feedback as per assessor category|
Click here to view
|Table 4: Correlation between selected score parameters and average score of knowledge|
Click here to view
Some of the comments that were given by the students are as follows
- More emphasis needed toward professionalism and ethics in the clinics
- Students should understand that professionalism is more important than quota fulfillment
- Good to know about our professionalism, but help if we know earlier (beginning of year 5)
- Assessment should be made as part of rules and regulations to be followed to train all students to be professional
- Courtesy should be taught to lecturers
- Assessment should be done as soon as possible so that it can be continued throughout the course duration
- Assessment should be done throughout study
- Assessment should be done throughout the clinical years
- Professionalism should be introduced before clinical sessions to ensure better practice
- More emphasis should be given to professionalism as it plays a major part in being a good dentist.
| Discussion|| |
Assessment is embedded deeply in education. It is hard work for the teachers and gets harder when the concept of assessing professionalism is added. Assessment is aimed at helping students focus on  their learning, identify individual strengths and weaknesses, provide an opportunity for improvement, highlight deficiencies in the content or delivery of the course, and in the case of health sciences education, protect the public against incompetent graduates. Recently, due to greater demand from the authorities, there has been an increased focus on ways to implement and assess professionalism among dental undergraduate students.
It has been accepted in relation to professionalism that, implementation and assessment face major hurdles. This is because, professionalism is a broad concept with lots of varying definitions that might, and probably will, change over time and is heavily influenced by society and culture.,, In addition to that professionalism is a subject that differs from many other types of knowledge and skills as explained with the help of Miller's pyramid. When it comes to professionalism, proof of knowledge in the lower levels does not necessarily mean that this knowledge will be used in practice. One might know a lot about ethical principles but not know how to use them in practice. Hence, a number of different ways and tools of assessments are needed since the assessment of professionalism is a contentious area due to its intangible nature.,, Keeping these issues in mind, this study was designed with the main aim of using a hybrid tool for assessing professionalism.
With the increased interest in teaching and assessing professionalism, a whole new set of assessment tools have been developed. Unfortunately, there is a lack of research on how to produce evidence supporting validity for these tools, and therefore it is argued that there exists no single method for the reliable and valid evaluation of professional behavior.
In this study, therefore, we have made an attempt to combine two facets which are assessment of knowledge and demonstration of professionalism in a clinical setting. The rationale behind combining the two elements was based on the suggestion that for assessing professionalism, Miller's pyramid could be used as a basis so that each level of the pyramid is assessed appropriate to the stage of the training of the student. This is to emphasize the fact- knowledge forms the base that cannot be undermined since the student has to first know “what is professionalism,” before demonstrating professional behaviors in different contexts. Therefore, assessment of professionalism ideally should not restrict to written test but should include behavioral aspects as well. It has been suggested that knowledge domain should be assessed for the new students to find out what they know about professionalism, while final year students and interns may be assessed at the “shows” and “does” levels. In our study, we assessed the knowledge as well as demonstration of professionalism for final year students as we wanted to evaluate how our students perform with the traditional format of teaching professionalism. This would also help us to identify areas where we need to reinforce for subsequent batches.
It has been suggested previously  that assessment of professionalism should take place in actual work settings instead of simulated settings. Therefore, for the workplace based assessment (WPBA) multisource feedback was chosen since in this approach for assessment, inputs from peers and colleagues are also taken to gather information about an individual's behavior in the workplace. The assessment for MSF was done in dental outpatient clinic when the student is attending to a patient. For this project for MSF, a pre-validated MSF form was used. The content experts validated the question paper for the written assessment, with vignettes for constructed response, for the students. The corrections of the written assessment, based on the domain to be assessed, were done by 3 experts to avoid any bias.
It has been stated that reliability of individual tools of WPBA, as well as the composite program – can be fairly possible by having 6-8 assessments per year by different assessors. To achieve that, we ensured that each student underwent at least 10 assessments. While two faculties, DSA and peer assessments were done, for statistical purposes only the average score of faculty, DSA and peer ratings was taken into consideration. The hybrid assessment tool used in the project was aimed toward formative assessment and therefore did not aim very high for reliability. For the implementation of any tool it has been suggested that it is important to look at the utility index. Utility index is a notional concept, which is represented as a product of a tool's validity, reliability, feasibility, acceptability, and educational impact., The hybrid tool used in the project is aimed toward assessment and educational impact. So though the tool, especially one that is used for assessing professionalism, ranks low numerically in terms of reliability, it has a high educational impact by improving the professional behavior of the students as has been seen in the feedback given by the students and staff. Therefore, while this tool aims toward objectivity but more so on educational impact, as has been suggested before.
The knowledge (knows) regarding professionalism among the final year dental undergraduate students
Assessing this domain tells us clearly what the students have learnt from what they have been taught so far through didactic lectures. The question paper that was made to assess this domain had five vital components of professionalism as defined by the institution, i.e., honesty, integrity, confidentiality, communication skills, and respect for patients. The question paper was validated by the content experts and the students were given 1 hour to answer the question paper. Three evaluators assessed the answer given by the students and the average score was found to be 17.7 with female students scoring significantly more than the male students. This could be due to heavy skewness of the batch toward more females. It also suggests that females in the given sample have taken the knowledge regarding professionalism more seriously.
The multisource feedback (does) for the final year dental undergraduate students
In our study, we observed that all the evaluators except the office staff assessed the students high giving an average score that was 40 and above. The patients gave the students the highest rating signifying that the patients were highly satisfied with the care offered by the student in other words depicting professionalism. The rating given by the faculty was lower than the self and peer assessment which indicates that the faculty expect much more in terms of professionalism than the students perceive themselves. This finding was found to be similar as reported by Zijlstra-Shaw et al. however, they attributed higher self-scores than faculty score to the fact that the student scores only himself while the faculty has to score so many students with no extra time given. The extra workload on the staff could have made this difference. In the present study, office staff who interact with the students, mainly related to administrative purposes, gave the lowest rating indicating that they were not happy with the communication skills of the students. This area needs to be emphasized to the students that while the clinical staff as in the DSA, laboratory assistants are crucial to their work and hence part of their team, the reception and office staff also an equally important role. The MSF scores of this project showed higher rating from the patients and lower from faculty contrary to the findings by Chandler et al. where patients rated the students lower than the faculty and nurses. The contrary finding could be attributed to the difference in the interaction of students with the patients compared to the faculty and members of the health-care team in the cohort of students who participated in the two studies. In the present study the patients' expectations seemed to have been met by the students while the faculty expected much more and hence the difference in scores. In the study by Nicole Chandler et al., the patient families felt the student interaction could have been better, and hence gave lower scores to the students.
Correlation between the knowledge scores and multisource feedback scores
A statistically significant fair correlation was found between the knowledge scores and the MSF scores. This indicates that the action of the student is based on his prior knowledge acquired through the lectures. The correlation was fair. This could be due to fact that the vignettes in the knowledge questions were not completely aligned to that of MSF questionnaire. Since the MSF questionnaire was prevalidated, changes could not be made to it, whereas not all the domains to be assessed for knowledge were there in the questionnaire. Probably aligning the vignettes to the questionnaire would show a higher correlation. Further there was a correlation between knowledge and self-assessment implying that the student assessed himself to be acting professionally which was based on his knowledge. Similarly, faculty assessment correlated with the knowledge of the student implying that the assessment by faculty of professionalism is based on knowledge of student. No correlation was seen between MSF scores given by other assessors, i.e., DSA, laboratory assistant, office staff, patients and peers and knowledge component implying that while rating the students these assessors are not looking at knowledge but at the communication skills, interpersonal skills, etc. The results imply that students' performance in clinic met the criteria set by faculty and the students' knowledge helped them to act as expected in a given situation.
Other studies have not combined knowledge with MSF, so these results could not be compared.
Feedback on this tool from the students
The feedback obtained from the students on this method was positive. Most of the students wanted professionalism to be taught and assessed. Further, they suggested assessment to start from year 3 when they enter clinical years.
Limitations of this study
- Since the batch was big comprising of 74 students, we used clinical groupings for ease of getting MSF. However, the assessment of students could not be done by the same assessors
- For statistical purpose of the project, faculty scores were clubbed together, so were the DSA, laboratory assistants, and office staff. It was a challenge to ensure all 74 assessed by the same individual in the assessor category, thereby intergroup, intragroup correlation
- Since the batch was skewed with more female students, the comparison of MSF scores of male versus female students was not done after we observed skewness in the knowledge scores of female student versus male students.
| Conclusions|| |
Assessment of professionalism using a hybrid tool that incorporates knowledge, as well as MSF, was done. It was found to be a useful method as it ensures that students have a prior knowledge of what they are doing and why they are doing. It, therefore, gives a more comprehensive assessment of professionalism among the dental students.
The authors would like to acknowledge all the students, Support staff and the patients for their contribution for this study.
Financial support and sponsorship
Conflicts of interest
There are no conflicts of interest.
| References|| |
Mueller PS. Incorporating professionalism into medical education: The mayo clinic experience. Keio J Med 2009;58:133-43.
Zijlstra-Shaw S, Kropmans TJ, Tams J. Assessment of professional behaviour – A comparison of self-assessment by first year dental students and assessment by staff. Br Dent J 2005;198:165-71.
Trathen A, Gallagher JE. Dental professionalism: Definitions and debate. Br Dent J 2009;206:249-53.
Hilton SR, Slotnick HB. Proto-professionalism: How professionalisation occurs across the continuum of medical education. Med Educ 2005;39:58-65.
Zijlstra-Shaw S, Robinson PG, Roberts T. Assessing professionalism within dental education; the need for a definition. Eur J Dent Educ 2012;16:e128-36.
Kramer GA, Albino JE, Andrieu SC, Hendricson WD, Henson L, Horn BD, et al.
Dental student assessment toolbox. J Dent Educ 2009;73:12-35.
Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65:S63-7.
Downing SM. Assessment of knowledge with written test forms. In: Norman G, Van der Vleuten C, Newble D, editors. International Handbook of Research in Medical Education. Dordrecht: Kluwer Academic Publishers; 2002. p. 647-72.
Chandler N, Henderson G, Park B, Byerley J, Brown WD, Steiner MJ, et al.
Use of a 360-degree evaluation in the outpatient setting: The usefulness of nurse, faculty, patient/family, and resident self-evaluation. J Grad Med Educ 2010;2:430-4.
Joshi R, Ling FW, Jaeger J. Assessment of a 360-degree instrument to evaluate residents' competency in interpersonal and communication skills. Acad Med 2004;79:458-63.
Karthikeyan G, Manoor U, Supe SS. Translation and validation of the questionnaire on current status of physiotherapy practice in the cancer rehabilitation. J Cancer Res Ther 2015;11:29-36.
Brinkman WB, Geraghty SR, Lanphear BP, Khoury JC, Gonzalez del Rey JA, Dewitt TG, et al.
Effect of multisource feedback on resident communication skills and professionalism: A randomized controlled trial. Arch Pediatr Adolesc Med 2007;161:44-9.
Kanters LG. Assessment of professionalism within dental education A review of studies. Malmo, Sweden: Malmö University; 2012.
Modi JN, Anshu, Gupta P, Singh T. Teaching and assessing professionalism in the Indian context. Indian Pediatr 2014;51:881-8.
Richmond M, Canavan C, Holtman MC, Katsufrakis PJ. Feasibility of implementing a standardized multisource feedback program in the graduate medical education environment. J Grad Med Educ 2011;3:511-6.
Van Der Vleuten CP. The assessment of professional competence: Developments, research and practical implications. Adv Health Sci Educ Theory Pract 1996;1:41-67.
[Figure 1], [Figure 2]
[Table 1], [Table 2], [Table 3], [Table 4], [Table 5]