REVIEW OF DISABILITY STUDIES: AN INTERNATIONAL JOURNAL, Volume 15, Issue 4 Research Article An Outcome Evaluation Study of the UConn Online Graduate Courses of the Certificate of Interdisciplinary Disability Studies in Public Health Tara M. Lutz, Ph.D., M.P.H., MCHES(r) & Mary Beth Bruder, Ph.D. University of Connecticut Center for Excellence in Developmental Disabilities Education, Research, and Service, School of Medicine Abstract: Healthy People 2020 addresses the need to increase disability-specific content in public health education. In 2014 the University of Connecticut Center for Excellence in Developmental Disabilities (UConn UCEDD) developed an online, interdisciplinary, graduate certificate in disability studies. Eighteen students provided evaluation data on the effectiveness of the Certificate as described in this article. Majority of participants reported applying knowledge gained in their other courses while half recommended a disability certificate course to others. Of those participants who are currently employed, 75% reported applying knowledge gained to carry out their current work position. Some students responded that at least a course like these with disability-specific content should be mandated for any health-related graduate program. Future implications for these findings include expanding disability-specific content across public health program curricula to increase the capacity of the future workforce to serve people with disabilities. Keywords: Disability; Public Health; Education Section 4302 of the Patient Protection and Affordable Care Act (ACA) established data collection standards for measuring disability status consistent with the International Classification of Functioning, Disability, and Health (ICF) (USDHHS, 2011). This model assumes humans function on three levels: 1) as a body or a body part, 2) as a whole person, and 3) as a whole person in a social context, and that disability is dysfunction at one or more of these levels: 1) as impairments, 2) as activity limitations, or 3) as participation restrictions (WHO, 2002, p. 9). The latest estimates from the Centers for Disease Control and Prevention's (CDC) Behavior Risk Factor Surveillance System (BRFSS) indicate that one in four adults, or 61 million people, are living with a disability in the United States (Okoro, Hollis, Cyrus, & Griffin-Blake, 2018). The BRFSS reports disability based on the following disability categories: hearing, vision, cognition, mobility, self-care, and independent living which are consistent with Section 4302 of the ACA. According to the BRFSS, the most common type of disability across all adult age groups was mobility (13.7%) followed by cognition (10.8%) (Okoro et al., 2018). This large number of people living with disability is significant because compared to people without disabilities, people with disabilities (PWD) experience health disparities across the social determinants of health: education, economic stability, social and community context, neighborhood and built environment, and health care --PAGE 1-- access (Yee et al., 2016; ODPHP, 2018c; Okoro et al., 2018). For example, PWD in general report higher rates of obesity, physical inactivity, and smoking and are less likely to receive preventive health care compared to people without disabilities (Courtney-Long, Romano, Carroll, & Fox, 2016; Krahn, Walker, & Correa-De-Araujo, 2015). Further, research documents that among those who are identified as having a disability, disparities in prevalence exist among disability categories, sex, race/ethnicity, socioeconomic status, among other descriptors (Courtney-Long et al., 2015; Okoro et al., 2018). From both a medical and public health perspective, disability historically has been regarded as an undesired health outcome to be prevented (e.g., Heaphy, Mitra, & Bouldin, 2011; Higgenson & Widerburg, 2009; Krahn & Campbell, 2011; Linker, 2013; McDonald & Raymaker, 2013). As stated in The Surgeon General's Call to Action to Improve the Health and Wellness of Persons with Disabilities, "The health and wellness of persons with disabilities today is a matter of public health concern" (USDHHS, 2005, p. 21). However, a study conducted by the CDC and Massachusetts Department of Public Health revealed that only 50% of accredited public health programs and schools offer disability content within their curricula (Sinclair, Tannenhaus, Courtney-Long, & Eaton, 2015). Healthy People 2020 objectives for disability and health include increasing the proportion of U.S. Master of Public Health (M.P.H.)- granting public health programs and schools that offer graduate-level studies in the topic area of disability and public health (ODPHP, 2018b). Similarly, the 10 Essential Public Health Services highlight the need for public health activities to "assure a competent public and personal health care workforce" (CDC, 2018, paragraph 3), which needs to incorporate skills to include PWD in public health activities. The UConn Certificate of Interdisciplinary Disability Studies in Public Health (disability certificate) was created in 2014 to address this gap in disability-specific content in graduate public health courses. The disability certificate is offered in partnership between the University of Connecticut Center for Excellence in Developmental Disabilities (UConn UCEDD) and UConn School of Medicine Department of Public Health Sciences Programs in Applied Public Health Sciences. It uses the 10 Essential Public Health Services as a framework for students to study the health and well-being of people with disabilities and their families, as well as the communities in which they live, learn, work, and play, and the populations of which they are a part. Key public health concepts addressed in the disability certificate courses are the social determinants of health, health disparities, disability as diversity, disability-related policy and legislation, disability surveillance, research ethics, self- determination, cultural competency, and evidence-based practice, among others. The disability certificate promotes interdisciplinary discourse and systems-level thinking. It is comprised of four graduate level courses offered entirely online to interested applicants who have earned at least a bachelor's degree. Because of the online format, students may participate from distances, an opportunity that traditional in-person lecture-style classes cannot offer. This also allows students from outside of UConn to pursue the courses and the disability certificate. Students who matriculate into the program and complete the four courses earn a graduate disability certificate from UConn. --PAGE 2-- The disability certificate incorporates resources and information from expert public health agencies that include the World Health Organization (WHO), the CDC, the Department of Health and Human Services (HHS), the National Association of County and City Health Associations (NACCHO), the National Center on Health, Physical Activity and Disability (NCHPAD), Association of Disability Centers on Disability (AUCD) which includes the National Center on Disability in Public Health and Including People with Disabilities: Public Health Workforce Competencies, and Special Olympics, among others. The Sinclair et al. (2015) study identified this disability certificate as a resource for public health programs to develop and implement disability-specific coursework into their public health programs. Course titles and brief descriptions are provided below: 1.Foundations of Public Health and Disability: This course is an introductory survey of the ways in which disability, both developmental and acquired, is affected by, and interacts with, public health policy and practice. Upon completion of this course students will have a foundational understanding of a comprehensive set of issues of both acquired and developmental disability as related to the core elements of public health. 2.Epidemiology of Disability: This course introduces epidemiologic research design and delves deeply into epidemiology as it applies to monitoring the health status of people with disabilities, diagnosing and investigating health problems, evaluating personal and population-based interventions, and conducting research. It also critically examines sources of public health and epidemiologic data that exist. 3.Disability Law, Policy, Ethics, & Advocacy: This course provides an introduction to policy and law affecting people with disabilities and public health approaches to meeting their individual needs as well as the needs of broader populations. Students will learn the essential tools for enforcing laws and regulations that protect health and ensure safety and for developing new policies and plans that support individual and community health efforts inclusive of people with disabilities. 4.Public Health Interventions in Disability: This course critically examines public health systems and programs across the lifespan available to people with disabilities that impact health. It extends on foundational principles that are evidence-based and driven by epidemiologic studies of disability within the context of existing laws and policies. (UConn, 2019). Learning objectives for each course align with foundational competencies determined by the Council on Education for Public Health (CEPH) public health program accreditation requirements as UConn's M.P.H. program is accredited by CEPH (CEPH, 2016). Each of the four courses are designed to follow the same format so that students can become familiar with the layout, rubrics, and expectations across courses. Courses are broken down into modules which are several weeks that cover a common theme. For example, a module within the foundations course is titled "disability as a unique identity and as an individual experience." At the end of each module students complete an assignment to assess their understanding of --PAGE 3-- that module's content. These assignments include short papers, presentations, and visuals. Students also complete a reflection at the end of every module to allow them to both synthesize the content and apply it to their personal and professional lives. Disability Studies in Public Health Disability studies, according to the Society for Disability Studies (SDS) "...sits at the intersection of many overlapping disciplines in the humanities, sciences and social sciences" (SDS, 2016). Disability studies "challenges the view of disability as an individual deficit and explores models and theories that examine social, political, cultural, and economic factors that define disability" (SDS, 2016). It also studies national and international perspectives, policies, culture, and history. Public health as a discipline is uniquely positioned to emphasize the importance of included PWD in all activities. As stated in the Institute of Medicine's Future of Public Health (1988), the mission of public health is "...the fulfillment of society's interest in assuring the conditions in which people can be healthy" (p. 40). This disability certificate uses a public health lens as a framework and incorporates disability studies to examine how culture, policy, history have shaped the treatment of PWD and their access to services including healthcare. There is a large emphasis on language and terminology used as related to PWD, as often medical terminology is the dominant syntax used by healthcare and other providers. Person-first language, identity-first language, intersectionality, and neurodiversity are all topics covered within the certificate to expand the understanding of disability beyond the medical model (e.g., Dunn & Andrews, 2015; Kapp et al., 2013, Moodley & Graham, 2015). PWD While public health is often closely affiliated with medicine and the medical model of disability, this disability certificate unpacks other models of disability early in the foundations course and carries the theme of 'defining' disability throughout the four courses (e.g., Wasserman, Asch, Bluestein, & Putnam, 2016). Healthy People 2010 described disability as "a demographic descriptor rather than a health outcome. It should be used to monitor disparities in health outcomes and social participation" (Andresen, 2011). These courses expand on this important statement. The courses also critically examine the advantages and disadvantages to using public health as a framework to study disability as public health is focused at the population rather than the individual within the U.S. and at an international level, as family and cultural understandings, perspectives, and values related to PWD differ (e.g., Cohen & Miguel, 2018; Ennis-Cole, Durodoye, & Harris, 2013; Ormel et al., 2008; Norwich, 2008). Spotlight on Foundations of Public Health and Disability The foundations course is the first course in the disability certificate sequence and is also recommended to students who cannot pursue the full certificate but are interested in disability and public health. The course consists of five modules: 1) introduction, 2) disability as a unique identity and as an individual experience, 3) disparity issues in the study of disability, 4) foundations of measurement issues for studying disability in public health, and 5) current and future issues in disability in public health. As is required for CEPH accreditation, the following is a sampling of a table that describes the learning objectives, --PAGE 4-- CEPH foundational competency addressed, and the evaluation/assessment for foundations (CEPH, 2016, p. 18) (Appendix A). Appendix B provides a sampling of the modules, topics, and readings from the foundations course. These readings come from an array of disciplines and sources related to disability studies including philosophy, sign language studies and D/deaf culture, gender studies, psychology, law, public health, special education, and others. While the courses use a public health framework and include content from public health or related sources, these courses were intentionally developed and are regularly updated to reflect the fact that disability spans across disciplines. The courses also incorporate the use of media such as YouTube, blogs, and credible websites to teach content. Each week's required reading for each course includes links to various media related to the week's topic. For example, during service systems: health within the foundations course, students watch the University of South Florida's Florida Center for Inclusive Communities (UCEDD), XCEL training video on how to provide excellent care for patients with disabilities (Perkins & VanZant, 2017). Since its inception, over 175 students from 14 distinct academic disciplines have completed at least one of the courses in the disability certificate. To date, ten students have earned the disability certificate and another five are expected to complete it in the next calendar year. These students represent the following disciplines: public health, social work, medicine, psychology, speech, language, and hearing sciences, dentistry, and nursing. Some of the disciplines and degree programs represented by students who take these courses, in addition to public health, include audiology, nursing, school psychology, dentistry, medicine, special education, and biomedical engineering among others. The foundations course, the first in the disability certificate, is also part of the required curricula completed by Connecticut CT Leadership Education in Neurodevelopmental and related Disabilities (CT LEND) trainees, an intensive, interdisciplinary training program for graduate students in various disciplines to prepare them to be leaders to improve the health of PWD and their families (AUCD, 2019). The overall aim of this study was to evaluate student and employment outcomes as a result of taking at least one disability certificate course. Methods Participants The 175 students who completed at least one of the four disability certificate courses between Spring 2014 and Summer 2018 were contacted via email invitation to participate. This study was approved by the UConn Health Institutional Review Board (IRB). Procedure Former students were contacted via the email address provided at the time when they were taking a course, but not all students provided an email address so could not be invited to participate. Invitations were sent to 78 eligible participants in early Fall semester 2018. Of --PAGE 5-- these 78, six invitations were returned as "undeliverable", indicating these email addresses were no longer active. A follow-up invitation was sent to the 72 active email addresses two weeks after the first invitation was sent. The final sample (n=18) had a response rate of 25%. The initial emails sent to former students contained a brief description of the nature of the evaluation study and details about participants, including an emphasis that participation was voluntary. The emails also contained a link to the survey for individuals interested in participating. Follow-up emails contained the same information as the initial email, but also included a statement thanking individuals who had already participated. Evaluation Survey Interested participants clicked on the link in the invitation email and were brought to the first page of the Qualtrics survey, the information sheet. This information sheet served as the consent to participate, as the IRB determined this study to be exempt and therefore did not require a more formal consent process. The information sheet provided information regarding the purpose of the study, study procedures, potential risks and benefits, information protection, participants' rights, and contact information regarding study content and rights. Participants were given the option to "accept" or "decline" participation. The brief, 5-10 minute evaluation study itself consisted of a total of 17 open and closed-ended questions (Appendix C). It was pre-tested using several graduate students who did not take any disability certificate courses to check for clarity of language and accuracy of the skip-logic used based on participant responses. Content of questions related to knowledge and employment outcomes were informed by UConn UCEDD evaluation measures required for federal reporting. The evaluation study collected the following participant demographic information including age, gender with which they identify, their status as a current or former student, and their major/discipline while they were taking a disability certificate course. Participants then selected which of the four disability certificate courses they completed and the initial reason they enrolled in their first course. Participants were asked if they completed more than one course, if so, why they did, and if they completed the entire disability certificate. They then had the opportunity to select from a list of possible student-related outcomes as a result of taking one course including completing additional disability certificate course(s), applying to an additional degree program, applying to the disability certificate, applying the course content in other courses, and recommending the course to others. For participants who did not pursue the disability certificate, they were asked to select the single best reason why they did not from the following options: 1) did not know about the certificate, 2) did not fit into my plan of study for my degree program, 3) was not interested in the certificate, 4) was not eligible for the certificate, 5) financial barriers, or 6) had the option to write-in another reason. Participants then were asked to select from a list of options the best description of their current work setting and type of work. They were asked if their current work relates to PWD. Finally, 5-point Likert scales from "strongly disagree" to "strongly agree" captured data on --PAGE 6-- employment-related outcomes on the following four areas: 1) increased knowledge in the area of disability and public health, 2) application to a position related to working with and for PWD, 3) application of knowledge gained to secure current position, and 4) application of knowledge gained to carry out current position. Participants had the opportunity to provide written responses to answer the following questions: 1) What is the single most important lesson you learned from your course(s)?, 2) What recommendation(s), if any, do you have to improve the course(s)?, and 3) any additional comments. See Appendix C for the instrument. Data Analysis All data collected through Qualtrics were downloaded into a Microsoft Excel file, cleaned, and analyzed. Data cleaning included the removal of cases that met the following criteria: 1) declined participation, or 2) selected "N/A; I did not take any of these courses" as the response to item 4, "Which of the Certificate of Interdisciplinary Disability Studies in Public Health (disability certificate) courses have you completed (select all that apply)." Descriptive statistics were calculated for all included data. In addition to data collected in the evaluation study, data from student reflections completed by each student in every course at the end of every module were included in the results to further describe the lessons learned from the disability certificate courses and to further qualitatively illustrate student outcomes as a result of taking at least one course. Results The total number of participants who opened the link in the recruitment invitation email and at minimum accepted participation was 19, however one did not meet the eligibility criterion of having taken at least one of the disability certificate courses. The final sample (n = 18) had a response rate of 25%. Almost 78% of this sample identified as female, 16.67% did not report a gender, and 5.56% identified as male. The average age of participants was 30.4 years (SD = 8.49). Student-related Results Majority of participants were current, Fall semester 2018 students (61.11%, n = 11). Eight unique disciplines were represented in the final sample including social work, public health, speech language, and hearing sciences, public policy, education, nursing, law, and medicine. The most common major/discipline was public health (44.44%) followed by "multiple" which included a combination of public health, medicine, law, and social work (16.67%). Almost 89% of the sample took 5501: Foundations of Public Health and Disability. The course with the fewest number of participants was 5504: Public Health Interventions in Disability. Over 55% of participants completed more than one course; of those, 30% completed all four and received the disability certificate. More student details can be found in Table 1. --PAGE 7-- Table 1. Student Information Item Percent Student status Currently a student 61.11% Not currently a student; completed degree program 33.33% Not currently a student; have not completed my degree/program 5.56% Major/discipline Public health 44.44% Speech, language, and hearing sciences 5.56% Public policy 5.56% Social work 11.11% Nursing 5.56% Education 11.11% Multiple 16.67% Courses 5501 88.89% 5502 --PAGE 8-- 38.89% 5503 38.89% 5504 33.33% Initial reason for enrolling in the first course. Half of participants initially enrolled in their first course because of the topic area of that specific course, while a third initially enrolled to fulfill requirements related to their graduate degree programs. Others indicated that they initially enrolled in their first course because of the appeal of the online course format. Most participants who completed more than one course indicated they did so because they wanted to learn more or because the course was in a new topic area of interest (66.67%). Completing the disability certificate. Of those who took more than one course but did not complete the disability certificate, the most commonly reported reason for not pursuing the disability certificate was because additional courses did not fit into their plan of study for their graduate degree (33.33%). About 17% of participants reported not taking any additional courses because they were not interested in pursuing the disability certificate. Student-related outcomes Almost 78% of all participants strongly agreed that their knowledge in the area of disability and public health increased as a result of completing a course, regardless of which course they completed. Figure 1 describes immediate outcomes for students as a result of taking one disability certificate course. Figure 1. Student outcomes as a result of taking one course --PAGE 9-- Image Description: Figure 1 is a vertical bar graph. The x-axis includes five bars: 1) "Applied the course content in other courses", indicating 12 students; 2) "Recommended the course to others, indicating", 8 students; 3) "Completed additional courses", indicating 5 students; 4) "Applied to an additional degree program", indicating 3 students; and 5) "Applied to the Disability Certificate", indicating 5 students. The y-axis include increments of two, with numbers starting from 0-14. Employment-related outcomes Two-thirds of participants are currently employed at work settings including clinical, research, education, and administrative settings. Of those currently employed, over 76% of participants indicated that their current place of work relates to PWD. About 28% reported clinical work or direct service provision as their type of work. A third of participants who are currently employed indicated that they agree that they applied to a work position related to working with and for people with disabilities as a result of taking one disability certificate course. A third also indicated that they applied the knowledge gained from one course to secure their current work position. Finally, 75% of participants indicated that they apply the knowledge gained in at least one certificate course to carry out their current work position. Table 2 contains more employment-related information. Table 2. Employment Information Item Percent Work setting N/A; I'm a student 33.33% Clinical, in-patient 5.56% Clinical out-patient 16.67%% Government agency 5.56% Research facility 5.56% Higher education/academia 22.22% Non-profit 5.56% --PAGE 10-- Other 5.56% Type of work N/A; I'm a student 27.78% Clinical 16.67% Other direct service provision 11.11% Administration 5.56% Education 11.11% Research 22.22% Other 5.56% Work relate to people with disabilities N/A; I'm a student 22.22% N/A; I'm currently not employed 5.56% Yes 55.56% No 16.67% Important Lessons and Reflections from Students When asked, "What is the single most important lesson you learned from your course(s)?": participants had the opportunity to type in responses. Some examples include: "Disability is a complex topic which needs to be involved as an active conversation across academic disciplines." "The course helped me become aware of my own biases and stereotypes that I held --PAGE 11-- about individuals with disabilities. My increased awareness allows me to better advocate for my clients and be a more compassionate and empathetic social worker." "The experiences of people with disability differ from person-to-person and the importance of person-centered care." Data collection from student reflections completed at the end of each module within each course also provide important evaluation information. These provide valuable insight and feedback about the courses and their content. Several themes have emerged from these reflections. Some students have inquired as to whether courses like these, especially the foundations course, are mandated for any health-related graduate programs, as the students themselves see the importance of the material. Some have indicated that completing at least one course has given them confidence to share what they have learned with colleagues and even supervisors, creating leaders in their respective fields with disability-specific knowledge. These courses themselves are interdisciplinary as they pull content from across disciplines. Then students come from different programs with different life experiences, creating an interdisciplinary 'classroom'. Students regularly reflect on how enjoyable and beneficial it is to work with those from other backgrounds who have unique insights and perspectives. Dental students in particular have reflected on their translation of knowledge into practice in their clinical placement sites to be more culturally competent and inclusive of PWD. Recommendations from Students Participants were also asked to provide any recommendations they have to improve the courses and overall disability certificate. The most common theme from these recommendations was related to the online course format. Some examples include: "Offer this in person! I realize it's intended for certificate-seekers, but this would be so much more engaging and impactful in a classroom setting," "While I know the convenience of an online format is a major draw for many students, I would have been interested in participating in an in-person disability class or having either optional or mandatory in-person class meetings. The organization of the course certainly maximized class discussions, but I think in person discussions would have made the course experience even more engaging and lively." Other participants provided constructive feedback related to reducing the amount of reading and writing assignments and their difficulty managing online courses: "Foundations of Public Health and Disability was extremely difficult to manage along with a regular course load. Too many small assignments." "I found the online format much more and more challenging and hard to keep up with by the time I got to a 3rd class. They were classes I took outside of my degree program for my own learning. I would have preferred to have several in-person classes rather than write so much without speaking to people face to face." --PAGE 12-- Additional comments, as stated by one respondent included the following quote regarding topical interest in disability and public health: My interest area is disability and public health and I have had difficulty getting training in this area. I was so pleased to learn about and take advantage of these courses. They provide much needed training that can be accessed from anywhere in the country. Discussion After five years, conducting a brief evaluation study of former students who took disability certificate courses allowed us to assess if and how students were using the information both as students and then later on as professionals in their respective fields and areas of employment. Results indicate that disability certificate courses are meeting the interests of students, not just in public health, but across academic disciplines. Majority of participants have applied course content in other courses and programs, thus exposing their peers to thoughts, ideas, and concepts which they otherwise would not learn in their specific programs as a result of the interdisciplinary nature of the disability certificate. Students are also pursuing employment related to working with and for PWD and incorporating their knowledge gained into these positions based on completing at least one course. As indicated by the recommendations from participants, online courses have their drawbacks and are not the same as traditional in-person courses. More responsibility and time management on behalf of the student are required to manage deadlines and assignments within an online course environment. Further, some students enroll in one of these disability certificate courses not out of topical interest, but out of necessity to earn the minimum- required credits for each semester. These students may have no interest in pursuing the entire disability certificate, but at least were exposed in some capacity for some reason to disability and health content. One limitation of this evaluation is the response rate. First, contact information was not available for all former students. Then, some contact information was no longer active. Some students have graduated and no longer have access to their student email address; others may have provided an email address which they do not check regularly, also shrinking the possible final sample pool. Participants represented 25% of the total possible sample. These participants self-selected to participate in this evaluation. Because they self-selected to participate it is likely that they had positive experiences in the certificate courses, thus potentially skewing responses and results toward positive outcomes. The steps described in the methods were taken to reduce the possibility of non-response bias, but those who chose not to or did not participate may be systematically different from those who did respond (Jacobsen, 2017, p. 111). To get a more representative outcome of the certificate, other methods of increasing participation such as collecting multiple modes of contact information of students and offering --PAGE 13-- incentives to participants may increase participation in future evaluations. These strategies and several other retention strategies identified in a systematic review by Robinson and colleagues (2015) include obtaining multiple contacts for each participant, including two who do not reside with the participant. These data could be collected from students as they enroll in each course. Additionally, if this study were to be replicated in the future, students could be informed that they may be contacted in the future separate from academic participation (Robinson et al., 2015). There is literature that indicates that for academic research, invitations to participate in research that are personalized can increase the response rate (Heerwegh, Vanhove, Matthjis, & Loosveldt, 2005; Joinson, Woodley, & Reips, 2007; Saleh & Bista, 2017). Saleh and Bista provide 11 recommendations for online studies and response rates (2017). This evaluation utilized most, but not all 11 recommendations as most are required by any institution's IRB. Among the recommendations that could be adopted in the future include personalizing invitations to participate (recommendation 9) and being aware of the time constraints related to the time of year (recommendation 11) (Saleh & Bista, 2017). If conducted in the future, technology to personalize email invitations can be utilized. Conclusion and Future Implications PWD are often not included in discussions regarding groups who experience health disparities (Krahn et al., 2015), yet account for a significant part of the population (Okoro et al., 2018). In order for current and future public health practitioners to meet the needs of the population, including PWD, schools and programs of public health must build capacity in the future workforce to include PWD (Frankena et al., 2018; Griffen, Risley, Petros, & Welter, 2018). As more academic pre-professional programs move toward interdisciplinary scholarship, this disability certificate presents a unique opportunity for students to develop knowledge and skills essential to their future professions to work collaboratively with others to serve communities and to include PWD. Increased opportunities such as those presented by this online disability certificate will continue to build a competent public health workforce with a skill set related to serving all populations including PWD. Providing these courses online allows for distance learning, collaboration, and sharing of resources across disciplines, programs, and universities. However, even these disability certificate courses are electives; students are not required to take any of them in order to earn their M.P.H. or other degrees. Further, participants indicated that they did not have the space or time during their academic degree program to fit the entire disability certificate. These results are indicative of a larger-scale barrier to ensuring a competent workforce as disability-specific content is not yet required by accrediting bodies that oversee degree- granting institutions. In some ways, students self-select into these courses as electives to learn the content; not every student who earns an M.P.H. can demonstrate that they have the knowledge and skills to apply public health concepts to include PWD and therefore cannot contribute to solving health disparities experienced by PWD. This disability certificate is a small example of a much larger network of resources working to ensure a competent public health workforce competent in including PWD into all public health activities. --PAGE 14-- More information about the disability certificate can be found at this website: https://ph.disability.certificate.uconn.edu/. Tara M. Lutz, Ph.D., M.P.H., MCHES®, is a faculty member of Public Health Sciences at the University of Connecticut School of Medicine where she is the training director of the UConn Center for Excellence in Developmental Disabilities Education, Research, and Service. She completed her M.P.H. and Ph.D. in public health at UConn. She is also a Master Certified Health Education Specialist (MCHES). Her research area of interest is health promotion, specifically comprehensive sex education, for individuals with developmental disabilities. Mary Beth Bruder, Ph.D., is a professor of Public Health Sciences, Pediatrics, and Educational Psychology at the University of Connecticut (U.S.A.) where she directs the Center for Excellence in Developmental Disabilities Education, Research, and Service. She has been in the field of disabilities for 43 years, and has directed over 80 federal and state research, demonstration and training grants and contracts. Her primary research interest focuses on the preparation and support of qualified and effective personnel, at both the individual and systems level. In particular, she is involved in the development of strategies to remediate health care disparities for those with disabilities, the development of inclusive early childhood systems, and the implementation of strength-based family engagement practices. Dr. Bruder is the editor of Infants & Young Children, serves on the editorial boards of other peer reviewed journals in disability, and is a board member of the International Society on Early Intervention. References Andresen, E. M. (2011). Epidemiology and biostatistics. In Lollar, D. J. & Andresen, E. M. (Eds.), Public health perspectives on disability: Epidemiology to ethics and beyond (pp. 17-54). New York, NY: Springer Science+Business Media, LLC. Association of University Centers on Disability (AUCD). (2019). About LEND. Retrieved from https://www.aucd.org/template/page.cfm?id=473 Bezyak, J., Sabella, S., & Gattis, R. (2017). Public transportation: an investigation of barriers for people with disabilities. Journal of Disability Policy Studies, 28(1), 52-60. https://doi.org/10.1177/1044207317702070 Centers for Disease Control and Prevention. (CDC). (2018). Public health professionals gateway: The public health system & the 10 essential public health services. Retrieved from https://www.cdc.gov/publichealthgateway/publichealthservices/essentialhealthservices .html Cohen, S. & Miguel, R. (2018). Amor and social stigma: ASD beliefs among immigrant Mexican parents. Journal of Autism and Developmental Disorders, 48(6), 1995-2009. doi:10.1007/s10803-017-3457-x --PAGE 15-- Council on Education for Public Health. (CEPH). (2016). Accreditation criteria: Schools of public health & public health programs. Retrieved from https://media.ceph.org/wp_assets/2016.Criteria.pdf Courtney-Long, E., Carroll, D., Zhang, Q., Stevens, A., Griffin-Blake, S., Armour, B., & Campbell, V. (2015). Prevalence of disability and disability type among adults United States, 2013. Morbidity Mortality Weekly Report, 64(29), 777-783. Retrieved from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4584831/ Courtney-Long, E., Romano, S., Carroll, D., & Fox, M. (2016). Socioeconomic factors at the intersection of race and ethnicity influencing health risks for people with disabilities. Journal of Racial and Ethnic Health Disparities, 4(2), 213-222. doi: 10.1007/s40615- 016-0220-5 Dunn, D. S., & Andrews, E. E. (2015). Person-first and identity-first language: Developing psychologists' cultural competence using disability language. The American Psychologist, 70(3), 255-264. doi: 10.1037/a0038636 Ennis-Cole, D., Durodoye, B., & Harris, H. (2013). The Impact of Culture on Autism Diagnosis and Treatment. The Family Journal, 21(3), 279-287. doi: 10.1177/1066480713476834 Frankena, T. K., Naaldenberg, J., Cardol, M., Garcia Iriatre, E., Buchner, T., Brooker, T., ... Leusink, G. (2018). A consensus statement on how to conduct inclusive health research. Journal of Intellectual Disability Research. Doi: 10.1111/jir.12486 Griffen, A. K., Risley, K., Petros, M., & Welter, C. (2018). Inclusion wheel: tool for capacity and public health leaders to serve people with disabilities. Health Promotion Practice, epub. Doi: 10.1177/1524839918788578 Heaphy, D.G., Mitra, M., & Bouldin, E. D. (2011). Disability and health inequity. In Lollar, D. J. & Andresen, E. M. (Eds.), Public health perspectives on disability: Epidemiology to ethics and beyond (pp. 117-150). New York: Springer Science+Business Media, LLC. Heerwegh, D., Vanhove, T., Matthijs, K., & Looseveldt, G. (2005). The effect of personalization on response rates and data quality in web surveys. International Journal of Social Research Methodology, 8(2), 85-99. https://doi.org/10.1080/1364557042000203107 Higginson, G. & Widerburg, B. (2009). A brief history of public health. In C. E. Drum, G. L. Krahn & H. Bersani, Jr. (Eds.), Disability and Public Health (pp. 9-64).Washington, D.C.: American Public Health Association. --PAGE 16-- Institute of Medicine. (1988). The Future of Public Health. Washington, D.C.: National Academy Press. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK218218/pdf/Bookshelf_NBK218218.pdf Jacobsen, K. H. (2017). Introduction to health research methods: A practical guide (2nd ed.). Burlington, MA: Jones & Bartlett Learning, LLC. Joinson, A. N., Woodley, A., & Reips, U. D. (2007). Personalization, authentication, and self- disclosure in self-administered Internet surveys. Computers in Human Behavior, 23(1), 275-285. https://doi.org/10.1016/j.chb.2004.10.012 Kapp, S., Gillespie-Lynch, K., Sherman, L., Hutman, T., Eccles, Jacquelynne, Akhtar, Nameera, & Jaswal, Vikram K. (2013). Deficit, Difference, or Both? Autism and Neurodiversity. Developmental Psychology, 49(1), 59-71. Doi: 10.1037/a0028353 Krahn, G. & Campbell, V. A. (2011). Evolving views of disability and public health: the roles of advocacy and public health. Disability and Health Journal, 4(2011), 12-18. doi: 10.1016/j.dhjo.2010.05.005 Krahn, G. L., Walker, D. K., & Correa-De-Araujo, R. (2015). Persons with disabilities as an unrecognized health disparity population. American Journal of Public Health, 105, Suppl2, S198-S206. doi: 10.2105/AJPH.2014.302182 Larson, D. (2008). Unconsciously regarded as disabled: Implicit bias and the regarded-as prong of the Americans with Disabilities Act. UCLA Law Review, 56(2), 451-488. Lee, C. (2012). Deafness and cochlear implants: A deaf scholar's perspective. Journal of Child Neurology, 27(6), 821-823. doi: 10.1177/0883073812441248 Linker, B. (2013). On the borderland of medical and disability history: A survey of the fields. Bulletin of the History of Medicine, 87(4), 499-535. Doi: 10.1353/bhm.2013.0074 McDonald, K. E., & Raymaker, D. (2013). Paradigm shifts in disability and health: Toward more ethical public health research. American Journal of Public Health, 103(12), 2165-2173. doi: 10.2105/AJPH.2013.301286 Moodley, J., & Graham, L. (2015). The importance of intersectionality in disability and gender studies. Agenda: Empowering Women for Gender Equity, 24-33. https://doi.org/10.1080/10130950.2015.1041802 National Council on Disability. (2016). The impact of the Affordable Care Act on people with disabilities: A 2015 status report. Retrieved from https://www.ncd.gov/publications/2016/impact-affordable-care-act-people-disabilities- 2015-status-report Norwich, B. (2008). Dilemmas of difference, inclusion and disability: International perspectives on placement. European Journal of Special Needs Education, 23(4), 287- 304. https://doi.org/10.1080/08856250802387166 --PAGE 17-- Office of Disease Prevention and Health Promotion (ODPHP). (2018a). Determinants of health. Retrieved from https://www.healthypeople.gov/2020/about/foundation-health- measures/Determinants-of-Health Office of Disease Prevention and Health Promotion (ODPHP). (2018b). Disability and health. Retrieved from https://www.healthypeople.gov/2020/topics-objectives/topic/disability- and-health/objective Office of Disease Prevention and Health Promotion. (ODPHP). (2018c). Social determinants of health. Retrieved from https://www.healthypeople.gov/2020/topics- objectives/topic/social-determinants-of-health Okoro, C. A., Hollis, N. D., Cyrus, A. C., & Griffin-Blake, S. (2018). Prevalence of disabilities and health care access by disability status and type among adults-United States, 2016. Morbidity Mortality Weekly Report, 67, 882-887. http://dx.doi.org/10.15585/mmwr.mm6732a3 Ormel, J., Petukhova, M., Chatterji, S., Aguilar-Gaxiola, S., Alonso, J., Angermeyer, M.C., .... Kessler, R. C. (2008). Disability and treatment of specific mental and physical disorders across the world. British Journal of Psychiatry, 368-75. doi: 10.1192/bjp.bp.107.039107. Perkins, E. A. & VanZant, S. (2017). XCEL - Providing excellent care for patients with developmental disabilities. Florida Center for Inclusive Communities, University of South Florida. Retrieved from http://flfcic.fmhi.usf.edu/program-areas/health.html Robinson, K. A., Dinglas, V. D., Sukrithan, V., Yalamanchilli, Mendez-Tellez, P. A., Dennison-Himmelfarb, C., & Needham, D. M. (2015). Updated systematic review identifies substantial number of retention strategies: using more strategies retains more study participants. Journal of Clinical Epidemiology, 68(12), 1481-1487. http://dx.doi.org/10.1016/j.jclinepi.2015.04.013 Saleh, A. & Bista, K. (2017). Examining factors impacting online survey response rates in educational research: perceptions of graduate students. Journal of MultiDisciplinary Evaluation, 13(29), 63-74. http://journals.sfu.ca/jmde/index.php/jmde_1/article/view/487 Sinclair, L. B., Tannenhaus, R. H., Courtney-Long, E., & Eaton, D. K. (2015). Disability within US public health school and program curricula. Journal of Public Health Management and Practice, 21(4), 400-5. Doi: 10.1097/PHH. 0000000000000114 Society for Disability Studies. (SDS). (2016). What is disability studies? Retrieved from http://disstudies.org/index.php/about-sds/what-is-disability-studies/ --PAGE 18-- UConn. (2019). Interdisciplinary Disability Studies in Public Health Online Graduate Certificate. Retrieved from https://ph.disability.certificate.uconn.edu/ U.S. Department of Health and Human Services. (USDHHS). (2005). The Surgeon General's Call to Action to Improve the Health and Wellness of Persons with Disabilities. U.S. Department of Health and Human Services, Office of the Surgeon General, 2005. Retrieved from https://www.ncbi.nlm.nih.gov/books/NBK44667/pdf/Bookshelf_NBK44667.pdf U.S. Department of Health and Human Services (USDHHS). (2011). Implementation guidance on data collection standards for race, ethnicity, sex, primary language, and disability status. Retrieved from https://aspe.hhs.gov/basic-report/hhs-implementation- guidance-data-collection-standards-race-ethnicity-sex-primary-language-and- disability-status Wasserman, D., Asch, A., Blustein, J. & Putnam, D. (2016). Disability: definitions, models, experience. The Stanford Encyclopedia of Philosophy. Retrieved from https://plato.stanford.edu/archives/sum2016/entries/disability/ World Health Organization. (2002). Towards a common language for functioning, disability, and health: ICF. Geneva: WHO. Retrieved from https://www.who.int/classifications/icf/icfbeginnersguide.pdf?ua=1 Yee, S., Breslin, M. L., Goode, T. D., Havercamp, S. M., Horner-Johnson, W., Iezzoni, L. I., & Krahn, G. (2016). Compounded disparities: health equity at the intersection of disability, race, and ethnicity, presented at roundtable workshop The Intersections Among Health Disparities, Health Equity, and Health Literacy, Washington, D.C. (June 14, 2016). Retrieved from http://nationalacademies.org/hmd/Activities/SelectPops/HealthDisparities/Commissio ned-Papers/Compounded-Disparities.aspx --PAGE 19-- Appendix A Foundations of Public Health and Disability Module Topic Learning objective CEPH foundational competency Evaluation/assessment 1. Introduction Disability & public health 1.5 Describe how the 10 Essential Services apply to disability Apply systems thinking tools to a public health issue (CEPH 22) Discussion board 4: Pick a quote from the Call to Action no longer than a paragraph in length. Explain how this quote exemplifies the shift in public health efforts to include people with disabilities. How does this quote relate to Healthy People 2020 objectives? How does it encompass the history of public health and disability? Module 1 interim assignment: Now with a background in public health, disability history, and the DD Act, create a 10 Essential Services Wheel that contains specific examples of how each service specifically applies to disability. In addition to creating a visual, provide a paragraph for each service explaining the specific contents and their application to disability to accompany the visual. Examples/applications should be as specific as possible and cited. --PAGE 20-- Appendix B Key Reading Sample from Foundations of Public Health and Disability Module Topic Key Reading Sample Citation 1. Introduction Major determinants of health (ODPHP, 2018a) Public health (CDC, 2018) Disability (Higginson & Widerburg, 2009) Disability & public health (USDHHS, 2005) 2. Disability as a unique identity and as an individual experience Models of disability (Wasserman et al., 2016) Living with disability; self- determination (Lee, 2012) Disability as diversity (Moodley & Graham, 2015; Kapp et al., 2013) Cultural perspectives (Larson, 2008) 3. Disparity issues in the study of disability Social determinants of health, health disparities, & the lifecourse (Krahn et al., 2015) Service systems: health (National Council on Disability, 2016) Service systems: education, social services (Bezyak, Sabella, & Gattis, 2017) 4. Foundations of measurement issues for studying disability in public health Screening & surveillance (USDHHS, 2011) Bioethics & research (Frankena et al., 2018) 5. Current and future issues in disability and public health Beyond the US (Norwich, 2008) Future issues (Griffen, Risley, Petros, & Welter, 2018) --PAGE 21-- Appendix C Data collection instrument 1. Please enter your age in years: ________________________ 2. Select the gender with which you most identify: a. Male b. Female c. Transgender d. Gender non-conforming e. Choose not to identify 3. What is your current status as a student? a. Currently a student b. Not currently a student; completed my degree/program c. Not currently a student; have not completed my degree/program 4. Which of the following Certificate of Interdisciplinary Disability Studies in Public Health (Disability Certificate) courses have you completed? (select all that apply) a. PUBH 5501: Foundations of Public Health and Disability b. PUBH 5502: Epidemiology of Disability c. PUBH 5503: Disability Law, Policy, Ethics, and Advocacy d. PUBH 5504: Public Health Interventions in Disability e. N/A; I did not take any of these courses 5. Why did you initially enroll in your first Disability Certificate course? (select one) a. Topic area of interest b. Recommended by an advisor/instructor c. Recommended by peer d. Met requirement for degree program (including Disability Certificate) e. Met requirement for a non-degree program (e.g., LEND) f. Appeal of online course format g. Other: please specify: ___________________________ 6. If you completed more than one course, why did you decide to take more? (select all that apply) a. N/A; I did not take an additional course b. New course in topic area of interest c. Recommended by an advisor/instructor d. Recommended by peer e. Met requirement for degree program (including Disability Certificate) f. Appeal of online course format g. Want to learn more 7. The following occurred as a result of taking one Disability Certificate course: (select all that apply) a. Completed additional Disability Certificate course(s) b. Applied to an additional degree program c. Applied to the Disability Certificate d. Applied the course content in other courses e. Recommended the course to others 8. What is the single most important lesson you learned from your course(s)? _____________________________________________________________________ ___________________________________________________________________ --PAGE 22-- 9. What recommendation(s), if any, do you have to improve the course(s)? _____________________________________________________________________ ____________________________________________________________________ 10. Did you complete the Certificate of Interdisciplinary Disability Studies in Public Health (Disability Certificate)? a. Yes b. No c. I'm currently completing it [Skip to question 12 if = yes; I'm currently completing it] 11. Which reason best describes why you did not pursue the Disability Certificate? a. Did not know about the Certificate b. Did not fit into my Plan of Study for my degree program c. Was not interested in the Certificate d. Was not eligible for the Certificate e. Financial barriers f. Other: please specify: ____________________________________________ 12. Which best describes your current work setting? a. N/A; I'm a student b. N/A; I'm not currently employed c. Clinical, in-patient (e.g., hospital, long-term care facility) d. Clinical, out-patient (e.g. community setting) e. Non-clinical, in-patient (e.g., hospital, long-term care facility) f. Non-clinical, out-patient (e.g. community setting) g. Schools (pk-12) h. Higher education/academia i. Government agency j. Research facility k. Non-profit agency l. Other: please specify:______________________________________________ 13. Which best describes your type of work? a. N/A; I'm a student b. N/A; I'm not currently employed c. Clinical d. Other direct service provision e. Administration f. Education g. Research h. Policy i. Other: please specify: _____________________________________________ 14. Does your current work relate to people with disabilities? a. N/A; I'm a student b. N/A; I'm not currently employed c. Yes d. No 15. As a result of participating in Disability Certificate course(s): --PAGE 23-- Strongly disagree Somewhat disagree Neither disagree nor agree Some- what agree Strongly agree N/A; I'm a student N/A; I'm not currently employed My knowledge in the area of disability and public health has increased I applied to a work position related to working with and for people with disabilities I applied the knowledge I gained to secure my current work position I apply the knowledge I gained to carry out my current work position 16. What was your major/discipline while taking a Disability Certificate course? (select all that apply) a. Public health b. Medicine c. Dentistry d. Speech, language, and hearing sciences e. Human development and family studies f. Public policy g. Social work h. Education i. Natural science j. Nursing k. Engineering l. Business psychology m. Non-degree n. Other: please specify: _____________________________________________ --PAGE 24-- 17. If you have any additional comments, please provide them here: _____________________________________________________________________ An Outcome Evaluation Study of the UConn Online Graduate Courses of the Certificate of Interdisciplinary Disability Studies in Public Health by Tara Lutz & Mary Beth Bruder is licensed under a Creative Commons Attribution 4.0 International License. Based on a work at https://rdsjournal.org/. Permissions beyond the scope of this license may be available at https://www.rds.hawaii.edu/. --PAGE 25--