Evaluating Policy Evaluation

In Spring 2023 I participated in Miami’s Howe Center for Writing Excellence Faculty Writing Fellows Program. As part of the program I redesigned my policy evaluation course to incorporate writing for a variety of audiences. Beginning in Spring 2024 I began collecting data from students in the class to measure their learning gains.

The syllabus for the Spring 2024 version of the course is available here.

Survey Instrument

1. Please rate the following statements. [Strongly disagree, disagree, agree, strongly agree]

  • I feel confident in my ability to write in university-level courses.
  • I feel confident in my ability to write in political science courses.
  • I know how to write an editorial for a local newspaper.
  • I know how to write a public comment for a proposed regulation.
  • I know how to write a policy brief.
  • I know how to create an ePortfolio.

2. During the current school year, how often have you done the following in your courses? [Very much, quite a bit, some, very little, not at all]

  • Discussed the ethical consequences of a course of action
  • Generated multiple solutions to a problem
  • Evaluated the credibility of sources
  • Written something that used information from a variety of sources
  • Written something that presented multiple perspectives

3. Please answer the following questions. [Not at all important, somewhat important, moderately important, very important]

  • How important is receiving feedback from your peers for your writing process?
  • How important is receiving feedback from your instructor for your writing process?
  • How important is revision for your writing process?
  • How important do you think writing is to your major?
  • How important do you think writing will be to your career?

4. What writing skills do you think will be most important in your future career? Why? [Open-ended]

5. The audience for a piece of writing refers to who will be reading the work. With that in mind:

  • How would you describe the audience of a newspaper editorial? [Open-ended]
  • How would you describe the audience of a public comment? [Open-ended]
  • How would you describe the audience of a policy brief? [Open-ended]

6. What is/are your major(s)? [Select all that apply.]

  • Political Science
  • Public Administration
  • Public Health
  • Other: ___________

7. What year do you intend to graduate?

  • 2024
  • 2025
  • 2026
  • 2027
  • Other:

8. What ethnicity/race do you identify as? [Select all that apply.]

  • Asian/Pacific Islander
  • Black/African American
  • Latino/Hispanic
  • Middle Eastern
  • Indigenous/ Native American
  • White/Caucasian
  • Multiracial
  • Other: _____________

9. Do you consider yourself a Native English speaker? [yes/no]

E-portfolio assessment

CriteriaPoints
Reflective essay
Demonstrate understanding of how writing differs by genre/audience10
Demonstrate understanding of revision process10
Editorial
Reflect on writing for a general audience17.5
Policy brief
Reflects on peer review process17.5
Public comment
Reflects on writing for expert audience17.5
Infographic
Reflects on transforming policy brief into a visual format17.5
Misc.
Organization, spelling, grammar, punctuation10
Total100

References

  • E-portfolios:
    Ambrose, Julie K., Kelly A. Delaney-Klinger, Kristina Hoeppner, Leanne Ngo, and Patsie Polly. 2017. “Transition to Career and Career Development.” In Field Guide to Eportfolio: Why it Matters for Learning. AAC&U. Washington, DC.

    Cahill, Alice S., Rebecca M. Nelson, Jane Strawhecker, and Phu Vu. 2022. “School Administrators’ Perceptions of Electronic Portfolios and the Hiring of K-12 Teachers.” International Journal of ePortfolio 12(1): 47-58.

    Jones, Beata, and Carrie Leverenz. 2017. “Building Personal Brands with Digital Storytelling ePortfolios.” International Journal of ePortfolio. 7(1): 67-91.

    López-Crespo,Ginesa, M. Carmen Blanco-Gandía, Sonsoles Valdivia-Salas, Camino Fidalgo, and Noelia Sánchez-Pérez. 2022. “The educational e-portfolio: preliminary evidence of its relationship with student’s self-efficacy and engagement.” Education and Information Technologies 27: 5233-5248.

    Lukitasari, Marheny, Rusdi Hasan, Akhmad Sukri, and Jeffry Handhika. 2021. “Developing Student’s Metacognitive Ability in Science through Project-Based Learning with E-portfolio.” International Journal of Evaluation and Research in Education 10(3): 948-955.

    Syzdykova, Zhibek, Kalybek Koblandin, Natalia, Mikhaylova, and Olga Akinina. 2021. “Assessment of E-Portfolio in Higher Education.” International Journal of Emerging Technologies in Learning (iJET) 16 (2): 120-134. 
  • Writing in the political science classroom:
    Anson, Ian G. 2017. “Audience, Purpose, and Civic Engagement: A Reassessment of Writing Instruction in Political Science.” Journal of Political Science Education 13(4): 389-403.

    Beaumont, Elizabeth, Anne Colby, Thomas Ehrlich, and Judith Torney-Purta. 2006. “Promoting Political Competence and Engagement in College Students: An Empirical Study.” Journal of Political Science Education 2: 249-270.

    Blings, Steffen, and Sarah Maxey. 2017. “Teaching Students to Engage with Evidence: An Evaluation of Structured Writing and Classroom Discussion Strategies.” Journal of Political Science Education 13(1): 15-32.

    Butcher, Charity. 2022. “Using In-Class Writing to Promote Critical Thinking and Application to Course Concepts.” Journal of Political Science Education 18(1): 3-21.

    Crowder-Meyer, Melody. 2021. “Blogging Your Way to a Research Paper: The Benefits of a Semester-Long Blogging Assignment in the Political Science Classroom.” Journal of Political Science Education 17(3): 482-492.

    Dickovick, J. Tyler. 2009. “Methods in the Madness: Integrative Approaches to Methodology in Introductory Comparative Politics.” Journal of Political Science Education 5:138-159.

    Hansen, Holley E. 2016. “The Impact of Blog-Style Writing on Student Learning Outcomes: A Pilot Study.” Journal of Political Science Education 12(1): 85-101.

    Mastro, Oriana Skylar. 2021. “Teach What You Preach: A Comprehensive Guide to the Policy Memo as a Methods Teaching Tool.” Journal of Political Science Education 17(S1): 326-340.
  • Political science and professional development:
    Bram, Curtis. 2023. “A Professional Development Course for Political Science Undergraduates.” PS: Political Science & Politics 56(1): 154-157.

    Chagas-Bastos, Fabrício, and Sean W. Burges. 2019. “The “Briefing Note” as a Pedagogical Tool for Teaching Politics and International Relations.” Journal of Political Science Education 15(2): 237-246.

    Kinsella, Chad, and Brandon Waite. 2021. “Identifying and developing desirable soft skills for public service.” Teaching Public Administration 39(3): 337-350.

    Lewis, Gregory B. 2017. “Do Political Science Majors Succeed in the Labor Market?” PS: Political Science & Politics 50(2): 467-472.

    Mallinson, Daniel J., and Patrick Burns. 2019. “Increasing Career Confidence through a Course in Public Service Careers.” Journal of Political Science Education 15(2): 161-178.

    McMillan, Samuel Lucas. 2014. “Bravo for Brevity: Using Short Paper Assignments in International Relations Classes.” International Studies Perspectives 15(1): 109-120.

    Pennock, Andrew. 2011. “The Case for Using Policy Writing in Undergraduate Political Science Courses.” PS: Political Science & Politics 44(1): 141-146.

    Rogers, Michael T. 2021. “A Career-Oriented Approach to Structuring the Political Science Major.” PS: Political Science & Politics 54(2): 387-393.