Participatory Design (PD) is envisioned as an approach to democratizing innovation in the design process by shifting the power dynamics between researcher and participant. Recent scholarship in HCI and design has analyzed the ways collaborative design engagements, such as PD situated in the design workshop can amplify voices and empower underserved populations. Yet, we argue that PD as instantiated in the design workshop is very much an affluent and privileged activity that often neglects the challenges associated with envisioning equitable design solutions among underserved populations. Based on two series of community-based PD workshops with underserved populations in the U.S., we highlight key areas of tension and considerations for a more equitable PD approach: historical context of the research environment, community access, perceptions of materials and activities, and unintentional harm in collecting full accounts of personal narratives. By reflecting on these tensions as a call-to-action, we hope to deconstruct the privilege of the PD workshop within HCI and re-center the focus of design on individuals who are historically underserved.