Productivity tracking tools often determine productivity based on the time interacting with work-related applications. To deconstruct productivity's diverse and nebulous nature, we investigate how knowledge workers conceptualize personal productivity and delimit productive tasks in both work and non-work contexts. We report a 2-week diary study followed by a semi-structured interview with 24 knowledge workers. Participants captured productive activities and provided the rationale for why the activities were assessed to be productive. They reported a wide range of productive activities beyond typical desk-bound work-ranging from having a personal conversation with dad to getting a haircut. We found six themes that characterize the productivity assessment-work product, time management, worker's state, attitude toward work, impact & benefit, and compound task and identified how participants interleaved multiple facets when assessing their productivity. We discuss how these findings could inform the design of a comprehensive productivity tracking system that covers a wide range of productive activities.