Developing Measures for Your Program Evaluation or Research Study

What is often not evident in the evaluation and academic research process is how much prep work and initial investigation goes into understanding the program or phenomena being assessed. Much like icebergs, where the bulk is underwater, the report and data visualizations are at the end of a long, and sometimes very involved, planning and data collection process. Part of the process is deciding what to measure and how to measure it.

The what in your measurement may come from theory, prior research, or the work you have done with the program stakeholders to create visual depiction of how the program works, called a logic model (You can see an example of a logic model here: 7.6: Example of a Logic Model with evaluation questions – Enhancing Program Performance with Logic Models. A theory of change will also likely be developed through this process, but that and logic models will be the topic of a separate article). Theory or logic models will give you ideas of what is important to measure, which we’ll call a concept. But how does one go from having a concept to figuring out how to measure it?

One step in this process of going from a concept to a measure is to think through what would vary between subjects based on this concept of interest. Though this might sound challenging, people are so different, you wouldn’t usually get a homogenous response to a question. Even with natural phenomena there will be variations. For example, the amount of snow that accumulates from a storm will vary by location. Even between locations that are relatively close to each other there will be a range of snow. Indeed, traditional statistical modeling is based on the idea of there being variation in data.

Once you figure out what varies, or what is the variable associated with your concept, the operationalization, or the translation of your concept into a measure (you can find a good discussion of operationalization here: Types of Measurement Validity - Research Methods Knowledge Base), is easier (though there will also be a lot of testing involved once you come up with a measure as discussed in Types of Measurement Validity - Research Methods Knowledge Base , or you can pick a measure that has an established history of use and established reliability/validity).

We can see how this would work through an example. Let’s say you are interested in assessing changes in income due to a job training program. In this case income would be the concept. Income variations between people could be due to differences in wages, but also income from public assistance, social security, rent, or other assets. Since your study is looking at the effect of a job training program, then you would most likely be interested in changes in wages or earnings. The resulting measure then might be something like this from the Panel Study of Income Dynamics (PSID): About how much did (you/he) earn altogether from working at [that/those job(s)]? (You can find this example and a plethora of other income measures used on the PSID at: T-2Income-PSID.pdf).

Deciding what and how to measure is one of the most critical, and least visible, parts of any evaluation or research study. It requires careful thinking, collaboration, and an understanding that concepts don’t automatically translate into clean metrics. By examining variation, grounding measures in theory or logic models, and selecting or designing tools with strong validity evidence, we set the foundation for meaningful analysis later on. When you see a polished chart or final report, remember that its clarity is possible only because of the thoughtful measurement decisions made early in the process. Strong measures aren’t just technical details; they’re the backbone of trustworthy findings and impactful evaluations.

To have Stacey talk to your organization about measurement for your data collection or to help design your data collection methods, please contact: Stacey@glissandostrategies.com.