The problems of measurements and its consequences for models
In social models, we usually represent quantities of some abstract constructs (e.g. happiness, opinion, stubbornness...). This comes so natural to us that we often do not even question this process. However, entities like opinions do not appear naturally in a numerical form, but they are transformed into numbers through the process of measurement.
One of the biggest problems of measuring in the social sciences is that we want to measure complex entities which go beyond the data. For instance, we may collect some data about smiling and laughing, however, usually, we are not interested in the data per se, but in what it tells us about some underlying construct, such as happiness. This results in two main problems.
The first one is related to ordinality. Indeed, even if we assume no error in our data, we still have that the relationship between our data and the construct we want to measure is usually non-linear. This is a well-known problem in the social sciences (even if it is still often forgotten) where tools such as non-parametric statistics are used. However, many models, especially in the social simulations, completely neglect this aspect and suppose that all variables are of interval type (i.e. non-ordinal).
The second problem relates to operationalization. Indeed, there are multiple pieces of evidence that the same measurement can be operationalized in many different ways, often producing very different results. This seems to suggest that sometimes some of the phenomena we would like to quantify are more complex than expected and cannot be summarized in a single number.
While these are very complex problems and no simple solution exists for now, I believe that we could strongly improve the situation by producing models which strongly mimic the data's behaviour. This could be achieved, for example, by producing models from experiments, or from sources of dynamic data.
While this won't directly inform us about the underlying constructs, it will definitely give us much better understanding of how some observables relate to each other in a dynamic way. Furthermore, since such approach would be focused on data (instead of the construct) we will not have to worry much about the previously mentioned problems.
List of Endorsements
Report inappropriate content
Is this content inappropriate?
Close debate
What is the summary or conclusion of this debate?
Comment details
You are seeing a single comment
You can check the rest of the comments here.
I completely agree with you on that the "measurement problem" in social modelling is one of the big issue to face and the elephant in the room. Even more after the explosion in data availability produced by the social web. I really appreciated this paper of yours on the subject: Propagation of measurement error in opinion dynamics models: The case of the Deffuant model.
That said, I wonder if it is worth exploring what the application of generative artificial intelligence algorithms can contribute to the resolution of both the operationalization and quantification problems. As an example, a large language model can in principle be fine tuned on a set of pairs (opinion message, self-reported numerical internal attitude) and subsequently be used to "measure" the numerical internal attitude of other messages. Obviously, the produced scale would be some how arbitrary and possibly non-linear, but those are not great problem as far as the measured quantities are self consistent across different datasets (something that Pablo Jensen @pablo call weak objectivity in his book Your Life in Numbers ). I think what is most promising about this approach is the possibility of standardizing the measurement process.
I briefly try to explain my point. Measurement are of two kinds, direct comparisons, as in the case of the size of an object, or by observing the effect of an interactions, as in the case of temperature, in which you observe the equilibrium state after thermal interactions. Because of that, I deliberately avoided "sentiment analysis" in the previous example, since we don't have a standard model of sentiment interactions. But we do have something very close to a standard model of opinion interactions: bounded confidence. An hypothetical opinion thermometer based on a fine tuned large language model could be instructed to behave exactly like a Deffuant agent, and adjust it's internal attitude in such a way that it could generate a message close to that it is measuring. This would be our measure of the expressed opinion.
Loading comments ...