Introduction
The Scout Mindset
Julia Galef's 'The Scout Mindset' introduced the idea of a mind frame involving searching for what the truth actually is, rather than what your personal beliefs and biases would incline to to think. The Scout Mindset is key to seeing the world clearly, having constructive disagreements and forming accurate models of the world. We need an increased understanding of what groups of people are more likely to possess the scout mindset, other correlated factors, how to measure it and how to increase the proportion of the population that have this mindset. Promoting these skills could improve society in a variety of ways. The work of scientists, researchers and other career paths oriented towards doing good would be enriched. Moreover, a greater understanding of how people make judgements will likely aid the development of strategies to improve people's rationality and decision making skills. Fields from education and healthcare to business could benefit.
Research Aims
Currently, there is a lack of a properly validated psychological scale to measure the scout mindset. From a scientific point of view, It would be helpful to derive some method of gauging the extent to which the scout mindset prevails in different people, as well as which characteristics or personality traits factor in. My project aim was to provide a start to exploring different methods of testing the characteristics of scout mindset and carrying out some of this research.
More specifically, I created two distinct psychological scales for measuring the Scout Mindset (one self-report and one using conditional reasoning), to examine the correlation between direct and indirect personality testing. In order to validate the scales, I gave each test to two groups of participants with different educational backgrounds (one consisting of Cambridge University undergraduate students, and one of people with no particular shared higher education), to see whether the results matched my expectations. My hypotheses were that the psychological scales would be somewhat correlated (which one is more accurate is another question to discuss) and that the university students should score more highly on both scales.
Types of Psychological Scales Investigated
Self-Report Scale
Self-report scales work on the basis of respondents evaluating and conveying their own characteristics. Often, they are presented with a statement or question about themselves, and they respond with the extent to which they agree on a scale of one to five.
These tests are not always the most accurate way to measure characteristics of people’s identity, because they rely on people consciously analysing and accurately reporting their traits. A lack of self awareness, or a desire to appear a certain way, can easily lead to people providing responses that do not truly represent them. This risk is especially significant in a test for the scout mindset, since I wish to test self-awareness and bias themselves.
Some measures can be taken to avoid any biases and mistakes within self report surveys, which I have employed in my survey design. For example, reverse scoring is when in some of the questions (for example half), the direction of the scale runs the opposite way. For example, in a question that asks ‘I am open to changing my beliefs’, picking ‘strongly agree’ would provide the most points in a test that checks for open mindedness - reverse scoring would require a question that asks ‘I am not open to changing my beliefs’, and agreeing with this would lead to a low number of points. This technique tests for intra-rater reliability - that individuals’ responses are cohesive. It ensures that people reply and represent themselves consistently, leading to accurate results. Otherwise, to give one example, the results would be skewed by someone who simply picks ‘agree’ for every option. Another method would be to rephrase questions in a more positive or negative way, which avoids response bias (people answering inaccurately in order to paint a more positive picture of themselves). For example, the question ‘I am stubborn’ could be rephrased to ‘I stick to my opinions’.
The items I selected to measure through a self-report scale were 'truth-seeking', 'open-mindedness' and 'questioning of beliefs and identity'. As described above, the questionnaire assessed these characteristics in a range of different ways. For example, for 'truth-seeking', my survey asked both ‘Part of my identity is being a truth-seeker’ and ‘I value stability over exploration’, demonstrating the reverse scoring method. For 'questioning of beliefs and identities' it included ‘I often think about the uncertainty of my beliefs and assumptions.’ and ‘I stick to my personal identity no matter what’, trying to avoid people presenting themselves in the way they thought they should. A factor that would have increased the reliability of the test was that most of the respondents were likely not well acquainted with the concept of the Scout Mindset, so there was a lower chance of them providing the answers that they thought they should.
Conditional Reasoning Test
The second method of psychological survey I will be investigating involves a more indirect investigation of personality tests. Implicit personality testing avoids pitfalls of self-report methods, since bias is less likely to occur and people are less able to manipulate results.
Conditional reasoning tests work by identifying the characteristic you want to test and the biases people are prone to based on this. Bias causes us to rationalise in order to justify our beliefs. So, people with a certain bias are more likely to reason a certain way. In the test, a concept or theory is introduced, and four possible responses are listed. Two are spoof answers, but the others are both logically plausible (just based on different assumptions). The one people intuitively pick reveals their motivated reasoning and underlying characteristics. Hence, conditional reasoning questions appear to test subjects’ logical reasoning like an IQ test, but in reality work to identify respondents’ root biases!
Trial studies show a good correlation between conditional reasoning tests and self report statistics. Implicit personality measurement techniques have even been shown to be better predictors (than self-report measures) of concrete things like career success.
Below is an example of a conditional reasoning question I used. Participants were presented with a few sentences on the topic of free will, and asked to identify a valid criticism of the argument. The analytical, Scout Mindset answer would be option three, which is based on the premise that we should evaluate our definitions and arguments. On the other hand, option two is also logically valid (again, depending on your definition of free will), except that it does not factor in the same mindset of truth-seeking and questioning beliefs which seems immediately obvious. Through questions like these, the hope was that participants would think they have identified the 'right' answer, regardless of which of the two logically viable options they picked, and so answer in a way that is representative of their beliefs and thinking.
Validating the Psychological Scales
Scout mindset is linked to bias and self-deception, so I decided to see whether conditional reasoning offers a more accurate way to test it (by checking whether results correlate with the self-report scale).
To validate the surveys, I decided to approach two groups of people with them. This way, if the relationship of results between groups followed the predicted outcome based on what I know about scout mindset, it would be some indication that the tests are valid. I selected a group of students attending Cambridge University (group 2). My hypothesis was that this group of individuals with a very good education, access to reading materials and an analytical mindset would score higher on my scout mindset metrics. The other sample I used was from a group of parents on Facebook (group 1), with no particular higher education in common.
I called the self-report, shorter answer survey 'survey 1' and the longer conditional reasoning test 'survey 2', giving both surveys to each group.
Research Methods
Creating the Self-Report Scale
The questions were:
- What is your age?
- I mostly know all I need to know about the world already.
- I value stability over exploration.
- I enjoy learning by listening to opinions different to mine.
- It is likely my beliefs about the world will change in the near future.
- I believe I am self aware.
- I stick to my personal identity no matter what.
- Part of my identity is being a truth-seeker.
- I often think about the uncertainty of my beliefs and assumptions.
- I prefer to accept my family's/community's beliefs rather than question them.
- I reflect over my actions and words often.
- I am happy to adjust my plans in light of new information.
- Failure has a deep discouraging effect on me.
- Your intelligence is something about you that you cannot change very much.
Crucially, several of these questions (1, 6, 13, 14) were not assigned point values. I simply wanted to investigate differences in age, reports of self-awareness and growth mindset between the two sample sets, to see whether there was a notable difference. Self-awareness, for example, could be interesting in that if people are more aware of their own biases, that should increase their levels of Scout Mindset. Additionally, there is a chance that growth mindset is correlated with Scout Mindset, if the latter is something learned and developed.
A pilot study using a small number of family and friends led to me rephrasing some of the questions in a more understandable and eloquent way, so that I was asking the questions that are most relevant. I ended up with a total of ten questions that were actually assessing Scout Mindset. I decided to keep the number of questions low, in order to hold the attention of as many respondents as possible, and make sure more people finish the survey.
I used Qualtrics to format the survey, which enabled me to assign points to the scales (I formatted them as matrix tables. For questions that aligned with the scout mindset, such as ‘part of my identity involves being a truth seeker’, ‘strongly agree’ gave 10 points and ‘strongly disagree’ gave 2, with five options total (strongly agree, somewhat agree, neither agree nor disagree, somewhat disagree and strongly disagree). This way, the total possible points added up to a round 100.
Creating the Conditional Reasoning Test
Designing these questions was a challenge, since each begins with the introduction of a new idea or theory and I had to take the perspective of people with different mindsets/biases. It was difficult to phrase the questions in a way that would enable the participants to understand the concepts as quickly as possible and be able to reason through an answer to the question. This involved another pilot study which involved me making a lot of changes, as I asked people who were unfamiliar with the ideas and had to adapt a few of the sentences to make them easier to follow. The questions I ended up with were structured like the example I presented earlier - two spoof answers, one from the perspective of someone without Scout Mindset and one from the perspective of someone with Scout Mindset. Both of the latter were inductively logical.
The questions and answers:
Results
Self-Report Survey
Group 1 - Parents:
Group 2 - University Students:
The standard deviation of Survey 1 was 6.64, whereas that of Survey 2 was 8.49 - which was quite a bit higher. This could perhaps suggest that the results of survey 2 would have been more significantly higher (or lower) if there was more agreement between respondents. It is also worth noting that the mode score of group 2 was 84, whereas for group 1 it was 74, which is a more notable difference. As it is, the average scout mindset score of group 2 was 5.56 points higher than that of group 1. I carried out a hypothesis test (seen below).
H0: The population mean scout mindset score of group one is 76.43
H1: The population mean scout mindset score of group two is lower than 76.43
Significance level = 5% (0.05)
Standard Deviation = 1.77 (8.49/root23)
Mean = 76.43
X~N(76.43,3.13)
p(X≤70.78) = 0.0007075
0.0007075>0.05
The test confirmed that this was a statistically significant result; Group 2 indeed had a higher average Scout Mindset score than Group 1.
Interestingly, however, there was not much of a correlation between growth mindset and the overall scout mindset scores between the groups. If anything, the group of were slightly more likely to report that they are not affected as strongly by failure, and that intelligence is not fixed. Perhaps this has something to do with the age difference between the groups, though - studies show that growth mindset consistently declines from early to late adolescence, and only rises in later adult life. With the self-awareness question though, group 2 described themselves as self-aware more consistently than group 1. Interestingly, zero participants in either group picked ‘somewhat disagree’ or ‘strongly disagree’ in response to the question ‘I believe I am self aware. And, in group 2, zero participants picked ‘neither agree nor disagree’! The results of this do seem to correlate with the scout mindset relationship between the groups. But the extent to which this particular self-report question can be trusted is something to consider.
Conditional Reasoning Test
Group 1:
Group 2:
Survey 1 had a lower mean than survey 2 (61.43 vs. 77.39), as well as a lower median score (60 vs. 80). The mode of survey 2 was also much higher (90 vs. 60). Visually, survey 2 also seems to have a higher standard deviation - the results were more spread out - and a much larger range. This was perhaps due to the larger sample size (24 vs. 14). However, the large differences in score visible in all these measures of central tendency indicate a pretty reliable trend: the average points earned by group 2 were larger than group 1. All in all, there was a correlation between the relationship of group 1 and group 2 scores for survey 1 and survey 2, meaning survey 2 seems to have been a successful measure of scout mindset (or at least thought patterns related to the scout mindset). Moreover, there was a significant divide between the responses of the two groups for many of the conditional reasoning questions, and a much greater proportion of people in group 1 seemed to be picking the non scout mindset answers than in group 2. This is promising, since it means that the question design was effective to some extent - it achieved the differentiation I was hoping for.
Evaluation
Validity
There was correlation between the two surveys (showing similar trends within each group) - the total score for each survey was 100, and the mean scores for each group were similar in both surveys they did. Moreover, the correctly predicted difference in results between the groups (that Cambridge students would score higher) indicates that the surveys could potentially be valid measures of the scout mindset. This is promising, since the success of conditional reasoning means it could be incredibly helpful in measuring these kinds of things in the future, especially if even more time goes into devising insightful questions.
Further Research Needed
We have concluded that there is some correlation between the results of the two tests. However, from a practical point of view, the next step needs to be figuring out which of the two surveys (conditional reasoning or self-report) is actually more accurate, since this will determine whether further research should go into investigating indirect methods of personality measurement - there is no point if direct measures are just as accurate. This could be done through finding some ground measure of the Scout Mindset, taken to be truth, and comparing it to my results. One option for this is a more subjective, qualitative method of analysis, such as observation of behaviour in individuals. Additional resources would be needed.
In the future, a number of additional techniques could also be implemented to check how representative my results are (which is necessary), and make future progress. The main problem perhaps was sample size - I was limited in time and access to respondents (especially of the groups I wanted). For survey 1, I only managed to receive 23 responses from group 1 and 42 from group 2. For survey 2 I received 14 responses from group 1 and 23 from group 2. This means there were 102 responses in total. In an ideal world, there should have been at least 400 participants in the surveys, in order to ensure reliable results. The results I did get should definitely be tested to see if they are repeatable or reproducible - this is something I can do in the future.
Factor analysis should be carried out on my results - grouping questions with similar trends in responses to analyse which questions are most useful. I would like to ask even more questions on a greater range of scout mindset traits. Then I will identify questions for which responses are naturally clustered, with a high correlation between answers. Clustering retrospectively is how the big five personality test was developed (you’re only really asking five questions, but they are asked in different ways to confirm results). Then, clean-up can be done after testing the survey - getting rid of irrelevant or only weakly related factors in order to make the questioning more precise.
Checking for validity is something I think I can do more work on - to check the tests are measuring exactly what I mean to measure. For example, a test is good if it correlates well with existing measures. In this case I am measuring something new, but the Scout Mindset is linked to other traits for which validated scales exist (such as open mindedness on the big five personality test). So, testing another established scale alongside my questions to check for correlation will help (for example, Tetlock states that there is a correlation between open mindedness, which factors into the scout mindset, and people’s forecasting ability). Additionally, It would be good to do further testing on whether results show a different trend for researchers/experts (especially forecasters), if I can get access to these. I will also need a way of determining whether I have added anything new to existing scales, and am therefore testing something worthwhile (incremental predictive validity).
An interesting question for further research would be whether the difference in results was caused by nature or nurture. The higher level of Scout Mindset for university students could indicate that resources, upbringing and access to high quality teaching lead to a Scout Mindset, which would mean that it is something learned rather than innate and can be promoted more throughout the general public! Equally, however, it could be concluded that the people going to Cambridge University have been born with this nature, and the Scout Mindset inevitably comes along with the other aspects of their mindset and abilities (rather than being developed as a by-product). Further research is needed to gain understanding of how factors that contribute to the Scout Mindset develop.
References
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best Practices for Developing and Validating Scales for Health, Social, and Behavioral Research: A Primer. Frontiers in Public Health, 6(149). https://doi.org/10.3389/fpubh.2018.00149
New personality test is faster -- and tougher to trick. (n.d.). ScienceDaily. Retrieved August 30, 2023, from https://www.sciencedaily.com/releases/2018/08/180830100800.htm
Galef, J. (2021). Scout Mindset.
Meade, A. W., Pappalardo, G., Braddy, P. W., & Fleenor, J. W. (2018). Rapid Response Measurement: Development of a Faking-Resistant Assessment Method for Personality. Organizational Research Methods, 23(1), 181–207. https://doi.org/10.1177/1094428118795295
Galić, Z., Bubić, A., & Kovačić, M. P. (2016). Alternatives to Self-Reports. The Wiley Handbook of Personality Assessment, 215–227. https://doi.org/10.1002/9781119173489.ch16
LeBreton, J. M., Grimaldi, E. M., & Schoen, J. L. (2019). Conditional Reasoning: A Review and Suggestions for Future Test Development and Validation. Organizational Research Methods, 23(1), 65–95. https://doi.org/10.1177/1094428118816366
Scout mindset - EA Forum. (n.d.). Forum.effectivealtruism.org. Retrieved August 30, 2023, from https://forum.effectivealtruism.org/topics/scout-mindset
How to Create Psychological Scales and Inventories. (n.d.). Www.youtube.com. Retrieved August 30, 2023, from https://www.youtube.com/watch?v=uTW_hH4LbYw
Chen, J., & Liu, C. (2023). The longitudinal association between children’s growth mindset in senior primary school and their parents’ growth mindset. Frontiers in Psychology, 14. https://doi.org/10.3389/fpsyg.2023.1110944
Great work! I really like the conditional reasoning test idea; it's something I hadn't really thought about in this context. I haven't reviewed the questions in detail yet but any thoughts on whether they'd be suitable for an application process?