Effective Global Scientific Research

Scientific advancements are one of the greatest drivers for improving the quality of life of everyone on the globe. In this post I’ll present an overview of an idea, which is basically aimed at improving the allocation of resources on scientific research such that it will correspond better to what is important for society.

We’ll start by a bit of background, and then go to the actual idea. The idea is simply to construct a graph of the important objectives in science, and how each problem relates to other problems.

Why work on better scientific processes?

Science in itself has some value (akin to arts, but maybe the cultural influence is more important), but the main importance is in it’s effects on technology. First, in order to have better technology (tools, infrastructure, hardware, etc.) we need the scientific understanding of how it can be done efficiently, or even realize that it can be done at all. Secondly, the use of the technology is much better when there is better knowledge. So, a physician prescribing pills to a sick patient relies on tons of science done before him, from the medical research done to make the diagnosis clearer, to the biological research explaining the causes of the disease, to the material research making the digestible polymer of the pill, and even the mechanical engineering research done to make the pill more affordable.

The effective altruism movement is about optimizing the amount of good you can do (and actually do).  One of the important focus areas examined by effective altruists is the advancement of scientific research, as explained in the Open Philanthropy Project website or analysed in Givewell’s blog. They will do a better job at explaining the situation than me.

For me, it seems like the improvement of the collective intelligence of humanity is a value of itself. That is one reason why I find this area motivating to think about, and I think that there is a high chance that I’ll find myself working in that domain in the future. I do think that science should be more directed at tackling the problems that would bring the greatest expected utility to mankind, which is exactly why I myself had decided to stop my career in abstract mathematics. Note that it does not mean that we should stop all research not directly associated to real life problems, because it is important to branch out and take risks, as well as I think it is okay for people to decide to work in fields which they find interesting but not beneficial even though they should be motivated otherwise.

How does science advance today, and what are it’s major obstacles?

I do not have a sufficient understanding to fully answer this question. I mainly want to stress that it is important to arrive at satisfying conclusions of what the main problems are before trying to solve them. While day-dreaming about interesting ideas, and contemplating how to “save the world” I often find that I do not have a clear vision of what exactly are the problems to be solved, and what the world needs rescue from.

There are some aspects of science as it is done today which are important to keep in mind:

1. The goals of the researchers in any domain are usually not to work on the problems which will translate to the highest expected value to humanity. It is an important goal for many researchers, but their goals are also composed of several other goals, including but not limited to:
   1. to satisfy their own curiosity.
   2. to have better publication record (causing researchers to take less risks).
   3. career goals. (Look at the difference between how researchers behave before/after tenure or at the difference between universities to private research institutes).
   4. more grants.
2. The experts in any given field have the best understanding of the challenges the field is facing, and the most effective research directions to push the boundaries of the field.
3. Many times, the advancements in science arise from better technologies for measurement.
4. There is a clear trend of obtaining more and more data, where data analysis plays a critical role in advancing science forward.
5. Many specific questions requires multidisciplinary approaches to be solved.

Mapping the landscape of science

Do you know what are the burning question in quantum thermodynamics? How would you figure that out? Well, the google search for “open questions in quantum thermodynamics” had as a first hit the physics.stackexchange question “What are the open problems in the field of quantum thermodynamics?” to which was a single answer which referenced to an article reviewing the entire topic. Seems like exactly what we’d want! Actually.. what do we want?

Goals

Why would we be interested in such a question? There are two cases:

1. An expert in the field, wants to communicate with other experts. In this case, the expert already knows the main problems, and wants to know what are the specific problems other researchers are working on or find important for their own research agendas. She may also want to contribute Her voice to the conversation.
2. Someone in or outside the field who is is interested in learning the field or doing research in it.

An expert will work to be up to date with the literature, and will communicate heavily with other researchers in her field. There is probably more room for improvement and community building in some domains, but that is not the point of this post.

As for the second case, of someone not deeply familiar with the field, the quantum thermodynamics review mentioned above seems a great place to start, although a conversation with an expert might be better. Anyway, that is also not the point of this post..

The (convolutedly confusing) point is that we need to ask a different question. We should ask as society what are the important research questions. What are the questions that, once solved, would bring the greatest benefit to humanity as a whole? To answer this question, it is insufficient to rely on the experts in the field. We must gain input from technologists, scientists in related field, governments and other outsiders to the field, in regards for what they find as the important objectives in the field.

How to measure it?

This can be done by backtracing recursively from a clearly important objective to different questions that may help solve that objective. For example, let’s say we want to cure cancer. There are several general (overlapping) paths available, say immunotherapy or gene therapy. Now, each of these can be decomposed into several questions or sub-objectives of it’s own and so forth.

Note that the graph obtained, where nodes are objectives and edges are the relation of “ helps solve “, can have circles, and large output or input degrees.  The key idea is that you can assign weights to the edges of this graph, which measure in some way the importance of the subproblem for the solution of the target problem. The goal of this weight is to help manage what are the most important subproblems to work on, for which there are three important dimensions, using the framework developed at givewell:

1. Importance: How much does solving the subproblem will solve the target problem?
2. Tractability: How much is the subproblem likely to be solved?
3. Neglectedness: How many people are already working on it? More precisely, what is the marginal utility of one more person working on the subproblem?

Out of these 3, only Importance is a property of the edge (and not of the subproblem node itself), thus it is a good idea to put a measure of Importance on the weights. Both Tractability and Neglectedness should be estimated and maintained for each node, so that together we could analyze the graph obtained and get answers to some questions such as how much does a research on a specific gene activation is expected to contribute to curing cancer. Note that all three can have actual numerical estimation, so that multiplying all three results in the expected amount by which one extra person working on the subproblem will help solve the target problem.

There are several limitations of the above framework.

1. The benefits of partial progress should be taken into account as well. Without taking it into account, we are actually assuming that partial progress is of linear benefit. This can be made possible by having a function which measures how much does a given percentage of the subproblem helps solve the target problem. However, this is also not an accurate description, as solving a problem is not 1 dimensional. Also, possibly more importantly, it is much harder to estimate. A better way to take that into account would be to divide the subproblem and the target problem further.
2. It does not take into account personal compatibility. This would also complicate the graph a lot, but fortunately it is possible to adjust when analyzing the graph. The Neglectedness metric is assuming some sort of capabilities of the average researcher, in itself not an exact notion, but that can be multiplied by a factor of a personal fit which measures how good you are compared to the average researcher.
3.   It is difficult to estimate properly. We’ll deal with it later:

Building the graph

The graph itself is difficult to construct. What are the key questions? what are the dependencies? How to divide problems into subproblems?

I like to think that this is best solved by “open sourcing” the graph. We can let anyone edit the graph, inserting further questions and connections as necessary. There should be a mechanism for resolving disputes, maybe like in Wikipedia or maybe to assign “admins” responsible to their specific regions of the graph. Ideally, this graph can be used as a framework for resource allocation in universities and other research institution (possibly because grants would depend on aligning research with societal goals), and then the experts would have the motivation to participate in the construction of the graph.

The question of why would the experts contribute is extremely important for such a project to succeed. There need to be a model for how to motivate contributions once the graph is large enough, arriving at the stable eventual state. There also need to be a model for how to kick it off, for example starting with one single department in one single university. I won’t pursue this line further.

Estimating the metrics on the graph

Giving a good estimation for each of the three metrics can never be done perfectly. However it is important to try and estimate it, and now we’ll discuss a bit on how it can be done.

In general, we’d want the people knowledgeable about these estimates to be the ones estimating. Also, it will be the most accurate and bias-free if the result would be some kind of an average over the different opinions. I’ll explain a possible way to get the relevant information by proper participation of experts.

An expert will not be familiar with all the problems contributing to her research goals, as well as all the problems her research goals impacts. Their area of expertise is limited, and hence there are many important nodes for which it is not clear who can make the best estimates. Despite of that, the aggregated understanding of many people partially knowledgeable about the question at hand can make an intelligent estimation of all metrics. This is why I think the best solution would eventually require many people to participate, perhaps in a weighted voting scheme of sorts.

Some day I will return to write about how to asses each one of the different metrics, and give more options in general. It seems that there is a wide variety of different options and technologies to use, so it would take a lot of effort do write about it. However, It seems to me that the obstacle is more economic than technological. That is, if we can find a way to get the scientific community involved in this project then finding the correct platform and equations to hold it all together would not be a big deal.

Importance

Tractability

Neglectedness

Conclusion

The main goal is to give a tool that would show what are the most promising research directions. This can be used by policy makers or private funds to better allocate their resources toward research with greater societal value.

Improving the efficiency of scientific research, and the translation of it to societal benefits, can have a huge effect on world.