Papers, Policies, and Prototypes

I've spent the past year exploring the frontiers of science funding. I've interviewed many of the leading economists and new metascience entrepreneurs. I've been to academic conferences and "DeSci" meetups. I've scoured the literature.

I've heard all the ideas, good and bad. And the wildest of the bunch might surprise you. 

It didn't come from a Silicon Valley pitch deck. It's not a complicated crypto scheme. It came from a team that includes someone who led a federal agency — someone who works in the White House Office of Science and Technology Policy right now.

The proposal: "Rethinking resource allocation in science"

It’s a 2019 paper from four researchers, including Jane Lubchenco. They propose abolishing peer review and replacing it with a "self-organized fund allocation (SOFA)" mechanism where researchers are forced to fund their peers with 50% of the funds that they accrue. In their words: 

To see how a system based on these two principles may work for fund allocation, consider the following two-step procedure:

1. Every participating scientist receives an equal portion of all available funding as his or her base starting budget.

2. Each participant anonymously donates a fixed percentage (say 50%) of his or her funding to other, nonaffiliated scientists.

This is repeated each funding round.

It is important to note that each scientist must distribute a percentage of everything he or she received in the previous round, i.e., the base funding plus what he or she previously received from other scientists. For example, suppose that a scientist receives the base amount of US$50,000 and receives US$150,000 from other researchers. The total received is $200,000, of which 50%, i.e., US$100,000, needs to be donated to other scientists. The scientist retains a total of US$100,000. Because every scientist participates, funding circulates through the community converging over time to funding levels that all scientists have collectively, yet independently determined (Fig. 1, right), unlike the current proposal-based system where small committees of reviewers make recommendations with respect to which projects (or whom) to fund (Fig. 1, left). Importantly, scientists that receive more funding than others also become the more significant funders in the system. This self-organized weighting resembles the mathematical technique of power iteration used to converge on stationary probability distributions of web page relevancy (Bollen et al. 2014, Bollen 2018).

They go on to address the obvious concerns of fraud and systematic bias and make the sensible recommendation that this could be implemented in a small-scale trial first. 

This is a radical idea and I commend the creative ambition. Unfortunately, this system would fail in ways they haven't imagined. 

I have some insight because we have done it on a small scale. We've prototyped and run a "science angel" program at the Experiment Foundation for the past year. We give scientists a budget and let them fund their peers. 

I didn't see their paper until well after we had started. Otherwise, I would have more closely adhered to their proposed study. But here's what I know already:

• Funding, deciding, and betting on others is an orthogonal skill to doing science. Some scientists are good at this, others are bad at it, and most want nothing to do with it. 
  
  [Sidenote: Studying who is great at this is a worthwhile research project, similar to Phillip Tetlock's Good Judgement Project.]

• Peer-to-peer funding helps to uncover and kickstart projects that otherwise might get overlooked. It should be part of the research funding portfolio, but not the whole thing, as they propose. 

• There are a million logistical challenges. I could write a book on everything that came up when we tried to do this with 10 people and <$1M. They suggest this process would be less bureaucratically intensive and cheaper. I have little confidence that would be true. 

I'll do a longer write-up later, but our lessons aren't the point here. The goal is to highlight the disconnect between theory and practice. Science hasn’t been lacking in ideas to improve itself. At the end of all my interviews, I always ask: “what are your ideas to improve science?”

The ideas overflow. What’s harder to convey: just do it, start, prototype them.

At the earliest levels, scientists are trained to solve problems with analysis and data — with papers. That skillset is proving to be limited in enacting change. It lacks teeth. But that’s changing as more scientists leave academia for entrepreneurship, and more entrepreneurs see an opportunity in fixing science.

There are actually two adjacent metascience movements brewing: the academic analysis of scientific output and the outsiders who are experimenting with new institutional models. The academics wield their papers and focus on policies while the entrepreneurs are busy prototyping new ideas. 

There are surprisingly weak links between these two worlds. It might not seem that way. The two groups end up on the same podcasts and in the same Twitter threads, seemingly part of the same conversation, but they are not as connected as they could or should be. 

I doubt Jane Lubchenco or her co-authors have any idea what we're doing at Experiment — that we're implementing a version of their model. We have a tangible experience (as well as a scaleable software product) to bring to the policy table.

This is a missed opportunity. To move science forward, we need papers, policies, and prototypes working together. 

The prototypers are currently being propped up by a handful of brave philanthropists. We need more traditional philanthropists and federal agencies to engage with these new models. 

The window of opportunity is open. The coalescing momentum of both metascience scenes is poised to help improve the institutional foundations of science, but it will fall flat without a bridge between the different worlds. 

I'm optimistic about the new bridge-builders that are emerging: Institute for Progress, FAS' new Impetus Institute, and the Good Science Project. May we all seize the moment.