Summary
Today we will discuss prediction markets and the problems they may help solve. We’ll hear a brief introduction of what Prediction Markets are by Robin Hanson, followed by a discussion of a few current projects (Metaculus, Augur, Replication Markets), and finish with a rundown of the challenges ahead by Chris Hibbert of Agoric and Martin Koeppelmann of Gnosis.
This meeting is part of the Intelligent Cooperation Group and accompanying book draft.
Presenters
Robin Hanson, George Mason University
Robin Hanson is associate professor of economics at George Mason University, and research associate at the Future of Humanity Institute of Oxford University. He has a doctorate in social science from California…
Anthony Aguirre, Metaculus
Physicist Anthony Aguirre studies the formation, nature, and evolution of the universe, focusing primarily on the model of eternal inflation—the idea that inflation goes on forever in some regions of universe—and what…
Thomas Pfeiffer, Replication Markets
Thomas Pfeiffer is an evolutionary biologist at Massey University in Auckland, New Zealand. He received a diploma in Biophysics from the Humboldt University Berlin and a doctoral degree in Environmental Sciences from the ETH Zurich. After a short period…
Chris Hibbert, Agoric
Chris is a software engineer who has worked on email security, financial cryptography, S/W development tools, highly social mobile games, and privacy. He wrote Zocalo, an open source platform for Prediction Markets. He has worked at…
Paul Gebheim, Augur
Paul Gebheim is a software architect, consultant and engineer based in northern California…
Presentation: Robin on Prediction Markets
- We frame argumentation in general as prediction. Many arguments can be ground.
- Let’s incentivize predictions transparently. Further, we want thse predictions people are making to be integrated into a consensus.
- It’s noticed that betting markets are kind of this. The bet is a transparent incentive based on a prediction, and the market is aggregating those predictions so others can se
- We should integrate this process inot regular organziation practice, so we need trials of this! I’m less interested in tools and platforms for this, we don’t really need them. Algorithms too.
- Less interested in curated prediction contests and more in a simple robust system for collecting bets.
- Less interested in information on interesting topics, more on actionable topics.
- We’ve seen organizations try these markets, and they work, but they are genearlly abaondoned for political reasons.
- Real organizations with real problems need to apply prediction markets to those problems not for the headlines but to actually solve the problem!
Presentation: Anthony on Metaculus and Non-market Prediction Aggregation
- There are three major ways we make predictions:
Presentation: Paul on Augur
- Augur has been approaching the problem that Robin invented years ago, but took a different approach. Instead of focusing on employing prediction markets in organizational settings, Augur set out to provide this system as a tool for the world. Its goal is to create ways to compensate those with information that is valuable but is otherwise not being compensated.
- Augur is built on the Ethereum network. There are a few very large problems that exist in creating a global distributed set of prediction markets.
- The Oracle Problem: we can create markets, and we can engage folks to predict or bet, but how do we resolve those markets to make sure results reflect the real world?
- Augur is attempting to solve this problem by creating a system called the Augur Oracle. People can report the outcome of a market and there is a set of bonds placed on disputing the outcomes. If outcomes aren’t clear, multiple “universes” are created where each outcome occured and let’s people bet on the versions of the world that they believe are the case.
- Now the main problem is; how do we create a good user experience for people to interact with the market. The issue is that it’s very expensive and slow to use right now.
Presentation: Thomas on Replication Markets
- Replication Markets is a bundle of projects that started around 2010, triggered by the replication crisis in science.
- This motivated a lot of fields to analyze this crisis, predominately in the behavioral sciences, to run large-scale replication studies in which many studies were colected and reproduced to see if they replicated.
- The folks behind Replication Markets were already working on prediction markets at the time, and these replication studies provided an opportunity to test whether prediction markets might be helpful in determining whether a particular study might replicate.
- They’ve run ~4 studies, with ~40 studies being bet on per trial by around 200 bettors. From these first projects, a paper was published showing that the markets are fairly predictive.
- They’re now thinking about what to do next: how to increase accuracy and scale, how to explore decision markets.
- DARPA Score is a large replication effort from DARPA, with 30,000 studies being annotated and 3000 being heavily annotated, with funding solicited to replicate the 3000.
- You can use a decision market to inform which studies should be attempted to replicate, with different strategies that give different information: attempt to replicate the ones predicted to NOT replicate, or TO replicate.
- Drop them an email to collaborate!
Presentation: Chris on Keynesian Beauty Contests
- Chris has worked on prediction markets for a long time, worked with Mark on Xanadu back in the day.
- Robin talked about some applications of prediction markets that he’d like to see.
- Marc Steigler has showed in his novels characters using prediction markets to make plans for highly variable situations: fog of war, adversarial action, new technology development.
- Chris’ point of view on the markets: when there is a question about an obscure subject with a surprisingly high probability of coming to account, that’s a signal that somebody might wanna take some action on.
- It’s the unusual situation in which Chris most cares about prediction markets coming up with the right answer, and that requires a decision criterion that isn’t subject to the whims of the crowd.
- The Keynesian Beauty Contest: when there are lots of different opinions about what had actually happened. Think about the last election: who wins the prediction market depends highly on how the question is asked.
- “Donald Trump will be inaugurated on January 20. Donald Trump will win the electoral college vote. Donald Trump will win the popular vote.” These are all different and hard to call at the same time. Some markets called winners in November, some could not.
- For laying off risk or making insurance, in unusual situations it matters a lot for the decision criteria to be very clear and concrete.
Presentation: Martin on Replication Markets
- Gnosis tried to acheive something similar to Augur: an open prediction market where people could bet cryptocurrency.
- It turns out, people were not that interested in it, so the Gnosis team shifted focus away from prediction markets and learned a few things along the way.
- In the pure market idea, you have the problem that prediciton markets are a zero sum game. For the most part, the only active betting markets are sports betting and maybe the US presidential election, everything else draws little attention.
- For zero-sum markets to work, you need people constantly losing money. To spur folks to really become experts at the probabilities of certain events, it requires a lot of money on the line to incentive that.
- Gnosis tried to fund markets with $5000 but was unsuccessful in inspiring folks to really put time into predicting well. Upping the markets size to $50,000 each was still not enough.
- Upon reflection, this isn’t too surprising: as a trader you could only really make a modest payout but a pretty high risk. It takes a very specific type of people that are long term thinking or have very high risk tolerance to do this kind of betting.
Q&A
Paul: Chris, what did you see on Augur?
Chris: Early on, the appeals process went to a wide panel of participants, which led to Keynesian Beauty Contests in which jurors were rewarded for voting with the majority.
Paul: This was never the whole system of appeals. How it works: when a market is being resolved, jurors stake on either side of the decision. If there is enough capital on both sides, then we split the market into two universes where the decision went either way. You can choose which universe to migrate your tokens to, and there are now two different tokens that have different supplies and trade separately.
Mark: Forking the universe into multiple universes is a bad outcome! It means the market didn’t do its job. We don’t have two different universes on a stock price. The missing mechanism is reputation feedback, where that feedback comes from is explicitly broadcasting who the judges will be and what criteria they will use to resolve the market.
Anthony: My experience with Metaculus is that well-constructed questions as Mark described above resolve with no problem. A very elaborate mechanism to treat the very small number of highly ambiguous questions seems to be avoded. How Metaculus handles this is to flag a question as ambiguous an refuse to attempt resolution.
Chris: It would be a real shame if the US election question was returned as “can’t answer because the public can’t decide.” It’s important to get fianl answers to these public questions.
Robin: When a particular customer is paying for the information it becomes very clear who the judges are and who gets to resolve. These ambiguity problems are problems in open public prediction markets.
Paul: For the record, we didn’t get anywhere close to a fork on the Biden-Trump election.
Anthony: Is there an advantage to a prediction market being decentralized?
Robin: I don’t see it.
Martin: It’s simple: I can start a market right now that’s globally accessible and anyone can bet. I don’t know how specifically useful that is.
Paul: The ability to access financial markets is not equally distributed. Decentralized systems democratize access.
Robin: The electronics give you that global access, the decentralized part just lets you get around financial laws, which is not clearly going to be allowed for long.
Mark: Robin ran the first prediction market at Xanadu in 1989 or 1990. There was a 30 year delay in clearing the markets because of the financial regulations around it. The decentrlized systems now are allowing us to do this without waiting for the rules.
Anthony: There is soon launching a prediction market in the US that does follow the rules.
Robin: The internal markets I’m interested in have never been illegal, just disruptive so not used much.
Robin: If you put out a new kind of job and you don’t really market well, people are reluctant to do that new kind of job. Your issue, Martin, was probably a marketing issue and the question of it taking time to introduce a new kind of labor market practice.
Martin: If there were billions to be made the market attracts participants, like sports betting and hedge funds. For less lucrative markets, it’s hard to get people to play.
Thomas: One of the problems is trying to achieve scientific consensus where prediction markets could help. How to get communities to use these? Working with a preprint server like arXiv could be an entry. The scientific community is great to work with on this because there is a lot of intrinsic self-motivation.
Robin: Can we get journal editors to use Replication Market? They would be the obvious people. If we can’t get them to care, then who cares??
Anthony: Yeah, the journals don’t care enough… The bottlenecks are actually doing the replications and aggregaring responses from peole.
Anna: In my experience the journals run by academics won’t say yes, but the for-profit journals might be more amenable.
Adam: It’s impossible to get referees for papers, so getting a market fir each is not gonna happen. But getting authors to self-report the odds of replication might help.
Thomas: But they are incentivized to just lie…
Anthony: One place to look is to the funders themselves to fund a market as part of their funding. They are incentivized in principle to fund replicable research. Adding it to the review process itself might help too.
Robin: Who are the customers of Metaculus?
Anthony: We have some paying customers.
Chris: The best example I know is Corning the glass company that funded a market on where the price for screens and glass would be at some point in the future.
Paul: A new example that might be interesting: in crypto world, there are so many projects with millions of dollars locked up in them. What we’re seeing is that part of the protocol operation could be worth paying the market for information as to how secure they are. This is mostly done through bug bounties. There is a project called Cover that offers “coverage” for asset loss on these protocols that operates mostly as insurance.
Chris: This sounds like an assassination market…
Paul: Everything is an assassination market! But the thing with decentralized systems is that there isn’t always anyone that you can kill to get that money.
Mark: It’s not the case that for moderate cost I can be invulnerable to assassination. It is the case that with reasonable investment you could build a secure protocol.
Robin: The key is how much could you make on the market vs how much you can make in the attack.
Mark: In the long run, successful attacks on insecure cryptosystems help the whole ecosystem. We’re building an ecosystem of the survivors of those attacks.
Seminar summary by James Risberg.
