Case study · Numerai Tournament, since 2020

Top 10 of 15,000 models · Expert tier

Top 0.1% in the hardest data science tournament in the world.

Numerai, a hedge fund in San Francisco, gives away its market data in obfuscated form and lets anyone train models on it. The models are combined into one Meta-Model that trades the global stock market, and participants stake the NMR cryptocurrency on their own predictions. We build top-performing models for it.

Talk to us about prediction

Top 10

Rank, of 15,000 models

Weekly rounds in profit

Models in production

120%

Annualised return

On Numerai

The thesis

Financial Machine Learning, Cryptocurrencies, and Crowdsourced Investing.

A deep dive into the Numerai competition, in 139 pages. The hedge fund, the DAO and the NMR token; the dataset, the metrics and the staking mechanics; the modelling methods that work on it; and the results of the models built along the way. Written because the material online was scattered and out of date, and made public for anyone who wants to compete.

Contents

01Introduction
02Blockchain and key terminology
03Applied machine learning
04Numerai: the fund, the DAO and NMR
05The Numerai dataset
06Evaluation metrics: correlation, MMC, TC
07Staking, scoring and payouts
08Competing: machine learning modelling
09Results and conclusions

The title page of the thesis: Financial Machine Learning, Cryptocurrencies, and Crowdsourced Investing, a deep dive into the Numerai competition, Aristotle University of Thessaloniki, September 2022

The master's thesis for the Data and Web Science MSc at the Department of Computer Science, Aristotle University of Thessaloniki, September 2022. Supervised by Dr Grigorios Tsoumakas. The degree closed at 9.14 out of 10.

PDF · 139 pages · 9 MB

Download the thesis Open in a new tab

The result

Top ten of 15,000 models, forty weekly rounds in profit.

The models built after the thesis, on the same framework, climbed into the top ten of the leaderboard and held it, out of about 15,000 models submitted each round. The portfolio ran forty consecutive weekly rounds in profit.

It began with the thesis. Twenty-four models went into production and were scored on live market data for twenty consecutive rounds. Every one finished with a positive correlation and a positive contribution to the Meta-Model, nine never had a losing round, and the random model submitted beside them as a control ended at zero. The best of them, PAPAEMMAN9, reached 43rd by true contribution; PAPAEMMAN17 sat in the top 8% by correlation.

Stakes are in NMR and payouts follow performance. Weekly returns ran between 0.1% and 3%, one round in twenty burned, and the three-month return on stake stood at 57.76% in September 2022. Between them the models collected three gold, 32 silver and 102 bronze medals.

The PAPAEMMAN9 model page on Numerai: true contribution rank 43, a 17.1% three-month return on stake, and the performance chart of recent rounds — From the thesis: the PAPAEMMAN9 model page at rank 43, and the cumulative correlation and contribution of all 24 production models.

Thesis figure 9.1: cumulative correlation score of the 24 production models, every line rising — From the thesis: the PAPAEMMAN9 model page at rank 43, and the cumulative correlation and contribution of all 24 production models.

The challenge

Five million rows, 1,200 features, almost no signal.

Numerai's data is hedge-fund quality and deliberately unreadable. Every row is a stock at a point in time, an era. The 1,200 features are obfuscated and binned to five values, the target is an abstract measure of performance some weeks ahead, and nobody is told what any of it means. A correlation of 0.035 with the target makes a top-8% model.

Rounds run weekly. You submit predictions for live stocks, wait four weeks for them to resolve, and are paid or burned in NMR on how much your model contributed to the Meta-Model. A model that fits the training data and not the market loses money in public. That is the test.

The ambition is larger than a tournament. In October 2020 Numerai's founder set out the plan to build the last hedge fund: one Meta-Model fed by every data scientist willing to contribute, and Numerai Signals for those who bring their own data.

The data

A stock, an era, and 1,200 numbers you cannot read.

Numerai releases a new version of the dataset every few years and new live rows every round. Everything is obfuscated. The model has to find structure without ever knowing what a feature is.

1,200

Features

Quantitative attributes of a stock at a point in time, obfuscated and binned to 0, 0.25, 0.5, 0.75 and 1. Grouped into families whose meaning nobody outside Numerai knows.

One per week

Eras

An era is one point in time and every stock in it. Validation has to be cut by era, never by row, or the model learns the calendar instead of the market.

Several

Targets

An abstract measure of return a fixed number of weeks ahead, with alternative targets for other horizons and different neutralisation.

Every round

Live data

New rows for the current week, unlabelled. Predictions are scored against the market for four weeks before the round resolves and the stake is paid or burned.

The bar

0.035

correlation with the target. That is what a top-8% model achieves. The signal is small, and the ranking is decided at the third decimal.

Five million rows do not fit on a laptop. The work moved to a cloud machine with 20 cores and 160 GB of memory, switched off between sessions, for about $150 in total.

How we built it

Validate by era. Control exposure. Then bet.

The models were tree ensembles on tabular data, and the model was never the point. The point was an evaluation that predicts live performance, features that do not overfit one market regime, and a portfolio that stakes more where the risk is lower.

Four model cards from the Numerai account: PAPAEMMAN1, PAPAEMMAN7, PAPAEMMAN9 and the RANDOM33 control, each with its wake date and medals — Production models on the Numerai account, from the thesis. RANDOM33 is the control.

01
Frame the problem three ways
Regression on the target, classification on its five bins, ordinal in between. Each framing was scored on the same era-wise validation, and the choice was made on tournament metrics, not on training loss.
02
An evaluation that behaves like the tournament
Cross-validation by era, never by row. Per-era correlation, its Sharpe ratio, the maximum drawdown and the feature exposure, the same numbers Numerai's diagnostics report. A model was kept only if this framework said so.
03
Keep feature exposure down
A model that leans on a few features wins in one regime and burns in the next. Regularisation, feature neutralisation and feature selection kept exposure low, at a small cost in correlation and a large gain in consistency.
04
Ensembles, era selection and retraining
LightGBM and other tree ensembles trained on different targets, feature sets, and hard or easy eras, then combined. Hyperparameters tuned against the era metrics. Retraining scheduled as new data arrived.
05
Twenty-four models and a staking portfolio
Numerai allows many models per account, so 24 went live, with one random model as a control. Stake multipliers followed each model's count of burning rounds, and the stake amount followed its Sharpe ratio and relative rank. More money where the risk was lower.

The weekly round runs itself. An Azure Function wakes every Sunday, downloads the live data through the Numerai API, runs every model, uploads the predictions and sends an email when it is done, on the free tier. The write-up is on Medium: How I automated my Numerai weekly submission pipeline for free, using Azure Functions and Python.

The talk

Why Numerai matters, in 37 slides.

An introduction for people who have never heard of it. What Numerai is and why it calls itself the hardest data science tournament on the planet; how the data scientists, the tournament and the hedge fund fit together; how NMR pays them; and where its founder is steering it.

PDF · 37 slides · 2.6 MB

Download the slides Open in a new tab

Slide 1 of 37, Why Numerai matters — 01 / 37

Slide 2 of 37, Why Numerai matters — 02 / 37

Slide 3 of 37, Why Numerai matters — 03 / 37

Slide 4 of 37, Why Numerai matters — 04 / 37

Slide 5 of 37, Why Numerai matters — 05 / 37

Slide 6 of 37, Why Numerai matters — 06 / 37

Slide 7 of 37, Why Numerai matters — 07 / 37

Slide 8 of 37, Why Numerai matters — 08 / 37

Slide 9 of 37, Why Numerai matters — 09 / 37

Slide 10 of 37, Why Numerai matters — 10 / 37

Slide 11 of 37, Why Numerai matters — 11 / 37

Slide 12 of 37, Why Numerai matters — 12 / 37

Slide 13 of 37, Why Numerai matters — 13 / 37

Slide 14 of 37, Why Numerai matters — 14 / 37

Slide 15 of 37, Why Numerai matters — 15 / 37

Slide 16 of 37, Why Numerai matters — 16 / 37

Slide 17 of 37, Why Numerai matters — 17 / 37

Slide 18 of 37, Why Numerai matters — 18 / 37

Slide 19 of 37, Why Numerai matters — 19 / 37

Slide 20 of 37, Why Numerai matters — 20 / 37

Slide 21 of 37, Why Numerai matters — 21 / 37

Slide 22 of 37, Why Numerai matters — 22 / 37

Slide 23 of 37, Why Numerai matters — 23 / 37

Slide 24 of 37, Why Numerai matters — 24 / 37

Slide 25 of 37, Why Numerai matters — 25 / 37

Slide 26 of 37, Why Numerai matters — 26 / 37

Slide 27 of 37, Why Numerai matters — 27 / 37

Slide 28 of 37, Why Numerai matters — 28 / 37

Slide 29 of 37, Why Numerai matters — 29 / 37

Slide 30 of 37, Why Numerai matters — 30 / 37

Slide 31 of 37, Why Numerai matters — 31 / 37

Slide 32 of 37, Why Numerai matters — 32 / 37

Slide 33 of 37, Why Numerai matters — 33 / 37

Slide 34 of 37, Why Numerai matters — 34 / 37

Slide 35 of 37, Why Numerai matters — 35 / 37

Slide 36 of 37, Why Numerai matters — 36 / 37

Slide 37 of 37, Why Numerai matters — 37 / 37

The meetup

Numerai, explained to a room in Thessaloniki.

The talk was given on 16 November 2022 at AI & Beers, a networking meetup for the data science and AI community in Thessaloniki. What Numerai is, how the tournament works, and what it takes to stay near the top of it.

Panagiotis Papaemmanouil presenting at the AI & Beers meetup, the intro slide on screen with the Numerai and EYDAP logos — AI & Beers, Thessaloniki, 16 November 2022. The Numerai talk and the room. Covered byThessToday.

Panagiotis Papaemmanouil in front of the Numerai title slide before the talk — AI & Beers, Thessaloniki, 16 November 2022. The Numerai talk and the room. Covered byThessToday.

Timeline

From the first model to the top 0.1%.

Jul 2020
The first model wakes. Machine learning knowledge, by his own account, still thin.
Apr 2021
The first public post about Numerai, on LinkedIn. Models submitted every week since.
Dec 2021
Thesis proposal accepted at the Aristotle University of Thessaloniki: financial machine learning, cryptocurrencies and crowdsourced investing, through Numerai.
May 2022
The Azure workspace, the era-wise evaluation framework and the production models. PAPAEMMAN9 wakes on 8 May.
Sep 2022
Thesis submitted. Twenty rounds in profit, PAPAEMMAN9 at 43rd, three-month return on stake 57.76%.
Nov 2022
The result goes public on LinkedIn, top 100 of the leaderboard, and the talk is given: Why Numerai matters, at AI & Beers in Thessaloniki on 16 November.
2023
Models built on the same framework enter the top ten of the leaderboard and stay there. Forty weekly rounds in profit.
Today
Expert tier on Numerai. The account is still scored every round.

A prediction problem with real money on it?

Demand, prices, risk, churn. If a wrong forecast costs you, we build models that are validated the way the market validates them, with the uncertainty shown. First results in weeks.

See forecasting and prediction Get in touch