Case study ·Fly-High Data Hackathon 2023

Winner · First place of 100

Predicting the price of every flight.

ProjectPro, a library of solved end-to-end data science projects, ran a five-week hackathon on airline fares: 45,000 priced flights to learn from, 5,000 to predict, and the lowest error wins. 100 participants entered. DataMinds took first place.

Talk to us about prediction

100

Participants

45k

Flights to learn from

Days to build

$600

First prize

Fourteen scored entries, ranked by root mean square error on the hidden test set. Official leaderboard at ProjectPro

Built with

Open-source Python, end to end.

The winning pipeline is scikit-learn, two boosting libraries and a notebook, and it runs top to bottom in one file. No cloud, no deep learning, nothing the judges could not read in an afternoon.

The result

First of 100, by three tenths of a point.

Entries were scored on the root mean square error of their predicted fares against the true fares of 5,000 held-back flights. The final DataMinds entry scored 12.11. Second place scored 12.41, third 13.57, and the field stretched past 260. The first prize went to DataMinds.

Fares in the data run from 173 to 1,262 and average 575, so a typical miss of twelve is about two per cent of a ticket. ProjectPro reserved the right to revise the ranking after reading the code and the two-page report. It stood, and the winning script was published on the hackathon page, beside the leaderboard.

Root mean square error · three-fold cross-validation

Stage	Hist	LightGBM	XGBoost
Default settings	14.20	13.97	15.84
Selected features	14.12	13.92	15.34
Tuned parameters	12.78	12.70	12.69

Ensemble of the three, cross-validation: 12.25
Leaderboard, hidden test set: 12.11

From the submitted script: 23 features at the start, ten or nine after selection, 100 random draws per model. Each model on its own feature subset; the ensemble is an equal-weight average of the three.

The challenge

One number per flight, and the lowest error wins.

ProjectPro publishes solved, end-to-end data science projects for people learning the trade, and in August 2023 it opened a hackathon to anyone who wanted to prove theirs. One dataset and one target: the fare of a flight, from the airline, the route, the times, the aircraft, the season, the demand, the weather and the price of fuel.

The rules were plain. One person per entry, no external data, Python or R. Submit the code, the predictions and a two-page report. The leaderboard ranked by error on the hidden test set, and ProjectPro kept the right to move entries after reading their code. So the code had to be as clean as the score was low.

The data

Two CSV files, one fare to predict.

Synthetic flight data with the shape of the real thing: three airlines named A, B and C, five aircraft types, invented city names, plausible fares, and blanks scattered through most columns.

45,000 flights

Training set

Eighteen attributes and the fare: airline, route, distance, departure and arrival times, duration, aircraft, stops, weekday, month, season, demand, weather, passengers, promotion and fuel price.

5,000 flights

Test set

The same columns without the fare. Predictions for these decided the leaderboard, and the true fares were never released.

11 of 19 columns

Missing values

The airline is blank in 3,573 rows, with gaps in the route, the calendar, demand, weather, promotion and fuel price. Gradient boosting handles them natively; nothing was imputed.

The catch

23,000

distinct departure cities among 45,000 flights.

As a category, the route is noise.

The pipeline reduced each name to its prefix, North, South, East, West, New, Port or Lake, and feature selection then dropped even that. Distance, demand, fuel price, duration and stops carried the signal instead.

How we built it

Three boosting models, one vote.

The pipeline is deliberately plain: hand-made features, three gradient boosting models each with its own feature subset and its own tuned parameters, and an equal-weight average. Every decision along the way was made on the same three folds, with the competition metric.

01
Encode everything by hand
Every text column became an integer: airline, aircraft, weekday, month, season, demand, weather, promotion. Times became an hour, a clock number and a day-or-night flag; the weekday gained a weekend flag; the month a holiday tier. City names were cut down to their prefix. Twenty-three features, no scaling, no imputation.
02
Three baselines on the same three folds
HistGradientBoosting, LightGBM and XGBoost at default settings, scored by root mean square error on three shuffled folds with a fixed seed. LightGBM led at 13.97, then Hist at 14.20, then XGBoost at 15.84. The folds never changed after this, so every later gain was a real one.
03
Forward selection on ranked features
Features were ranked by permutation importance for Hist and by split counts for LightGBM and XGBoost, then added one at a time in that order, with a full cross-validation at each step. Each model kept the subset with the best fold score: ten features for Hist and LightGBM, nine for XGBoost. The city prefixes did not survive.
04
A hundred random draws per model
Randomised search over learning rate, depth, leaves, trees and regularisation: 100 configurations per model, on its own feature subset, on the same folds. The three finished almost level, at 12.78, 12.70 and 12.69. LightGBM settled on GOSS boosting with 690 trees, XGBoost on 900 trees of depth four.
05
Average the three and submit
Each model wrapped its column selector and its parameters in a scikit-learn pipeline, and a VotingRegressor averaged the three with equal weights: 12.25 in cross-validation, 12.11 on the hidden test set. Refit on all 45,000 flights, predict the 5,000, write submission.csv.

ML_Pipeline_(v2).py · the ensemble

hist_pipeline = Pipeline([
    ('selector', ColumnTransformer([('selector', 'passthrough', hist_features_list)], remainder='drop')),
    ('hist_reg', HistGradientBoostingRegressor(**hist_best_params, random_state=42)),
])
lgb_pipeline = Pipeline([
    ('selector', ColumnTransformer([('selector', 'passthrough', lgb_features_list)], remainder='drop')),
    ('lgb_reg', lgb.LGBMRegressor(**lgb_best_params, random_state=42)),
])
xgb_pipeline = Pipeline([
    ('selector', ColumnTransformer([('selector', 'passthrough', xgb_features_list)], remainder='drop')),
    ('xgb_reg', xgb.XGBRegressor(**xgb_best_params, random_state=42)),
])

ensemble_pipeline = VotingRegressor(
    estimators=[
        ('hist_pipeline', hist_pipeline),
        ('lgb_pipeline', lgb_pipeline),
        ('xgb_pipeline', xgb_pipeline),
    ],
    weights=[1, 1, 1],
)

# Test CV Score: 12.25  <-- Best Model
ensemble_pipeline.fit(full_df[FEATURES], full_df[TARGET])
preds = ensemble_pipeline.predict(test_df[FEATURES])

Condensed from the submitted script. Three pipelines, one VotingRegressor, refit on every training row before predicting the test set.

The full pipeline, as submitted: one Python file with every experiment, every score and the parameters that won. ProjectPro publishes it beside the second and third place solutions, and we keep a copy here.

Python · 1,735 lines · 40 KB

Download the script Original at ProjectPro

Timeline

One month, start to submission.

5 Aug 2023 · Opens
Train and test sets released with the data dictionary. One month, one entry per person.
Aug 2023 · Build
Features, three-fold cross-validation, three boosting baselines. Feature selection and tuning, one model at a time.
5 Sep · Final submission
The fourth and last submission, three days before the deadline: the ensemble, version two of the pipeline.
8 Sep · Deadline
Submissions closed at 11:59 PM IST. ProjectPro published the ranking and the three winning scripts.

A price or a demand you need to predict?

Fares, orders, load, churn. If a wrong number costs you, we build models that are validated the way the leaderboard validates them, on data they have never seen. First results in weeks.

See forecasting and prediction Get in touch