Alberto Cámara

github: @ber2

web: ber2.github.io

• Maths PhD

• Data Science & Engineering

• PyBcn member (join us!)

• Code mostly in Python & Scala

Today's goals

• Review the Archival Datathon Kaggle competition.
• Discuss how to be agile when approaching an ML problem.
• Learned lessons for doing ML in Production.

The Archival Problem

• Ran from May 7th to 13th, 2021
• We are given short text titles
  • Origin: OCR scans of titles of archive documents
  • Full of typos and spelling mistakes, not cleaned
• Documents have a topic classification label: QdC
• We are given a full training dataset and a portion of the validation dataset
• Optimize for F1-score

Agile approach to the problem

We need to be able to iterate fast

• Choice of weapons
• The data engineering bits need to be automatised away ASAP:
  • Load training data
  • Make a submission to Kaggle
• Aim at having a clear Machine Learning pipeline structure.

Choice of weapons

• YData profiling (formerly Pandas Profiling) for dataset exploration
• Git + DVC for code + data versioning
• MLflow for experiment tracking and model registry
• Jupyter Lab for running almost everything on notebooks

Pipeline structure

• Read training data into numpy arrays
• Feature engineering
• Model Selection & Fit Save to MLflow
• Submission of a given MLflow run to Kaggle

Critical: make your first submission ASAP

My first submission

• No text cleaning (but filtering nltk stopwords for Catalan)
• Vectorization: played with bag-of-words & tf-idf
• Model: logistic regression & linear SVM classifier
• Score on public testing dataset:

Kaggle competitions vs Business

• The rest of the time went by attempting to get to
• Many practices were attempted, sometimes running into serious overfitting
  • Feature engineering
  • Model fine-tuning

Feature Engineering

• Stopword removal
• Lemmatization
• Accent removal
• Spelling corrections
• Different vectorization methods
  • By unigrams and bi-grams
  • fastText word embeddings in Catalan, dim. 300

Model selection

• Wide range of scikit-learn models (spoiler: no model overtook LinearSVC)
• Comprehensive grid-searches for optimal parameters
• Neural networks on top of fastText embeddings

Chronology

• Day 1: Competition starts, first submission scores
• ~Day 3: Other competitors start making submissions
• Day 4: Toni Lozano challenges the leading position
• Day 5: Toni takes leadership
• Day 6: I overtake with the first submission over
• Day 7: Toni overtakes with his final submission

Result

• The winner was decided on a difference of score points
• Both scores went down on the private validation dataset due to overfitting

Thanks!

A few links:

• Kaggle Competition frontpage
• Code for the winning attempt
• DVC, MLflow, YData Profiling