Note: the current releases of this toolbox are a beta release, to test working with Haskell's, Python's, and R's code repositories.
Metrics provides implementations of various supervised machine learning evaluation metrics in the following languages:
- Python
easy_install ml_metrics - R
install.packages("Metrics")from the R prompt - Haskell
cabal install Metrics - MATLAB / Octave (clone the repo & run setup from the MATLAB command line)
For more detailed installation instructions, see the README for each implementation.
EVALUATION METRICS
Evaluation Metric
Python
R
Haskell
MATLAB / Octave
Absolute Error (AE)
✓
✓
✓
✓
Average Precision at K (APK, AP@K)
✓
✓
✓
✓
Area Under the ROC (AUC)
✓
✓
✓
✓
Classification Error (CE)
✓
✓
✓
✓
F1 Score (F1)
✓
Gini
✓
Levenshtein
✓
✓
✓
Log Loss (LL)
✓
✓
✓
✓
Mean Log Loss (LogLoss)
✓
✓
✓
✓
Mean Absolute Error (MAE)
✓
✓
✓
✓
Mean Average Precision at K (MAPK, MAP@K)
✓
✓
✓
✓
Mean Quadratic Weighted Kappa
✓
✓
✓
Mean Squared Error (MSE)
✓
✓
✓
✓
Mean Squared Log Error (MSLE)
✓
✓
✓
✓
Normalized Gini
✓
Quadratic Weighted Kappa
✓
✓
✓
Relative Absolute Error (RAE)
✓
Root Mean Squared Error (RMSE)
✓
✓
✓
✓
Relative Squared Error (RSE)
✓
Root Relative Squared Error (RRSE)
✓
Root Mean Squared Log Error (RMSLE)
✓
✓
✓
✓
Squared Error (SE)
✓
✓
✓
✓
Squared Log Error (SLE)
✓
✓
✓
✓
TO IMPLEMENT
- F1 score
- Multiclass log loss
- Lift
- Average Precision for binary classification
- precision / recall break-even point
- cross-entropy
- True Pos / False Pos / True Neg / False Neg rates
- precision / recall / sensitivity / specificity
- mutual information
HIGHER LEVEL TRANSFORMATIONS TO HANDLE
- GroupBy / Reduce
- Weight individual samples or groups
PROPERTIES METRICS CAN HAVE
(Nonexhaustive and to be added in the future)
- Min or Max (optimize through minimization or maximization)
- Binary Classification
- Scores predicted class labels
- Scores predicted ranking (most likely to least likely for being in one class)
- Scores predicted probabilities
- Multiclass Classification
- Scores predicted class labels
- Scores predicted probabilities
- Regression
- Discrete Rater Comparison (confusion matrix)