Glossary of Terms

This glossary provides definitions for key terms and components used in Nkululeko.

Core Components

  1. Experiment: Central class orchestrating data loading, feature extraction, and model training (nkululeko/experiment.py)

  2. Dataset: Base class for loading and managing audio datasets from different sources (nkululeko/data/dataset.py)

  3. Dataset_CSV: Class handling CSV-based datasets with audio paths and labels (nkululeko/data/dataset_csv.py)

  4. FeatureExtractor: Orchestrates feature extraction using different extraction methods (nkululeko/feature_extractor.py)

  5. Featureset: Abstract base class for all feature extraction implementations (nkululeko/feat_extract/featureset.py)

Feature Extraction Methods

  1. Opensmileset: Extracts features using OpenSMILE toolkit (nkululeko/feat_extract/feats_opensmile.py)

  2. Wav2vec2: Extracts embeddings using Facebook’s wav2vec2 model (nkululeko/feat_extract/feats_wav2vec2.py)

  3. PraatSet: Extracts acoustic features using Praat speech analysis software (nkululeko/feat_extract/feats_praat.py)

  4. TRILLset: Extracts TRILL (Google’s Transfer Representation for IL) embeddings (nkululeko/feat_extract/feats_trill.py)

  5. AudmodelSet: Extracts embeddings from a wav2vec2-based emotion model (nkululeko/feat_extract/feats_audmodel.py)

Experiment Management

  1. Runmanager: Manages multiple runs of an experiment, collecting and comparing results (nkululeko/runmanager.py)

  2. Modelrunner: Manages training and evaluation for a single run with multiple epochs (nkululeko/modelrunner.py)

Models

  1. Model: Base class for all machine learning models (nkululeko/models/model.py)

  2. TunedModel: Fine-tunes pre-trained transformer models like Wav2Vec2 (nkululeko/models/model_tuned.py)

  3. MLPModel: Multilayer perceptron model for classification tasks (nkululeko/models/model_mlp.py)

  4. SVM_model: Support Vector Machine model implementation (nkululeko/models/model_svm.py)

  5. XGB_model: XGBoost model implementation (nkululeko/models/model_xgb.py)

  6. GMM_model: Gaussian Mixture Model implementation (nkululeko/models/model_gmm.py)

Reporting and Analysis

  1. Reporter: Generates evaluation metrics, plots, and reports for experiments (nkululeko/reporting/reporter.py)

  • Supports multiple metrics: UAR (Unweighted Average Recall), EER (Equal Error Rate), MSE, MAE, CCC

  • See EER Implementation for details on the Equal Error Rate metric

  1. ensemble_predictions: Function combining predictions from multiple models (nkululeko/ensemble.py)

Audio Processing

  1. Silero_segmenter: Segments audio using Silero Voice Activity Detection (nkululeko/segmenting/seg_silero.py)

  2. Resampler: Handles audio resampling to a specified sample rate (nkululeko/augmenting/resampler.py)

Prediction and Demonstration

  1. TestPredictor: Handles prediction and result storage for test datasets (nkululeko/test_predictor.py)

  2. Demo_predictor: Handles real-time or file-based demonstration of trained models (nkululeko/demo_predictor.py)

  3. FeatureAnalyser: Analyzes feature importance using SHAP and other methods (nkululeko/feat_extract/feats_analyser.py)