Train Tonnage Anomaly Detection

Published: Mar 16, 2024 by

Prerequisites

• Configure RHODS workspace has been completed.

Run Jupyter notebooks

1. Accessing an AWS S3 bucket & downloading multiple JSON files - advanceddownloadData.ipynb

Data Collection Change the following to notebook

AWS_ACCESS_KEY_ID = 'YOUR_ACCESS_KEY'
AWS_SECRET_ACCESS_KEY = 'YOUR_SECRET_KEY'
bucket_name = 'edge-anomaly-detection-bucket-name'
# List of file names to download
file_names = [
    'edge-datalake-bullet--Wed Oct 04 13:56:26 GMT 2023.txt',
    'edge-datalake-bullet--Wed Oct 04 13:57:26 GMT 2023.txt',
    # Add more file names here if needed
]

Example Bucket contents

Train Tonnage Anomaly Detection Model - TrainTonnage.ipynb

Data Exploration Preprocess the data (remove rows with missing values)

Make an Isolation Forest Model

Show Anomalies

Check the Correlations in the Data

Check fo Missing Data in the Dataset

Detect Negative Values in the ‘TrainSpeed’ Column

Print the Anomaly Scores and Correlation Matrix

Train Tonnage Over Time with Anomalies Highlighted

List Detected Anomalies in Train Tonnage Data

Visualize of All Dataframe Columns

Visualize All Anomalies Dataframe Columns

Correlation Heatmap

Review Box Plot of Key Features

Pairwise Scatter Plot of Key Features

Scatter Plot of Primary Suspension Stiffness vs. Train Acceleration

Scatter Plot of Train Tonnage vs. Elapsed Time

Scatter Plot of Train Tonnage vs. Anomaly Scores

Scatter Plot of Anomaly vs. Anomaly Scores

Scatter Plot of TrainSpeed vs. TrainAcceleration with Correlation Line

Model Conversion to ONNX Format

Load an ONNX Model for Inference

List Features in a Sample DataFrame

Feature Extraction and Inference

Inference using ONNX Model

Visualizing Model Output with a Bar Chart

Visualizing Anomaly Scores with a Bar Chart