NIT 6130 Introduction to Research

Assignment – 4

Experiment Design & Result Analysis

Big Data Predictive Analytics to Overcome Flight Delays

Masters in Applied Information Technology (NMIT)
Victoria University, Melbourne, Victoria

Table of Contents

1. Collection of data for experiment

1a. Identification and selection of available data sources

1b. Collection of Raw Data

2. Experiment Design and Implementation

2a. Data pre-processing

2b. Feature Selection or Dimension Reduction

2c. Experiment Design

2d. Experiment Implementation Records

3. Experiment Result Analysis and Summary

4. Outline of Experiment and Result Analysis Chapter

1. Collection of data for experiment

1a. Identification and selection of available data sources

In order to conduct experiment analysis, the available data sources are analysed and collected. The following table gives a brief description of the available data sources.

1b. Collection of Raw Data

The relevant data for the experimental purpose is downloaded from the web and saved in a folder called ‘Raw Data’. These files are in the Microsoft Excel (*.csv) format. The details about these records are summarised in the given table.

2. Experiment Design and Implementation

2a. Data pre-processing

Huge amount of raw data is available for the research experiment. All this data cannot be utilised for the experimentation. Therefore, this collection of data needs to be pre-processed to conduct the experiment.

2b. Feature Selection or Dimension Reduction

The entire data collection files consist of multiple data features. Not all of them are relevant to the experimental process. So, few fields have been eliminated from the existing records and new files are updated accordingly. The dimensionality of the collected data is reduced in order to simplify data processing during experiment analysis. The new result data set are recorded in the following sample table.

2c. Experiment Design

2d. Experiment Implementation Records

A basic and simple delay model can be built with the help of Empirical Cumulative Distribution Model. The Kernel Density Estimation method is a basic function of the programming language that will be used. A Map-Reduce algorithm will be used that will split the input data set into individual chunks which will be processed be the map tasks in a completely parallel manner. The Linear Regression Model of the average daily delay analyzes the effects of arrival delay, airport capacity, traffic congestion and weather conditions.

3. Experiment Result Analysis and Summary

After conducting the aforementioned experiments, there are certain results that are desired to be obtained. They are analysed as below –

• Flight delays are one of the major causes of Total Passenger Trip Delay (TPTD). Other passenger trip delays are due to either missed connections or flight cancellations.
• Airline network design also has a significant impact on the trip delay of passengers. The gap between direct and connected itineraries, frequency of the service, time wasted between banks at the hubs, aircraft size selection and target load factor also play a major role to determine the trip reliability of passengers.
• Flight delay caused due to bad weather conditions should be forecasted much before the scheduled flights so that necessary alternative arrangements can be made. The passengers can also be advised in advance about the future delay in their trip so that they can also plan their journey accordingly. This will reduce chaos among the passengers.
• The delays that passengers experience because of the late or diverted flights can be minimised. The passengers are affected by the trip delay because of the cancelled flights, missed connections or boarding issues. To avoid such situations a new flight can be implemented in order to avoid delay in further flights. But this will increase the cost to the airlines effectively. If the frequency of the flight decreases, then the load factor of the re-booked flights increases. The experimental result can minimize the trip delay of passengers by either rescheduling passengers on the next flight that were late from connecting flights or by holding the next flight until the passenger arrives. The second case may eventually delay all the other passengers and their connecting flights.

4. Outline of Experiment and Result Analysis Chapter

4.1 Data Analysis

4.1.1 Data Pre-processing and Transformation

4.1.2 Target Data Creation

4.1.3 Model descriptions and variables

4.1.3.1 The training dataset

4.1.3.2 Decision Trees

4.1.3.3 Random Forest Model

4.2 Delay Prediction

4.2.1 Classification Technique

4.2.2 Hadoop MapReduce

4.3 Analysis of Variance (ANOVA) on the average daily arrival delay

4.3.1 ANOVA test on seasonal pattern

4.4 BRYAGH: Basic Reduction Yare Approach for Flights

4.4.1 BRYAGH Algorithm

4.4.2 Pseudo code

4.5 Conclusion