Homework Help UK

Assessment Title : Individual Project Development
Brunel University London

MAIN OBJECTIVE OF THE ASSESSMENT In this assessment, you are required to demonstrate the appropriate practical skills and abilities to implement solutions using modern large-scale data storage and processing infrastructures, and to critically reflect on the concepts, theory and use of high performance computational infrastructures.

DESCRIPTION OF THE ASSESSMENT You are required to identify and analyse a real-world problem, design and implement a solution to the problem using Hadoop, and evaluate your implementation. The problem can be a simplified version from its original scale, extent or level of difficulties etc. An indicative list of sample problems have been provided at the end of this document. You may choose one of the problems in the list, but you are encouraged to identify your own problem for the project.

The assessment has two weighted components:

1. Oral presentation (20%). A workshop will be held near the end of the term. Each candidate will be allocated with 10 minutes (including question time) to present their individual project development and demonstrate, if any, your prototype software. You should take this as an opportunity to seek feedback and improve your project for the final submission.
2. Report (80%). A written report including the theory behind and the development of the individual project needs to be submitted.

LEARNING OUTCOMES AND MARKING CRITERIA

Learning Outcomes:

LO1: Demonstrate the appropriate practical skills/abilities required to implement solutions using modern large-scale data storage and processing infrastructures.

LO2: Reflect critically on the concepts, theory and appropriate use of large-scale data storage and processing infrastructures (commonly used in modern organisational environments).

Marking Criteria:

The coursework will be marked for 4 main criteria:

1. Demonstrating an understanding of the relevant theory underpinning distributed file systems & data analysis (LO2)
2. Identifying a real data analytics problem with strong motivation for using distributed processing methods (LO1)
3. Implementing and applying a working solution using distributed analytical techniques (LO1)
4. Critically evaluating the results of the implementation on the data with a discussion of how the approach is different from standard non-distributed methods (e.g. relational databased, serial data-mining) (LO2)

Grade Band E and F (E+, E, E-, F) The candidate fails to meet the minimum requirements as outlined in the learning outcomes.

Grade Band D (D+, D, D-) The work demonstrates significant weaknesses, but all of the learning outcomes have been met at the minimum requirement level. The work provides evidence of some critical understanding of the concepts and theories of large-scale data storage and processing infrastructures, and demonstrates some abilities and skills to implement solutions using these technologies.

Grade Band C (C+, C, C-) In addition to the requirements for a grade in D-band, the work demonstrates a critical and substantial understanding of the concepts and theories of large-scale data storage and processing infrastructures. It demonstrates the ability to develop an independent, systematic, logical and effective solution to the problems identified. It also demonstrates a significant degree of competence in the appropriate use of the relevant literature, theory, methodologies, practices, and tools, etc., to analyse the problems and evaluate the solutions.

Grade Band B (B+, B, B-) In addition to the requirements for a grade in C-band, the work clearly demonstrates a well-developed, critical and substantial understanding of the concepts and theories of large-scale data storage and processing infrastructures. It clearly demonstrates the ability to develop an independent, systematic, logical and effective solution to the problems identified. It also demonstrates a high degree of competence in the appropriate use of the relevant literature, theory, methodologies, practices, and tools, etc., to analyse the problems and evaluate the solutions.

Grade Band A (A*, A+, A, A-) In addition to the requirements for a grade in B-band, the work clearly demonstrates a sophisticated, critical and thorough understanding of the concepts and theories of large-scale data storage and processing infrastructures. It provides evidence of originality of thought and clearly demonstrates the ability to develop an independent, systematic, logical and effective solution to the problems identified. It also demonstrates excellence in the appropriate use of the relevant literature, theory, methodologies, practices, and tools, etc., to analyse the problems and evaluate the solutions.

FORMAT OF THE ASSESSMENT

There is no word/page limit for this assessment, but the best effort should be made to ensure the submission is as concise as possible. You should include sections on (percentage of overall mark):

• Introduction (10%) - criteria 1
• Problem description & associated dataset (15%) -criteria 2
• Design & Implementation (20%) - criteria 3
• Results (20%) - criteria 4
• Conclusions (15%) - criteria 1

Indicative Coursework Topics

In this assessment, you are required to identify and analyse a real-world problem, design and implement a solution to the problem using Hadoop, and evaluate your implementation. The problem can be a simplified version from its original scale, extent or level of difficulties etc. Please refer to the official assessment specifications for the objectives, descriptions, marking criteria, format and submission requirement of the coursework.

An indicative list of sample problems is given as below. You may choose one of the problems in the list, but it would be better, and you are most encouraged, to identify your own problem and provide your own solutions for the coursework.

1. Word counting. We have used word counting as a "Hello World" problem in the lectures and labs. However, there is still space to extend the problem, for example, the dealing of upper/lower cases, punctuation marks, top N frequent words, cooccurrence words etc.

2. Scientific data analysis. We have used the UK weather data as an example in the lab. You may extend this application by developing tools to provide more in-depth analysis of weather and climate.

3. Image conversion. The story of converting millions of image documents from TIFF to PDF by the New York Times has been a highlight of Hadoop. Technically, it is not a very complicated task. Perhaps you will have a try.

4. Network traffic analysis. Take a web server log file, and write a program to analyse the traffic to the server, for example, the number of visit for each IP address per unit time, the top N visitors etc, for a starting point.

5. Monte Carlo simulation. Estimation of pi (the ratio of a circle's circumference to its diameter) using the Monte Carlo method.

6. Social Media Analytics. Pete Warden’s infamous story with Facebook has caught the eye of people from both within and outside the data science community. Regardless your opinion on this particular case, the power of big data technologies has been demonstrated to a great extent. You do not have to get yourself into the troubles like Pete Warden did, but certainly you can use the legitimate approaches to explore the hidden values of social media such as capturing consumer attitudes, managing online reputation, anticipating customer needs and making recommendations, etc.

High Performance Distributed Processing & Analysis – A Case Study on XXXXX Data

(Presentation - 20%) Introduction (10%)

Brief presentation of distributed methods to manage and analyse data and how they differ from traditional approaches

The Problem and Associated Dataset (15%)

• Motivation of problem
• What is the data?
• Why does the data need to be analysed?
• Why does it suit distributed methods?
• How will analysing the data solve the associated problem?

Design & Implementation (20%)

• What are the detailed characteristics of the data (use summary statistics to explain)?
• What algorithms exist for solving such a problem – distributed or not
• How can MapReduce be used to solve the problem?
• What are the details of your implementation – annotated code and description of experiments

Results (20%)

• Documentation of the execution of the code, e.g. time efficiency
• Documentation of the results, e.g. charts, graphs, sample outputs
• What do the results tell you about the use of MapReduce compared to other techniques?

Conclusions (15%)

• Discussion of the results: Were they a success?
• If not why not? If they were, how could they be improved?
• What would you have done differently with hindsight?
• What other approaches / algorithms / implementations could be explored beyond MapReduce?