The Liveability of Melbourne

About The Project

This is a group project for COMP90024 Cluster and Cloud Computing (Semester 1, 2022), The University of Melbourne.

YouTube videos: https://www.youtube.com/playlist?list=PLhue6Y7TCUD3a12XNekhmNJNRnn6a2gsQ

Team members:

• Xinyi Jin (Melody)
  • design the system and the role of each instance
  • set up the couchDB cluster
  • deploy the back-end and front-end application with Ansible and docker
  • perform sentimental analysis for the tweets
  • find related environment data from AURIN and BOM
  • contribute to report writing and video recording
• Yan Ying (Eliza)
  • design overall structure of Twitter harvesting
  • build the four Twitter harvesters
  • store processed tweets and AURIN datasets in CouchDB
  • design the structures of databases in CouchDB
  • perform sentimental analysis for the tweets
  • contribute to report writing and video recording
• Xinhao Hao (Budd)
  • front end design and implementation
  • Twitter data visualisation
  • scenario 3 and 4
  • prepare the live presentation
  • contribute to report writing and video recording
• Liqin Zhang
  • find appropriate data from AURIN to work on
  • visualisation on the AURIN and other data
  • contribute to report writing and video recording

(back to top)

Built With

• Melbourne Research Cloud
• AURIN
• Ansible
• Docker
• Twitter API
• Flask
• CouchDB
• nltk

(back to top)

Getting Started

Prerequisites

This is an example of how to list things you need to use the software and how to install them.

• Twitter developer account

  • list your twitter developer account tokens and named it as token_and_query.json . See the sample format in token_and_query.json

• MRC access

  • download the related cloud file and rename it as clouds.yaml. Put the file to host_vars. This file will be used by OpenStack tools as a source of configuration on how to connect to a cloud. See the example in clouds-example.yaml

  • get the OpenStack RC file for the project. For example, we are group 18 and our file is called unimelb-COMP90024-2022-grp-18-openrc.sh

Installation

Clone the repo

git clone https://github.com/MelodyJIN-Y/Liveability-of-Melbourne.git

(back to top)

Usage

System deployment

We use Ansible and Docker to configure and deploy the system. The foder Ansible contains all the deployment files. The entire system can be deployed with one single playbook entire_process.yaml and the file run-entire-process.sh

We also provide a step-by-step version to deploy the system for testing and illustration purpose. The deployment-intermediate-step folder contains breakdown steps to configure and deploy the system. We use dynamic inventory methods in the depoloyment thus no inventory files are provided. We need to run dynamic inventory in each intermediate deployment step to get the ip and group inforamtion.

• Option1: one-step deployment

  ./run-entire-process.sh

• Option2: step-by-step deployment

  1. launch and configure instance:

     realted playbook: s1-create-instances.yaml

     ./example-s1-run.sh

  2. set up CouchDB and CouchDB cluster

     realted playbook: s2-setup-couchdb.yaml

     ./example-s2-run.sh

  3. deploy backend applications

     realted playbook: s3-deploy-backend.yaml

     ./example-s3-run.sh

  4. deploy front-end applications

     realted playbook: s4-deploy-frontend.yaml

     ./example-s4-run.sh

Back-end

Functionalities of back-end multiprocessing programs are to to collect tweets, transmit AURIN data, and do sentiment analysis concurrently. Thus, under the "back-end" folder, there are three separate folders consist of their corresponding programs. Within each folder, execution can starts by using "python" command on the main program. The following commands show how to run main functions. Since the overall design of the back-end system is complexed, some more details are described in another README.md file under the "backend" folder.

Aurin folder:

store local AURIN data to CouchDB

python write_AURIN_couchdb.py

twitter folder:

• history data do pre-processes on historical tweets and store them to CouchDB

  python historical_tweets.py

• live_data set up four Twitter harvesters to collect tweets through Twitter APIs

1. essential

python EssentialAPI_process_search+stream.py

2. elevated

python ElevatedAPI_process_search+timeline.py

Sentiment folder

Based on tweet text databases, do sentiment analysis and get wordcloud on content of tweets.

python sentimental_analysis.py

Front-end

With Cisco connected, you can access the front end by visiting one of the following links:

• http://172.26.129.113/
  
• http://172.26.132.132/
  

The front end(Flask app) should run instantly on two of our instances. However, if it is down for any reason or for the purpose of local testing, using command line python3 app.py in directory /frontend should start the app.

(back to top)

License

Distributed under the GNU License. See LICENSE for more information.

(back to top)

Contact

Project Link: https://github.com/MelodyJIN-Y/Liveability-of-Melbourne

• Xinyi Jin (Melody): [email protected]
• Yan Ying (Eliza): [email protected]
• Xinhao Hao (Budd): [email protected]
• Liqin Zhang: [email protected]

(back to top)

Acknowledgments

• README template

(back to top)