New Twitter Infrastructure

Get Started. It's Free
or sign up with your email address
Rocket clouds
New Twitter Infrastructure by Mind Map: New Twitter Infrastructure

1. Code

1.1. Producer

1.1.1. Manage Twitter Stream (connection, error logging, ...)

1.1.2. Manage RabbitMQ (connection, error logging)

1.1.3. Transfer serialized tweets to the queue

1.1.4. Test suite

1.1.5. Documentation

1.2. Consumer

1.2.1. Manage RabbitMQ (connection, error logging)

1.2.2. Manage DB-agnostic layer (connection, error logging)

1.2.3. Deserialize and process tweets

1.2.4. Store tweet to DB-agnostic interface

1.2.4.1. MySQL implementation

1.2.4.2. NoSQL implementation

1.2.5. Test suite

1.2.6. Documentation

2. Prerequisites

2.1. Environment setup

2.1.1. RabbitMQ

2.1.2. WAMP

2.1.3. Twitter account and api key

2.1.4. Git/GitHub

2.2. Backup MySQL (185G)

2.3. RabbitMQ learning

2.4. Study NoSQL alternatives

2.4.1. Cassandra

2.4.2. MongoDB

2.4.3. Voldemort

3. Backup

3.1. Create regular data Backup from the Database

3.2. How often ?

3.3. Where to store it ?

3.4. Which format ?

4. Server Configuration

4.1. Launch at startup

4.1.1. RabbitMQ

4.1.2. Artifactory

4.1.3. MySQL

5. Continuous Integration

5.1. Jenkins

5.1.1. Installation

5.1.2. Builds setup