Get Started. It's Free
or sign up with your email address
Rocket clouds
GetBooks by Mind Map: GetBooks

1. List of sources (free or not)

2. Each source has associated struct to find entries

3. Each source is inspected one by one or in parallel?

4. Each source at the same time should be processed only by one process

4.1. Lock by source

5. Source can be braked to pages (one process only?), where each page contains multiple entries

5.1. Each entry can be processed separately and independently

6. Jobs

6.1. General part and specific

6.2. E.g. BookLoadingJob

6.3. Let's model it in Mongo

6.4. It's better to have progress report

7. Fault-tolerant

7.1. BookLoadingJob?

7.2. PageLoadingJob?

7.3. SourceRefreshJob?

7.4. retryability, elasticity, monitoring, ...