Just programming a basic web crawler system with Go, mongoDB, Docker and MongoDB-Atlas for prod.
Describing the system as a whole unit.
Describing the activity/flow of the crawling use case.
Describing the behaviour of the crawling use case.
See the CMD Results with GOMAXPROCS set to 8 here

See the CMD Results with GOMAXPROCS set to 8 here

I've done a full-text search production test to mimic the flow of a real search engine via MongoDB Atlas Search Tester UI.
- Being able to design and program a crawler from scratch.
- Leveraging Go's productive ecosystem.
- Leveraging Go's concurrency patterns to speed-up crawling by 86%.
- Interacting with a local MongoDB instance for test (Docker containers).
- Interacting with a remote MongoDB instance for prod (MongoDB Atlas).
- Splitting application environments (Docker compose profiling).
- Using UML diagrams.
- Being able to collect a lot of metrics.
- Not able to implement Politeness feature for the URL Frontier.
- Not able to implement Priority feature for the URL Frontier.
- Limited control over data/UI noise.
