I spend the few last days working on he database performances, and it’s till not meeting my expectations. I was able to improve it enough to re-open the server, so you can stat again to run the clients.
However, this will still need a lot of improvement. I’m looking at a completely different database engine/schema to store the data. But that will take time to implement. At lest one week. So in the meantime, I will continue to tweak the existing database a bit and let the clients run.
More to come.
On the other side, I’m now thinking about what can be done with the data retrieved. Maybe the crawler will need to retrieve a bit more information from the pages.