It has been a long time since I gave you some update.
Just to show you that things are still in progress, here are some statistics.
We currently have more than 24 millions of pages retrieved, and more than 500 millions of links waiting to be retrieved. One graphic has been prepared to display the pages progress. It’s not fully finished but you can see a preview there.
Also, we are currently working on an export tool which will aggregate the current data we have and list a tree of all websites already parsed by the application. More to come soon…