There is now 6 servers serving workloads and receiving the results. The time to retrieve and submit data might be constant what ever the size of the database will be.
The database now contain XXXXX entries and is used at only 2.07 % of its capacity. If more capacity is required, it will be easy to add some servers/disk space to increase it without impacting the running applications.
I’m very happy with the results. I will now most probably add some load-balancing in front of the application server to split it between 2 different servers.
The total response time for both get load and submit results is now always below 2 seconds. From that, about 50% is because of the client to server network transfer.
Now that all this new architecture is in place, I will be able to work back on the crawler algorithm.