python - Concurrent or parallel requests on a 10-nodes cluster -


i wrote simple script 200 concurrent threads makes http requests , processes results. need around 50,000,000 requests. script simple modification of "twistedless solution" suggested here (the first reply).

if want run on 10-nodes aws cluster, how can parallelize it? if copy script onto 1 node , run it, happens? same on local machine?

being new this, need starting point. thanks!

on aws can have elastic load balancer (elb). can set manually or within autoscaling group. either way, x ( x=10 :d ) instances attached elb. need set elb health check properly, sees instances healthy , serves traffic of them. if done, need send concurrent requests elb endpoint. worth check sticky sessions if need or causes false result, sending traffic 1 instance because source of requests same.


Popular posts from this blog