multithreading - Is there any way for executing a method multiple times, but managing connections/threads? (.NET) -


  1. I have a method that uses a connection (for example, a page that downloads the page is).
  2. I have to execute this method several times (like download 1000 pages).
  3. Doing this takes a long time to synchronously and sequentially.
  4. I have limited resources (8 maximum threads and / or 50 max simultaneous connections)
  5. I want to use all the resources to increase it.
  6. I know that parallelization (PLINQ, parallel extensions, etc.) can solve the problem, but I have already tried to do this, and this approach fails due to the rare resources .
  7. I do not want to change the wheel that parallels this type of work during the management of resources, someone will have done it

Anyone Can help?

Updates When you start, things get more complicated. It has been applied to multiple downloaders to mix asynchronous calls with the equivalent of maximum performance; Firefox downloader, it downloads 2 downloads together and when one of them is complete, then it gets the next file and so on. It may seem very easy to implement, but when I applied it, I was also upset about making it normal (useful for WebRequest and DbCommand) and to deal with problems (i.e. timeout)

Bounty Hunter This reward will be added to a trusted and free ($ $) NET library, which is a simple C # of the ASINCC function to parallel the functions of HTTP WebBarver Offers. Batting Response and SQL Commands.BizenExecute Non parallel should not wait to complete the N work to start the next work, but as soon as one of the first things ends, it should start a new task. The method should provide timeout handling.

Can you give more information why parallel links will not work?

To look at my view, your work is the best suit with PLinq. If you run on 8 core machines, then PLinq will split into 8 tasks, and all the remaining work queues for you.

Here is a draft code, 

  PagesToDownload.AsParallel () .All (DownloadMethodWithLimitConnections);

I did not understand why PLinq consumes your resources. Based on my trial, PLINQ performance is even better than using ThreadPool.


Comments

Post a Comment