c# - Parallel.For freezes after around 1370 iterations, no idea why -

i'm running parallel.for loop on little on 7500 objects. inside loop, i'm doing number of things each of objects, calling 2 web services , 2 internal methods. web services inspect object, process , return string set property on object. same goes 2 internal methods.

i'm not writing out disk or reading disk.

i update ui in winforms app label , progress bar let user know it's at. here's code:

var task = task.factory.startnew(() => {   parallel.for(0, upperlimit, (i, loopstate) =>   {      if (cancellationtoken.iscancellationrequested)         loopstate.stop();      lblprogressbar.invoke(        (action)        (() => lblprogressbar.text = string.format("processing record {0} of {1}.", (progresscounter++), upperlimit)));      progbystep.invoke(        (action)        (() => progbystep.value = (progresscounter - 1)));        callsvc1(entity[i]);       conversion1(entity[i]);       callsvc2(entity[i]);       conversion2(entity[i]);   }); }, cancellationtoken); 

this taking place on win7 32bit machine.

any ideas why freezes when incrementer around 1370 or (it's been 1361, 1365 , 1371)?

any ideas how can debug , see what's locking if anything?

edit:
answers comments below:
@brokenglass - no, no interop. i'll try x86 compilation , let know.

@chibacity - because it's on background task, it's not freezing ui. until time freezes, progress bar , label tick along @ 2 every second. when freezes, stops moving. can verify number stops @ has been processed, no more. cpu usage on dual core 2.2ghz minimal during operation @ 3-4% each , 1-2% once freezes.

@henk holterman - takes around 10-12 minutes 1360 , yes, can verify of records have been processed not remaining records.

@codeinchaos - thanks, i'll try that! code work if take out parallel, takes forever , day. haven't tried restricting number of threads, will.

edit 2:
details what's going on webservices

basically what's going on web services pass in data , receive data (an xmlnode). node used in conversion1 process in turn sets property on entity sent callsvc2 method , on. looks this:

private void callsvc1(entity entity) {     var svc = new mywebservice();     var node = svc.callmethod(entity.someproperty);     entity.fieldtoupdate1.loadxml(node.innerxml); } private void conversion1(entity entity) {     // xml inspection/conversion stuff     if (entity.fieldtoupdate1.selectsinglenode("somenode") == "something") {         entity.fieldtoupdate2 = somethingthatwasconverted;     }     else {         // more logic     } } private void callsvc2(entity entity) {     var svc = new someotherwebservice();     var xmlnode = svc.methodtocall(entity.fieldtoupdate2.innerxml);     entity.anotherxmldocument.loadxml(xmlnode.innerxml); } 

as can see, it's pretty straightforward stuff. there's lot going on in of conversion methods, none of should blocking. , noted below, there 1024 threads in "waiting" status sitting on webservice calls. read here http://www.albahari.com/threading/ maxthreads defaulted 1023 .net 4 on 32 bit machine.

how can release waiting threads given have here?

a possible explanation: you've got process state can't create more threads, preventing work making progress, why everything's grinding halt.

frankly, whether or not hypothesis turns out correct, need take different approach this. parallel.for wrong way solve this. (parallel best suited cpu-bound work. have here io-bound work.) if need have thousands of web service requests in progress, need move on using asynchronous code, instead of multithreaded code. if use async apis, you'll able start thousands of requests simultaneously while using handful of threads.

whether requests able execute simultaneously matter - whether use current "thread apocalypse" implementation or more efficient async implementation, may running throttling. (.net can limit number of requests it'll make.) can ask make many requests like, might find of requests sat waiting earlier ones complete. e.g. think webrequest limits concurrent connections single domain 2... firing 1000+ threads (or 1000+ async requests) going result in loads more requests sitting waiting 1 of 2 current requests!

you should own throttling. need decide how many outstanding requests have simultaneously, , make sure start many requests @ once. asking parallel launch many can can bog down.

updated add:

a quick fix might use overload of parallel.for accepts paralleloptions object - can set maxdegreeofparallelism property limit number of concurrent requests. stop thread-heavy implementation running out of threads. remains inefficient solution problem. (and know, need make thousands of concurrent requests. if you're writing web crawler, example, that's reasonable thing want do. parallel not right class job though. use async operations. if web service proxies you're using support apm (beginxxx, endxxx), can wrap in task objects - task.taskfactory offers fromasync that'll provide task representing async operation in progress.

but if going try have thousands of requests in flight @ once, need think throttling strategy. throwing requests out there fast possible unlikely optimal strategy.