c# - Read CSV data in batches and Process it -

i have csv file looks

#delta,1#     risk1,10 risk2,10 risk3,10 risk4,10 risk5,10 #delta,1#     risk6,10 risk7,10 risk8,10 risk9,10 risk10,10 

and on. these large files (in order of gbs).

what want able read them in batches like

start streamreader csv file first line before next #delta starts

---batch 1--- #delta,1 risk1,10 risk2,10 risk3,10 risk4,10 risk5,10 --batch 2----- #delta,1 risk6,10 risk7,10 risk8,10 risk9,10 risk10,10 ---------------------- 

and once batch put subset processing , come , restart preparing batch , on till end of file reached.

i have tried making linq's take , take while understanding of linq not getting far.

basically in summary have stream data in batches based on pattern in stream.. maybe brain cells dead or maybe late in evening. appreaicate anyone's help

assuming can keep each batch in memory @ time, , can keep streamreader open time, might want write this:

public static void processbatches(textreader reader,     func<string, bool> delimiterdetector,     action<list<string>> batchaction) {     string line;     list<string> batch = new list<string>();     while ((line = reader.readline()) != null)     {         if (delimiterdetector(line))         {             batchaction(batch);             batch = new list<string>();         }     }     batchaction(batch); } 

this assuming delimiter isn't required when processing batch.

you'd call this:

using (textreader reader = file.opentext("foo.csv")) {     processbatch(reader, line => line == "# delta,1", batchaction); } ... private static void batchaction(list<string> batch) {     ... }