java - Merging a large table with a large text file using JPA? -

we have large table of approximately 1 million rows, , data file millions of rows. need regularly merge subset of data in text file database table.

the main reason being slow data in file has references other jpa objects, meaning other jpa objects need read each row in file. ie imagine have 100,000 people, , 1,000,000 asset objects

person object --> asset list

our application uses pure jpa of data manipulation requirements. there efficient way using jpa/orm methodologies or going need revert pure sql , vendor specific commands?

why doesnt use age old technique: divide , conquer? split file small chunks , have parallel processes work on these small files concurrently.

and use batch inserts/updates offered jpa , hibernate. more details here

the ideal way in opinion though use batch support provided plain jdbc , commit @ regular intervals.

you might wants @ spring batch provided split/parallelization/iterating through files etc out of box. have used of these application of considerable size.