python - Iterate over large collection in django - cache problem -


i need iterate on large collection (3 * 10^6 elements) in django kind of analysis can't done using single sql statement.

  • is possible turn off collection caching in django? (caching data not acceptable data has around 0.5gb)
  • is possible make django fetch collection in chunks? seems tries pre fetch whole collection in memory , iterate on it. think observing speed of execution:
    • iter(coll.objects.all()).next() - takes forever
    • iter(coll.objects.all()[:10000]).next() - takes less second

use queryset.iterator() walk on results instead of loading them first.


Comments

Post a Comment

Popular posts from this blog

Add email recipient to all new Trac tickets -

is there configuration change can made trac send notification email address upon creation of new tickets? if can't done through config, plugin second best option, source code modification last resort. note: setting smtp_always_cc in notification section of tracini send messages on updates. i'm in need of email notifications only on creation of new ticket. does have email notification? 1 option have create query list ten (or many) recently-created tickets. when looking @ results of query, use "rss feed" button @ bottom of page subscribe feed notify whenever results of query updated.
Read more

400 Bad Request on Apache/PHP AddHandler wrapper -

i'm trying create wrapper/handler called on apache server whenever requests php script inside of directory. way can authorize users entire directory or write other stuff called when directory called. this best configuration i've been able come with... <directory "/srv/http/innov/public_html"> options -indexes allowoverride order allow,deny allow directoryindex index.php </directory> then in /srv/http/innov/public_html/kb/ have .htaccess file... options -indexes addhandler auth_handler .php action auth_handler ../auth_handler.php then in /srv/http/innov/public_html/kb/auth_handler.php follows... <?php $file = $_server['path_translated']; echo $file; ?> access log: - - [02/dec/2010:17:43:15 -0500] "get /kb/index.php http/1.1" 400 590 error log: [thu dec 02 17:50:19 2010] [error] [client xxx.xxx.xxx.xxx] invalid uri in request /kb/ http/1.1 i've checked browser , seems making proper request...
Read more

asp.net - repeatedly call AddImageUrl(url) to assemble pdf document -

i'm using abcpdf , i'm curious if can recursively call addimageurl() function assemble pdf document compile multiple urls? something like: int pagecount = 0; int theid = thedoc.addimageurl("http://stackoverflow.com/search?q=abcpdf+footer+page+x+out+of+", true, 0, true); //assemble document while (thedoc.chainable(theid)) { thedoc.page = thedoc.addpage(); theid = thedoc.addimagetochain(theid); } pagecount = thedoc.pagecount; console.writeline("1 document page count:" + pagecount); //flatten document (int = 1; <= pagecount; i++) { thedoc.pagenumber = i; thedoc.flatten(); } //now try again theid = thedoc.addimageurl("http://stackoverflow.com/questions/1980890/pdf-report-generation", true, 0, true); //assemble document while (thedoc.chainable(theid)) { thedoc.page = thedoc.addpage(); theid = thedoc.addimagetoc...
Read more