java - Nutch API advice -


i'm working on project need mature crawler work, , i'm evaluating nutch purpose. current needs relatively straightforward: need crawler able save data disk , need able recrawl updated resources of site , skip parts crawled. have experience working nutch code directly in java, not via command line. start simple: create crawler (or similar), minimally configure , start it, nothing fancy. there example this, or resource should looking at? i'm going on nutch documentation, of command line, search , other stuff. how usable nutch crawling module without need index , search? appreciated. thanks.

nutch different have ever practiced probably. because framework not has front query & search, athough solr seems more powerfull native nutch search front end. has crawling part , indexing (into lucene indexe).

if want use crawled other purposes search, need developp own programms , familiar hadoop , mapreduce programming.

not sure want crawling, doesn't nutch solution


Comments

Post a Comment

Popular posts from this blog

Add email recipient to all new Trac tickets -

is there configuration change can made trac send notification email address upon creation of new tickets? if can't done through config, plugin second best option, source code modification last resort. note: setting smtp_always_cc in notification section of tracini send messages on updates. i'm in need of email notifications only on creation of new ticket. does have email notification? 1 option have create query list ten (or many) recently-created tickets. when looking @ results of query, use "rss feed" button @ bottom of page subscribe feed notify whenever results of query updated.
Read more

400 Bad Request on Apache/PHP AddHandler wrapper -

i'm trying create wrapper/handler called on apache server whenever requests php script inside of directory. way can authorize users entire directory or write other stuff called when directory called. this best configuration i've been able come with... <directory "/srv/http/innov/public_html"> options -indexes allowoverride order allow,deny allow directoryindex index.php </directory> then in /srv/http/innov/public_html/kb/ have .htaccess file... options -indexes addhandler auth_handler .php action auth_handler ../auth_handler.php then in /srv/http/innov/public_html/kb/auth_handler.php follows... <?php $file = $_server['path_translated']; echo $file; ?> access log: - - [02/dec/2010:17:43:15 -0500] "get /kb/index.php http/1.1" 400 590 error log: [thu dec 02 17:50:19 2010] [error] [client xxx.xxx.xxx.xxx] invalid uri in request /kb/ http/1.1 i've checked browser , seems making proper request...
Read more

php - Change action and image src url's with jQuery -

i parsing form 1 of sites using php proxy. working fine form action , img src of submit button url's relative. how can prepend full http//www.domain.com/ in front of each of url's in form? edit: here's form code once fed through new domain: <div style="display: none;" id="subscribeform" class="newform"> <h5>subscribe our newsletter</h5> <form action="/campaignprocess.aspx?listid=27057" method="post" onsubmit="return checkwholeform93426(this)" name="catemaillistform93426"> <div class="form"> <div class="item"> <label for="clfullname">full name</label> <br /> <input type="text" maxlength="255" id="clfullname" name="fullname" class="cat_textbox_small" st...
Read more