ruby on rails - delayed_job stops running after some time in production -
in production, our delayed_job
process dying reason. i'm not sure if it's crashing or being killed operating system or what. don't see errors in delayed_job.log
file.
what can troubleshoot this? thinking of installing monit monitor it, tell me precisely when dies. won't tell me why died.
is there way make more chatty log file, can tell why might dying?
any other suggestions?
i've come across 2 causes of delayed_job failing silently. first actual segfaults when people using libxml in forked processes (this popped on mailing list time back).
the second issue 1.1.0 version of daemons delayed_job relies on has problem (https://github.com/collectiveidea/delayed_job/issues#issue/81), can worked around using 1.0.10 own gemfile has in it.
logging
there logging in delayed_job if worker dying without printing error it's because it's not throwing exception (e.g. segfault) or external killing process.
monitoring
i use bluepill monitor delayed job instances, , far has been successful @ ensuring jobs remain running. steps bluepill running application quite easy
add bluepill gem gemfile:
# monitoring gem 'i18n' # not sure why complained didn't have gem 'bluepill'
i created bluepill config file:
app_home = "/home/mi/production" workers = 5 bluepill.application("mi_delayed_job", :log_file => "#{app_home}/shared/log/bluepill.log") |app| (0...workers).each |i| app.process("delayed_job.#{i}") |process| process.working_dir = "#{app_home}/current" process.start_grace_time = 10.seconds process.stop_grace_time = 10.seconds process.restart_grace_time = 10.seconds process.start_command = "cd #{app_home}/current && rails_env=production ruby script/delayed_job start -i #{i}" process.stop_command = "cd #{app_home}/current && rails_env=production ruby script/delayed_job stop -i #{i}" process.pid_file = "#{app_home}/shared/pids/delayed_job.#{i}.pid" process.uid = "mi" process.gid = "mi" end end end
then in capistrano deploy file added:
# bluepill related tasks after "deploy:update", "bluepill:quit", "bluepill:start" namespace :bluepill desc "stop processes bluepill monitoring , quit bluepill" task :quit, :roles => [:app] run "cd #{current_path} && bundle exec bluepill --no-privileged stop" run "cd #{current_path} && bundle exec bluepill --no-privileged quit" end desc "load bluepill configuration , start it" task :start, :roles => [:app] run "cd #{current_path} && bundle exec bluepill --no-privileged load /home/mi/production/current/config/delayed_job.bluepill" end desc "prints bluepills monitored processes statuses" task :status, :roles => [:app] run "cd #{current_path} && bundle exec bluepill --no-privileged status" end end
hope helps little.
Comments
Post a Comment