Nrthugu

I decided to upgrade a rails application from 5.0.1 to 5.1.5. This included an upgrade from Zurb foundation from 6.2.3 to 6.4.x so I could start covering to grix-xy.

I have two Rails applications that are deployed to Debian boxes. One is to a local box that is a custom application to manage a VFW (Veterans of Foreign Wars) Post. I did about the same upgrade on that box a few months ago and had only minor problems. Not having yarn and node.js seemed to be one of the problems, but it's be up for months. The other is deployed to a Digital Oceans virtual server. This is the one I'm stuck.

My process was

• added and checked out a rails5.1 branch


• change gemfile for new rails and foundation-rails


• used rails app:update to update the system stuff


• did any new generate installs


• cleaned up views to use the newer version of foundation


• tested what I could in development


• merged the rails5.1 branch into master and committed changes to my repo


• tried to deploy using Capistrano - stuck

I was worried about SECRETS, but my method worked on the VFW app (using rbenv-vars to set secrets). I did a cap production deploy that started my trip into problems.

I got the yarn error

 DEBUG [27ff41fe] Command: cd /home/deploy/apps/pt_golfer/releases/20181111214738 && ( export RBENV_ROOT="$HOME/.rbenv" RBENV_VERSION="2.3.1" RAILS_ENV="production" RAILS_GROUPS="" ; RBENV_ROOT=$HOME/.rbenv RBENV_VERSION=2.3.1 $HOME/.rbenv/bin/rbenv exec bundle exec rake assets:precompile )

 DEBUG [fee53ebb]   Yarn executable was not detected in the system.

 DEBUG [fee53ebb]   

 DEBUG [fee53ebb]   Download Yarn at https://yarnpkg.com/en/docs/install

 DEBUG [fee53ebb]   

 DEBUG [fee53ebb]   I, [2018-11-11T16:37:28.513883 #11920]  INFO -- : Writing /home/deploy/apps/pt_golfer/releases/20181111214738/public/assets/application-683e0728d4efb1932c42933c1d3e66430959c9f4575bf91d83518c647597847f.js

 DEBUG [fee53ebb]   I, [2018-11-11T16:37:28.514512 #11920]  INFO -- : Writing /home/deploy/apps/pt_golfer/releases/20181111214738/public/assets/application-683e0728d4efb1932c42933c1d3e66430959c9f4575bf91d83518c647597847f.js.gz

 DEBUG [fee53ebb]   I, [2018-11-11T16:37:28.516684 #11920]  INFO -- : Writing /home/deploy/apps/pt_golfer/releases/20181111214738/public/assets/application.js-d85e0f2634e381cd64bda8c1bc5a6097e295cbdbbd7ef0880e0016d19cfa6e35.copy

 DEBUG [fee53ebb]   I, [2018-11-11T16:37:28.519517 #11920]  INFO -- : Writing /home/deploy/apps/pt_golfer/releases/20181111214738/public/assets/_settings.scss-b317c61809f397c15bef60771da1f85f01f265fa195aabb8dc44da529c36319c.copy

 DEBUG [fee53ebb]   rake aborted!

Autoprefixer doesn’t support Node v0.10.29. Update it.

But the deploy continued and I'm fairly sure it set 'current' link to the failed deploy
I then installed yarn on the DO virtual server and tried to deploy again.

This time it seemed to complete the deploy but failed in the unicorn:restart task

 INFO [3dfc7929] Running RBENV_ROOT=$HOME/.rbenv RBENV_VERSION=2.3.1 $HOME/.rbenv/bin/rbenv exec bundle exec unicorn -c /home/deploy/apps/pt_golfer/current/config/unicorn.rb -E production -D as deploy@my.do.ip.238

DEBUG [3dfc7929] Command: cd /home/deploy/apps/pt_golfer/current && ( export RBENV_ROOT="$HOME/.rbenv" RBENV_VERSION="2.3.1" ; RBENV_ROOT=$HOME/.rbenv RBENV_VERSION=2.3.1 $HOME/.rbenv/bin/rbenv exec bundle exec unicorn -c /home/deploy/apps/pt_golfer/current/config/unicorn.rb -E production -D )

DEBUG [355f81af]  master failed to start, check stderr log for details

on my terminal I did see the following error

(Backtrace restricted to imported tasks)

cap aborted!

SSHKit::Runner::ExecuteError: Exception while executing as deploy@@my.do.ip.238: cat /home/deploy/apps/pt_golfer/shared/tmp/pids/unicorn.pid.oldbin exit status: 1

cat /home/deploy/apps/pt_golfer/shared/tmp/pids/unicorn.pid.oldbin stdout: Nothing written

cat /home/deploy/apps/pt_golfer/shared/tmp/pids/unicorn.pid.oldbin stderr: cat: /home/deploy/apps/pt_golfer/shared/tmp/pids/unicorn.pid.oldbin: No such file or directory

At this point unicorn was still running the old version. I headed down a path to more failures.

I tried a cap production unicorn:hard-restart. That killed the old version but didn't start new server

Figuring I was in failing in something I didn't understand, I did a cap production rollback. That failed with the error 'Rails assets manifest file (or backup file) not found'

Looking at my deploy structure 'current' was pointing to the last failed deploy release (there were two). I then jumped the shark and linked current to the last good version (rails 5.0.x) and started unicorn. It came up with old ruby, but new css! It workable, fortunally I had not did that many css changes, my top-bar is screwed up with the change of foundation css, but functional.

My question is where do I go from here? I can try to revive my staging server, but that may take days. I guess I could also try to put staging on my DO virtual host somehow.

If I was a git expert, I'm sure that is a way to fall back to my last good commit but still keep the rails5.1 branch until I can figure out what went wrong.

My next path, since a know I can bring up the old version, it to try to figure out what the unicorn failure err is. Guess doing a bundle exec unicorn should take Capistrano out of the picture.

The Capistrano errors were not clear on where to find out what the error was.

Since I didn't get any suggestions on where to look, I switched back to the new version in current and tried to start it with service unicorn start and got an error that I could follow.

Nov 13 04:19:36 stevealex systemd[1]: Starting LSB: starts the unicorn app server...

Nov 13 04:19:41 stevealex unicorn_ptgolf[28314]: master failed to start, check stderr log for details

Nov 13 04:19:41 stevealex systemd[1]: unicorn_ptgolf.service: control process exited, code=exited status=1

Nov 13 04:19:41 stevealex systemd[1]: Failed to start LSB: starts the unicorn app server.

Nov 13 04:19:41 stevealex systemd[1]: Unit unicorn_ptgolf.service entered failed state.

That eventually pointed me to the unicorn.stderr.log (if you haven't had deploy problems for a few years, you kinda forget stuff!). There I found:

NoMethodError: undefined method `skip_before_filter' for SaController:Class

Did you mean?  skip_before_action

Then I remembered that was one of the items in upgrade from 5.0 to 5.1 to check.

Fixed that and I'm off and running.

I guess the answer to my obscure question is: "Check all you system logs and you will eventually find an error that means something"