Email 1

EXEC SUMMARY: Grader restarted (hopefully draining queues) - problem may re-occur - I will monitor carefully

So I'm taking a detailed look and documentating my process:

1. SSH'd to the grader and checking the logs in ~/rag_production/logs

ubuntu@ip-10-81-170-195:~/rag_production/logs$ ls -lat | head
total 66396
-rw-rw-r--  1 ubuntu ubuntu  156554 Dec  1 10:03 log2015-12-01-09:56:44.txt
drwxrwxr-x  2 ubuntu ubuntu    4096 Dec  1 09:56 .
-rw-rw-r--  1 ubuntu ubuntu   13948 Dec  1 09:56 log2015-12-01-09:49:28.txt
-rw-rw-r--  1 ubuntu ubuntu   23856 Dec  1 09:49 log2015-12-01-09:09:52.txt
-rw-rw-r--  1 ubuntu ubuntu   18860 Dec  1 09:04 log2015-11-30-14:23:57.txt
-rw-rw-r--  1 ubuntu ubuntu 1220743 Nov 30 14:23 log2015-11-27-04:56:13.txt
-rw-rw-r--  1 ubuntu ubuntu  266671 Nov 27 04:47 log2015-11-26-22:34:32.txt
-rw-rw-r--  1 ubuntu ubuntu  219306 Nov 26 22:34 log2015-11-26-21:23:24.txt
-rw-rw-r--  1 ubuntu ubuntu  209169 Nov 26 21:23 log2015-11-26-20:11:33.txt

so I can tail the latest log like so:

tail -1000lf log2015-12-01-09:56:44.txt

The last log entry is 2015-12-01T10:03:13.441858, and the time according to the server is

ubuntu@ip-10-81-170-195:~$ date Tue Dec 1 18:52:23 UTC 2015

so it looks like things have been stuck for the last 6 hours

the last entry in the log is as follows:

Running reference Cucumber scenarios:
Disabling profiles...
Feature: search for movies by director

  As a movie buff
  So that I can find movies with my favorite director
  I want to include and serach on director information in movies I enter

  Background: movies in database      # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:7
    Given the following movies exist: # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:9
      | title        | rating | director     | release_date |
      | Star Wars    | PG     | George Lucas | 1977-05-25   |
      | Blade Runner | PG     | Ridley Scott | 1982-06-25   |
      | Alien        | R      |              | 1979-05-25   |
      | THX-1138     | R      | George Lucas | 1971-03-11   |

  Scenario: add director to existing movie                # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:16
    When I go to the edit page for "Alien"                # features/step_definitions/web_steps.rb:48
    And I fill in "Director" with "Ridley Scott"          # features/step_definitions/web_steps.rb:60
    And I press "Update Movie Info"                       # features/step_definitions/web_steps.rb:52
    Then the director of "Alien" should be "Ridley Scott" # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:20

  Scenario: find movie with same director                       # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:22
    Given I am on the details page for "Star Wars"              # features/step_definitions/web_steps.rb:44
    When I follow "Find Movies With Same Director"              # features/step_definitions/web_steps.rb:56
    Then I should be on the Similar Movies page for "Star Wars" # features/step_definitions/web_steps.rb:230
    And I should see "THX-1138"                                 # features/step_definitions/web_steps.rb:105
    But I should not see "Blade Runner"                         # features/step_definitions/web_steps.rb:123

  Scenario: can't find similar movies if we don't know director (sad path)   # /home/ubuntu/rag_production/submissions/001eef52a29ea522413dfdc503f8513e/assignment4/2015-12-01-10-02-54/required_features/director.feature:29
    Given I am on the details page for "Alien"       # features/step_definitions/web_steps.rb:44
    Then I should not see "Ridley Scott"       # features/step_definitions/web_steps.rb:123
    When I follow "Find Movies With Same Director"       # features/step_definitions/web_steps.rb:56
    Then I should either be on the home page or the RottenPotatoes home page # features/step_definitions/saas_web_steps.rb:1
    And I should see "'Alien' has no director info"       # features/step_definitions/web_steps.rb:105

3 scenarios (3 undefined)
17 steps (13 skipped, 4 undefined)
0m0.056s

You can implement step definitions for undefined steps with these snippets:

Given(/^the following movies exist:$/) do |table|
  # table is a Cucumber::Ast::Table
  pending # express the regexp above with the code you wish you had
end

Then(/^the director of "(.*?)" should be "(.*?)"$/) do |arg1, arg2|
  pending # express the regexp above with the code you wish you had
end


 *** ERROR: #<SystemExit: exit> ***
Test passed. (+0.5)
----------------------------------------
  0 out of 3 scenarios passed
  Score: 0/40
----------------------------------------
Total score: 20 / 100.0
Completed in 18.401020025 seconds.

which is also the last thing I see when I attach to the grader tmux session, so I kill it with ctrl-c and restart using the stored command:

$ while true; do bundle exec ruby run_autograder.rb /home/ubuntu/rag_production/config/config.yml; done

So the grader is now running again (and hopefully draining the queues), but I guess we may have the same problem again if some students code causes the system to exit again.

I guess I will just have to check the system every few hours for the next two weeks until all the deadlines have passed, unless we can work out a better auto restart approach and/or solution.

Funny, because the while true; ...; done loop we start this in should keep the grader up continuously ...

I will also try a new deploy of the grader ...