What happens when some code is evaluated

Here's an example of the sequence of events when a person evaluates the following input:

      print 2
      sleep(10)
      print 3
      plot(sin)
      print "hello"
      graph_editor()


The cast is as follows:

 * USER -- a human or a program controlling a web browser (selenium) or other user interface
 * CLIENT -- a program, possibly in javascript that displays something
   to the USER
 * SERVER -- a program that handles requests from the CLIENT,
   typically a web server such as flask + mod_wsgi + apache.
 * DATABASE -- stores data
 * DEVICE -- queries the DATABASE for work that needs to be performed,
   does that work, and updates the database in response

 1. The USER types the above into an input object and submits this input.

 2. The CLIENT (e.g., javascript) instantly adds some confirmation that the input is being sent, e.g., a spinning wheel, a green bar, or something.   This CLIENT widget will timeout with an error if no output appears after 15 seconds (say).  

 3. The CLIENT sends a message to the SERVER using this URL schema: 
           /home/wstein/17/5/evaluate  
    The request does contain the input to the cell (POST).  Here worksheet_id=17, cell_id=5.

 4. (Alternatively, if the input was not changed -- e.g., in evaluate all.) the CLIENT sends a message to the SERVER using this URL schema: 
           /home/wstein/17/5/evaluate
    The request does *NOT* contain the input to the cell.  The same URL, but the absence of the cell input text field means don't change it. 

 5. The SERVER receives the above request (let's just assume it is the evaluate one).  
    First it checks (via a DATABASE query) if a session_id has been assigned for this worksheet document.  (It hasn't.)

    The SERVER does a query about device status and load, runs a very
    fancy algorithm to conclude that DEVICE 1 (not DEVICE 0) is the
    way to go.

    The SERVER upserts the following cell document in the DATABASE:
               
             {cell_id:5, worksheet_id:17, 
              input:"""print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""", 
              status:"needs_work", device:1, user_id='wstein',
              sequence_number: 0}

   The sequence number is global to the entire worksheet. After inserting this into the database, it returns a message to the CLIENT as follows:

             {cell_id:5, status:'needs_work'}

   The CLIENT receives the message and changes the cell 5 output to "working", and adds 5 to the list of needs_work cells.

 6. DEVICE 1 does a query for all cells that have status "needs_work"
    and for which device:1.  It gets back an iterator with one
    document in it, namely the above inserted document (from step 5).

    It then:

     - Allocates a fresh Python process with id 1974 for evaluation of code in the worksheet: 'wstein/17'
       We have an in-memory table that maps wstein/17 to 1974.

     - Does a DATABASE query to change the cell:

             {status:"working..."}

     - Sends a message to the Python process with id 1974 to evaluate """print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""". 
    
 7. The CLIENT queries the SERVER    

           /home/wstein/17/5/update

    The SERVER does the following:

        - Responds to the CLIENT with nothing much, since nothing happened yet.

                   {cell_id:5, status:'working', sequence_number: 1}        (JSON)

    It happens this time that the CLIENT does *not* get the response message, due to a flakie network.

 8. Meanwhile, DEVICE 1 checks on its message queue with the Sage process and finds the
    following output for process 1974: sys.stdout:"2".  It then does
    this:

        - Update DATABASE cell 5 document:

                  {... output:{stdout_0:{type:'text', order:0, content:'2', state:'open'}}, sequence_number: 2 ...}

 9. Next, again the CLIENT queries for updates on cell 5 via the URL: 

           /home/wstein/17/5/update

    The SERVER queries the DATABASE for info about cell 5 gets 

           ... {stdout_0:{type:'text', order:0, content:'2', state:'open'}} ...

    and returns the JSON message:

                  {cell_id:5, status:'working', output:{stdout_0:{type:'text', order:0, content:'2', state:'open'} }}

    The CLIENT gets the update back and calls a (javascript) function
    that renders sys.stdout.  It also stores the number of characters
    from the sys.stdout stream that it has received because that
    stream is open.

  10. Now DEVICE 1 notices that more output has appeared from process
  1974, namely "3\n" and a new stream has started, since Sage's plot command has called the 
  api function to make a new output block, so the DEVICE updates the DATABASE:

             {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}},
                  sequence_number: 3 ...}

  Incidentally, if there were any files 'foo.png' and 'bar.png' (say)
  created as a side effect (check modification times of files that are *closed* and compare them
  with the time the block started), then we would add them to the database as well.

             {... output:{stdout_0:{type:'text', order:0, 
                          files:{'foo.png':"lkajsfljsd", 'bar.png':"lksjflkjssdlfkj..."}, 
                          content:'2\n3\n', state:'closed'}} ...}
  
  In order to detect these automatically generated files, the code the device 
  actually asks the worker process to execute will look like the following:

                    try:
                        block_api.new_block() # store current time, output sentinal character to stdout, etc.
                        print 2
                        sleep(10)
                        print 3
                        plot(sin) # calls new_block()
                        print "hello"
                        graph_editor() # calls new_block()
                    finally:
                        block_api.close() # check for files created since the beginning of the last block

  Note that the plot(sin)... function will actually call
  block_api.new_block().  At this moment, the block_api object will
  know it's in a block, and check the filesystem for all new files
  created until now.

  This output will actually send 3 streams, including two copies of the file a.png, one a.png displayed above the cosine
  plot and one beneath:

     g = plot(sin)
     g.save('a.png')
     plot(cos)
     g.save('a.png')

  whereas this will display just one sine plot

     g = plot(sin)
     g.save('a.png')
     print 1+2 
     g.save('a.png')
  
  However, this example would only display one copy of the file test.txt since the file was not closed when
  the first stdout block was ended:

     f=open('test.txt','w')
     f.write('test1')
     plot(cos)
     f.write('after plot')
     f.close()

 
  11. The CLIENT queries for updates on cell 5, sending a parameter 'closed' or the number of characters (or bytes 
      if the stream is not text) for each stream it has received information about.

           /home/wstein/17/5/update?stdout_0=1

    The SERVER queries the database and gets

         output:{stdout_0:{content:'2\n3\n', state:'closed', ...}}

    The SERVER then sends the JSON message to the CLIENT.

        {cell_id:5, status:'working', output:{stdout_0:{content:'\n3\n', state:'closed'}} }
        
    (note that it did not send the first character since the client said it already had the first character)

  12. Next DEVICE 1 sees that a plot appeared (in a new chunk of
  output), so it updates this into the DATABASE:

        {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
                     plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'}} },
             sequence_number: 4  ... }

  13. The CLIENT queries for updates on cell 5.

     # closed means we already got it all, so do not bother sending it again or telling us it is closed.

           /home/wstein/17/5/update?stdout_0=closed  

    The SERVER queries, gets stuff, and responds with a message:

            {cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} }

    The whole thing gets dropped!  The client sees nothing.

  14. The CLIENT queries for updates on cell 5.

     # closed means we already got it all, so do not bother sending it again or telling us it is closed.

           /home/wstein/17/5/update?stdout_0=closed  

    The SERVER queries, gets stuff, and responds with a message:

            {cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} }

    This time the CLIENT draws (using javascript, somehow) the content.  It gets the actual image file using the URL

            /home/wstein/17/5/plot_0/plot0.png
 
    So, for example, the CLIENT could insert an <img src="/home/wstein/17/5/plot_0/plot0.png"/> tag in the html of the page

  15. The DRIVER 1 see a marker in the 1974 process stdout which says "another new output block".  Also, it
    sees the output "hello".  It puts this in the DATABASE.

        {... output:{{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
                    plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'}},
                    stdout_1:{type:'text', order:2, content:"hello", state:'open'} }},
             sequence_number: 5
        }

  16. The CLIENT queries for updates:

           /home/wstein/17/5/update?stdout_0=closed&plot_0=closed

    Gets back this JSON document:

          {cell_id:5, status:'working', output:{stdout_1:{type:'text', order:2, content:'hello', state:'open'}} }

   
  17. Finally, the DRIVER 1 sees a marker in the process stating that
      there is a new output block, and the full computation of that
      cell is done.  It also sees that a new stream called
      "graph_editor_0" with type 'graph_editor' was placed in the
      output along with a payload.
     
      It updates the DATABASE to look like this:

        {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
                    plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'} ,
                    stdout_1:{type:'text', order:2, content:"hello", state:'closed'} ,
                    graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} } ,

            status:'done',
            sequence_number: 6
        }

  18. The CLIENT queries for updates:
    
           /home/wstein/17/5/update?stdout_0=closed&plot_0=closed&stdout_1=5

      and gets back this JSON:

        {cell_id:5, status:'done', output:{stdout_1:{content:'',state:'closed'}, 
                   graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} }}

      The client renders the graph editor, and stops painting the cell green, and stops querying for updates.