What happens when some code is evaluated
Here's an example of the sequence of events when a person evaluates the following input:
print 2
sleep(10)
print 3
plot(sin)
print "hello"
graph_editor()
The cast is as follows:
* USER -- a human or a program controlling a web browser (selenium) or other user interface
* CLIENT -- a program, possibly in javascript that displays something
to the USER
* SERVER -- a program that handles requests from the CLIENT,
typically a web server such as flask + mod_wsgi + apache.
* DATABASE -- stores data
* DEVICE -- queries the DATABASE for work that needs to be performed,
does that work, and updates the database in response
1. The USER types the above into an input object and submits this input.
2. The CLIENT (e.g., javascript) instantly adds some confirmation that the input is being sent, e.g., a spinning wheel, a green bar, or something. This CLIENT widget will timeout with an error if no output appears after 15 seconds (say).
3. The CLIENT sends a message to the SERVER using this URL schema:
/home/wstein/17/5/evaluate
The request does contain the input to the cell (POST). Here worksheet_id=17, cell_id=5.
4. (Alternatively, if the input was not changed -- e.g., in evaluate all.) the CLIENT sends a message to the SERVER using this URL schema:
/home/wstein/17/5/evaluate
The request does *NOT* contain the input to the cell. The same URL, but the absence of the cell input text field means don't change it.
5. The SERVER receives the above request (let's just assume it is the evaluate one).
First it checks (via a DATABASE query) if a session_id has been assigned for this worksheet document. (It hasn't.)
The SERVER does a query about device status and load, runs a very
fancy algorithm to conclude that DEVICE 1 (not DEVICE 0) is the
way to go.
The SERVER upserts the following cell document in the DATABASE:
{cell_id:5, worksheet_id:17,
input:"""print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""",
status:"needs_work", device:1, user_id='wstein',
sequence_number: 0}
The sequence number is global to the entire worksheet. After inserting this into the database, it returns a message to the CLIENT as follows:
{cell_id:5, status:'needs_work'}
The CLIENT receives the message and changes the cell 5 output to "working", and adds 5 to the list of needs_work cells.
6. DEVICE 1 does a query for all cells that have status "needs_work"
and for which device:1. It gets back an iterator with one
document in it, namely the above inserted document (from step 5).
It then:
- Allocates a fresh Python process with id 1974 for evaluation of code in the worksheet: 'wstein/17'
We have an in-memory table that maps wstein/17 to 1974.
- Does a DATABASE query to change the cell:
{status:"working..."}
- Sends a message to the Python process with id 1974 to evaluate """print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""".
7. The CLIENT queries the SERVER
/home/wstein/17/5/update
The SERVER does the following:
- Responds to the CLIENT with nothing much, since nothing happened yet.
{cell_id:5, status:'working', sequence_number: 1} (JSON)
It happens this time that the CLIENT does *not* get the response message, due to a flakie network.
8. Meanwhile, DEVICE 1 checks on its message queue with the Sage process and finds the
following output for process 1974: sys.stdout:"2". It then does
this:
- Update DATABASE cell 5 document:
{... output:{stdout_0:{type:'text', order:0, content:'2', state:'open'}}, sequence_number: 2 ...}
9. Next, again the CLIENT queries for updates on cell 5 via the URL:
/home/wstein/17/5/update
The SERVER queries the DATABASE for info about cell 5 gets
... {stdout_0:{type:'text', order:0, content:'2', state:'open'}} ...
and returns the JSON message:
{cell_id:5, status:'working', output:{stdout_0:{type:'text', order:0, content:'2', state:'open'} }}
The CLIENT gets the update back and calls a (javascript) function
that renders sys.stdout. It also stores the number of characters
from the sys.stdout stream that it has received because that
stream is open.
10. Now DEVICE 1 notices that more output has appeared from process
1974, namely "3\n" and a new stream has started, since Sage's plot command has called the
api function to make a new output block, so the DEVICE updates the DATABASE:
{... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}},
sequence_number: 3 ...}
Incidentally, if there were any files 'foo.png' and 'bar.png' (say)
created as a side effect (check modification times of files that are *closed* and compare them
with the time the block started), then we would add them to the database as well.
{... output:{stdout_0:{type:'text', order:0,
files:{'foo.png':"lkajsfljsd", 'bar.png':"lksjflkjssdlfkj..."},
content:'2\n3\n', state:'closed'}} ...}
In order to detect these automatically generated files, the code the device
actually asks the worker process to execute will look like the following:
try:
block_api.new_block() # store current time, output sentinal character to stdout, etc.
print 2
sleep(10)
print 3
plot(sin) # calls new_block()
print "hello"
graph_editor() # calls new_block()
finally:
block_api.close() # check for files created since the beginning of the last block
Note that the plot(sin)... function will actually call
block_api.new_block(). At this moment, the block_api object will
know it's in a block, and check the filesystem for all new files
created until now.
This output will actually send 3 streams, including two copies of the file a.png, one a.png displayed above the cosine
plot and one beneath:
g = plot(sin)
g.save('a.png')
plot(cos)
g.save('a.png')
whereas this will display just one sine plot
g = plot(sin)
g.save('a.png')
print 1+2
g.save('a.png')
However, this example would only display one copy of the file test.txt since the file was not closed when
the first stdout block was ended:
f=open('test.txt','w')
f.write('test1')
plot(cos)
f.write('after plot')
f.close()
11. The CLIENT queries for updates on cell 5, sending a parameter 'closed' or the number of characters (or bytes
if the stream is not text) for each stream it has received information about.
/home/wstein/17/5/update?stdout_0=1
The SERVER queries the database and gets
output:{stdout_0:{content:'2\n3\n', state:'closed', ...}}
The SERVER then sends the JSON message to the CLIENT.
{cell_id:5, status:'working', output:{stdout_0:{content:'\n3\n', state:'closed'}} }
(note that it did not send the first character since the client said it already had the first character)
12. Next DEVICE 1 sees that a plot appeared (in a new chunk of
output), so it updates this into the DATABASE:
{... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
plot_0:{type:'plot', order:1, files:{'plot0.png':r"[email protected]#$#@[email protected](^!..."}, state:'closed'}} },
sequence_number: 4 ... }
13. The CLIENT queries for updates on cell 5.
# closed means we already got it all, so do not bother sending it again or telling us it is closed.
/home/wstein/17/5/update?stdout_0=closed
The SERVER queries, gets stuff, and responds with a message:
{cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} }
The whole thing gets dropped! The client sees nothing.
14. The CLIENT queries for updates on cell 5.
# closed means we already got it all, so do not bother sending it again or telling us it is closed.
/home/wstein/17/5/update?stdout_0=closed
The SERVER queries, gets stuff, and responds with a message:
{cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} }
This time the CLIENT draws (using javascript, somehow) the content. It gets the actual image file using the URL
/home/wstein/17/5/plot_0/plot0.png
So, for example, the CLIENT could insert an <img src="/home/wstein/17/5/plot_0/plot0.png"/> tag in the html of the page
15. The DRIVER 1 see a marker in the 1974 process stdout which says "another new output block". Also, it
sees the output "hello". It puts this in the DATABASE.
{... output:{{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
plot_0:{type:'plot', order:1, files:{'plot0.png':r"[email protected]#$#@[email protected](^!..."}, state:'closed'}},
stdout_1:{type:'text', order:2, content:"hello", state:'open'} }},
sequence_number: 5
}
16. The CLIENT queries for updates:
/home/wstein/17/5/update?stdout_0=closed&plot_0=closed
Gets back this JSON document:
{cell_id:5, status:'working', output:{stdout_1:{type:'text', order:2, content:'hello', state:'open'}} }
17. Finally, the DRIVER 1 sees a marker in the process stating that
there is a new output block, and the full computation of that
cell is done. It also sees that a new stream called
"graph_editor_0" with type 'graph_editor' was placed in the
output along with a payload.
It updates the DATABASE to look like this:
{... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'},
plot_0:{type:'plot', order:1, files:{'plot0.png':r"[email protected]#$#@[email protected](^!..."}, state:'closed'} ,
stdout_1:{type:'text', order:2, content:"hello", state:'closed'} ,
graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} } ,
status:'done',
sequence_number: 6
}
18. The CLIENT queries for updates:
/home/wstein/17/5/update?stdout_0=closed&plot_0=closed&stdout_1=5
and gets back this JSON:
{cell_id:5, status:'done', output:{stdout_1:{content:'',state:'closed'},
graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} }}
The client renders the graph editor, and stops painting the cell green, and stops querying for updates. 
