|
Size: 13506
Comment:
|
← Revision 19 as of 2022-04-05 02:11:30 ⇥
Size: 0
Comment: outdated sagenb
|
| Deletions are marked like this. | Additions are marked like this. |
| Line 1: | Line 1: |
| ## page was renamed from Notebook design <<TableOfContents(2)>> = Overview of Design = Click for larger version of the picture: [[attachment:design.png|{{attachment:design.png|Notebook Design|height=500}}]] = What happens when some code is evaluated = {{{ Here's an example of the sequence of events when a person evaluates the following input: print 2 sleep(10) print 3 plot(sin) print "hello" graph_editor() The cast is as follows: * USER -- a human or a program controlling a web browser (selenium) or other user interface * CLIENT -- a program, possibly in javascript that displays something to the USER * SERVER -- a program that handles requests from the CLIENT, typically a web server such as flask + mod_wsgi + apache. * DATABASE -- stores data * DEVICE -- queries the DATABASE for work that needs to be performed, does that work, and updates the database in response 1. The USER types the above into an input object and submits this input. 2. The CLIENT (e.g., javascript) instantly adds some confirmation that the input is being sent, e.g., a spinning wheel, a green bar, or something. This CLIENT widget will timeout with an error if no output appears after 15 seconds (say). 3. The CLIENT sends a message to the SERVER using this URL schema: /home/wstein/17/5/evaluate The request does contain the input to the cell (POST). Here worksheet_id=17, cell_id=5. 4. (Alternatively, if the input was not changed -- e.g., in evaluate all.) the CLIENT sends a message to the SERVER using this URL schema: /home/wstein/17/5/evaluate The request does *NOT* contain the input to the cell. The same URL, but the absence of the cell input text field means don't change it. 5. The SERVER receives the above request (let's just assume it is the evaluate one). First it checks (via a DATABASE query) if a session_id has been assigned for this worksheet document. (It hasn't.) The SERVER does a query about device status and load, runs a very fancy algorithm to conclude that DEVICE 1 (not DEVICE 0) is the way to go. The SERVER upserts the following cell document in the DATABASE: {cell_id:5, worksheet_id:17, input:"""print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""", status:"needs_work", device:1, user_id='wstein', sequence_number: 0} The sequence number is global to the entire worksheet. After inserting this into the database, it returns a message to the CLIENT as follows: {cell_id:5, status:'needs_work'} The CLIENT receives the message and changes the cell 5 output to "working", and adds 5 to the list of needs_work cells. 6. DEVICE 1 does a query for all cells that have status "needs_work" and for which device:1. It gets back an iterator with one document in it, namely the above inserted document (from step 5). It then: - Allocates a fresh Python process with id 1974 for evaluation of code in the worksheet: 'wstein/17' We have an in-memory table that maps wstein/17 to 1974. - Does a DATABASE query to change the cell: {status:"working..."} - Sends a message to the Python process with id 1974 to evaluate """print 2\nsleep(10)\nprint 3\nplot(sin)\nprint "hello"\ngraph_editor()""". 7. The CLIENT queries the SERVER /home/wstein/17/5/update The SERVER does the following: - Responds to the CLIENT with nothing much, since nothing happened yet. {cell_id:5, status:'working', sequence_number: 1} (JSON) It happens this time that the CLIENT does *not* get the response message, due to a flakie network. 8. Meanwhile, DEVICE 1 checks on its message queue with the Sage process and finds the following output for process 1974: sys.stdout:"2". It then does this: - Update DATABASE cell 5 document: {... output:{stdout_0:{type:'text', order:0, content:'2', state:'open'}}, sequence_number: 2 ...} 9. Next, again the CLIENT queries for updates on cell 5 via the URL: /home/wstein/17/5/update The SERVER queries the DATABASE for info about cell 5 gets ... {stdout_0:{type:'text', order:0, content:'2', state:'open'}} ... and returns the JSON message: {cell_id:5, status:'working', output:{stdout_0:{type:'text', order:0, content:'2', state:'open'} }} The CLIENT gets the update back and calls a (javascript) function that renders sys.stdout. It also stores the number of characters from the sys.stdout stream that it has received because that stream is open. 10. Now DEVICE 1 notices that more output has appeared from process 1974, namely "3\n" and a new stream has started, since Sage's plot command has called the api function to make a new output block, so the DEVICE updates the DATABASE: {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}}, sequence_number: 3 ...} Incidentally, if there were any files 'foo.png' and 'bar.png' (say) created as a side effect (check modification times of files that are *closed* and compare them with the time the block started), then we would add them to the database as well. {... output:{stdout_0:{type:'text', order:0, files:{'foo.png':"lkajsfljsd", 'bar.png':"lksjflkjssdlfkj..."}, content:'2\n3\n', state:'closed'}} ...} In order to detect these automatically generated files, the code the device actually asks the worker process to execute will look like the following: try: block_api.new_block() # store current time, output sentinal character to stdout, etc. print 2 sleep(10) print 3 plot(sin) # calls new_block() print "hello" graph_editor() # calls new_block() finally: block_api.close() # check for files created since the beginning of the last block Note that the plot(sin)... function will actually call block_api.new_block(). At this moment, the block_api object will know it's in a block, and check the filesystem for all new files created until now. This output will actually send 3 streams, including two copies of the file a.png, one a.png displayed above the cosine plot and one beneath: g = plot(sin) g.save('a.png') plot(cos) g.save('a.png') whereas this will display just one sine plot g = plot(sin) g.save('a.png') print 1+2 g.save('a.png') However, this example would only display one copy of the file test.txt since the file was not closed when the first stdout block was ended: f=open('test.txt','w') f.write('test1') plot(cos) f.write('after plot') f.close() 11. The CLIENT queries for updates on cell 5, sending a parameter 'closed' or the number of characters (or bytes if the stream is not text) for each stream it has received information about. /home/wstein/17/5/update?stdout_0=1 The SERVER queries the database and gets output:{stdout_0:{content:'2\n3\n', state:'closed', ...}} The SERVER then sends the JSON message to the CLIENT. {cell_id:5, status:'working', output:{stdout_0:{content:'\n3\n', state:'closed'}} } (note that it did not send the first character since the client said it already had the first character) 12. Next DEVICE 1 sees that a plot appeared (in a new chunk of output), so it updates this into the DATABASE: {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}, plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'}} }, sequence_number: 4 ... } 13. The CLIENT queries for updates on cell 5. # closed means we already got it all, so do not bother sending it again or telling us it is closed. /home/wstein/17/5/update?stdout_0=closed The SERVER queries, gets stuff, and responds with a message: {cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} } The whole thing gets dropped! The client sees nothing. 14. The CLIENT queries for updates on cell 5. # closed means we already got it all, so do not bother sending it again or telling us it is closed. /home/wstein/17/5/update?stdout_0=closed The SERVER queries, gets stuff, and responds with a message: {cell_id:5, plot_0:{type:'plot', order:1, files:["plot0.png"], state:'closed'}} } This time the CLIENT draws (using javascript, somehow) the content. It gets the actual image file using the URL /home/wstein/17/5/plot_0/plot0.png So, for example, the CLIENT could insert an <img src="/home/wstein/17/5/plot_0/plot0.png"/> tag in the html of the page 15. The DRIVER 1 see a marker in the 1974 process stdout which says "another new output block". Also, it sees the output "hello". It puts this in the DATABASE. {... output:{{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}, plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'}}, stdout_1:{type:'text', order:2, content:"hello", state:'open'} }}, sequence_number: 5 } 16. The CLIENT queries for updates: /home/wstein/17/5/update?stdout_0=closed&plot_0=closed Gets back this JSON document: {cell_id:5, status:'working', output:{stdout_1:{type:'text', order:2, content:'hello', state:'open'}} } 17. Finally, the DRIVER 1 sees a marker in the process stating that there is a new output block, and the full computation of that cell is done. It also sees that a new stream called "graph_editor_0" with type 'graph_editor' was placed in the output along with a payload. It updates the DATABASE to look like this: {... output:{stdout_0:{type:'text', order:0, content:'2\n3\n', state:'closed'}, plot_0:{type:'plot', order:1, files:{'plot0.png':r"ASDFJAIEAJSJSF@#$#@$@(^!..."}, state:'closed'} , stdout_1:{type:'text', order:2, content:"hello", state:'closed'} , graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} } , status:'done', sequence_number: 6 } 18. The CLIENT queries for updates: /home/wstein/17/5/update?stdout_0=closed&plot_0=closed&stdout_1=5 and gets back this JSON: {cell_id:5, status:'done', output:{stdout_1:{content:'',state:'closed'}, graph_editor_0:{type:'graph_editor', order:3, content:"(^%$*^@S...", state:'closed'}} }} The client renders the graph editor, and stops painting the cell green, and stops querying for updates. }}} = FAQ = 1. Why do you use polling? Won't that be too slow? One of the goals here is to scale to 500 simultaneous users, so communication between the devices and the database is done in batches to minimize communication overhead. Obviously the device would poll the database intelligently: if there was no one using the device, back it off, but if the device was receiving a lot of work to do, poll more frequently. Clearly some experiments should be run to determine the right polling interval, or determine if there should be another very lightweight communication between the database and the device to notify the device that there is work to pick up. 2. Why is there a database in the middle of everything? Shouldn't the flask processes just communicate directly with the devices? There are several things to explain here. 1. Flask processes only last a few milliseconds---just enough to handle a single client request. You see in the design that multiple requests from the clients spawn different flask processes. So there is no long-running flask process assigned to a worksheet that maintains the state of the worksheet 2. We need a central store for the worksheet information that provides synchronization for multiple processes accessing the same data (for example, multiple clients sharing a worksheet, or a device and a client both accessing a cell's data). We could code up some central store ourselves, but a key assumption is that databases excel at exactly that sort of thing. That's an awful lot of tricky and subtle work that we'd rather not reinvent ourselves. 3. Many web applications work this way. The server merely queries and updates the database with work orders, while some backend process performs the work asynchronously and modifies the database with the results or status. The design works and is very standard in many cases. = Files = The original OpenOffice drawing for the design diagram: [[attachment:design.odg]]. |
