Author Topic: Demonstration of 'best practises' remoting techniques for Pyxll  (Read 643 times)

phrrngtn

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
A common use-case for me is to use Excel as a UI as follows:
1 data-entry of input in worksheet with worksheet-scoped, well-known names.
2 press menu-item for a particular operation
3 gather up all necessary inputs (via the well-known range names) and serialize values as a blob of  YAML/JSON
4 run a command remotely (on a Linux host/cluster) on the serialized blob and retrieve results (usually a dictionary) as a blob of YAML/JSON
5 'distribute' the results to Excel (keys of dictionary correspond to Excel named ranges

step 4 can use any transport (I use ssh (via plink) and HTTP) but when the remote end is also a Python program, it may be convenient to remove the explicit serialization step and have whatever Python glue you are using do the work transparently. Some candidates:
1) zerorpc
2) https://github.com/ipython/ipyparallel (the newly packages IPython parallel stuff)
3) IPython/Jupyter client (Tony has already contribute an embedded IPython *kernel* within Excel; I am talking about the other direction)
4) Rpyc etc.
5) cloud stuff (don't know much about this)

One big potential advantage I see to using the ipyparallel stuff is that you get a nice abstraction with the directview to allow for parallel function application (either sync or async, load-balanced or round-robin). I think it would fit in nicely into Excel.

I am interested in exploring APIs to remoting in a flexible way that plays nicely with Excel UDFs and 'background' operations.



plink -batch -ssh remote-host.somecompany.com ipcluster start  --profile=remoting_fun --cluster=excel --n=4

#Here is the client code running within pyxll

def wrapped_getpid():   
    import os           
    return os.getpid() 

f = '/some/path/which/works/on/windows/.ipython/profile_remoting_fun/security/ipcontroller-excel-client.json'
c = Client(f)
v = c.direct_view()
v.apply_sync(wrapped_getpid)
[/font]

Tony

  • Administrator
  • Hero Member
  • *****
  • Posts: 610
  • Karma: +14/-1
    • View Profile
Re: Demonstration of 'best practises' remoting techniques for Pyxll
« Reply #1 on: August 14, 2015, 05:27:43 PM »
The RTD example might be of interest to you:
https://github.com/pyxll/pyxll-examples/tree/master/rtd
https://github.com/pyxll/exceltypes/blob/master/exceltypes/demos/excelRTDServer.py

Alternatively, if you don't want to use an RTD function but just want to have something call back into your Excel process later you could use a background timer to periodically poll for messages (like the poll_ioloop function in the ipython example).

Best regards,
Tony

phrrngtn

  • Newbie
  • *
  • Posts: 8
  • Karma: +0/-0
    • View Profile
Re: Demonstration of 'best practises' remoting techniques for Pyxll
« Reply #2 on: August 14, 2015, 06:10:14 PM »
I will take a look at the RTD examples but on thinking about it a bit (i.e. the couple of minutes it took me to write the original post!), I think I am mistakenly conflating a number of orthogonal concerns:

functions: either synchronous or asynchronous X either local or 'remoted' (i.e. where macro=False, no access to Excel, all parameters are serialized).
menu-item handlers: gather information synchronously; run something asynchronously (either local or remote); reliably update Excel with the results (see other post on the issues with async_call reliability)

The prompts a question on async functions. Can one do something like this:



@xl_func("async_handle h, int x")
def my_async_function(h, x):
    # we do the async return before the function return .. this is very unlikely to happen in the wild but is theoretically possible.
    xlAsyncReturn(h, x*2)
    return


The reason I ask is that I would like to have a number of 'stackable' decorators for:
a) async vs sync
b) remoting vs local
c) dogpile caching (I like the region-based approach in dogpile and would like to use it for Excel/pyxll)

I presume the function you register with Excel has to be registered as async up front. I presume the xlAsyncReturn does not have to happen from another thread? Do you know if it can happen *before the return*?