When downloading data from a remote server it’s often useful to be able to preprocess the data. For example, if you’re downloading temperature data for a study on global warming it’s very convenient to be able to do a global average before downloading the complete data — this will greatly reduce the bandwidth used.
In 2007 I proposed a syntax/protocol for server-side processing in Opendap servers. There are currently two unofficial syntaxes for this, and both rely on encoding on the URL the script to be run by the server. My alternative proposal is to create a RESTful endpoint for adding functions to the server (the Opendap spec already supports function calls), with each script being referenced as a newly created function.
I decided to implement my proposal with Pydap, with one alteration: the functions themselves will be written in Javascript, not Python. My motivations for this are:
- Security: I would never run a user-submitted Python script on a server exposed to the world; and
- User-friendliness: I think more people know Javascript than Python, and — even though my proposal is language-agnostic — using Javascript slightly increases the chance of it being implemented by third parties.
How would it work? The server will implement a few basic functions (mean, for example). From them we will be able to create new functions. For example:
POST /capabilities/anomalies Host: server.example.com Content-type: text/javascript function(variable) { climatology = mean(variable, 0); return variable - climatology; }
It will then be possible to call the function using the already existing syntax for server-side functions:
http://server.example.com/dataset.dods?anomalies(var)
I’m using python-spidermonkey for this. In this case, it’s more a Python-with-Javascript-syntax: the number crunching all happens in the Python-side, and the Javascript namespace is limited to the variables and functions that I explicitly add. There are no __builtins__, and it’s possible to block dangerous introspective calls.
