Project Plan for Inferno Experiment

Aim of the project

We would like to create a simple distributed system which is built up from services running on remote machines. This might include three services:

1. A data extraction service (like GADS)
2. A data processing service (maybe finding the depth of an isosurface or similar)
3. A data rendering/visualisation service (perhaps creating a simple movie from the results of the processing)

The ultimate aim is to be able to "pipe" these services together in a very simple fashion. So the scientist (end user) would type in a command like this:

extract | process | render

The output from the extract service is piped directly into the input of the process service, and so forth. There are few notable advantages of this approach:

1. The data can be "streamed" directly from service to service without having to be written to any hard drive in an intermediate format
   • This is because the extract service will write its output to the standard output and the process service will read from standard input, just like Unix pipes
2. It should be possible for the process service to start running before the extraction service has finished.
   • If the extract service is extracting a time series, then the process service can start work as soon as it receives the first timestep's worth of data. Processing of the first timestep can proceed simultaneously with extraction of the second.
3. The Inferno system should make it easy to monitor the progress of the running services.

These features are very hard to achieve using a Web/Grid Services architecture.

[Question: It is likely that, as a first attempt, the data will not actually go from service to service but will pass through the controlling machine (red path on diagram below):
What would be the best way to make the data pass from service to service (green path on diagram)? Not sure yet, but see InfernoServiceToServiceCommunication for more info.]

Project breakdown

We shall follow this series of steps:

1. Become familiar with the Inferno environment and the Limbo language
   • We'll have to work out whether to use the Limbo language to implement the services. We might not have to; we could run native processes (written in C/C++/Java/etc) via Inferno.
2. Write a "dummy" extraction service that outputs some data
3. Work out how to run this service on a remote machine
   • And work out how to monitor its progress
4. Write a "dummy" processing service that takes in raw data and does something to it
5. Run the processing service on a different remote machine
6. Try to stream data between the two services
   • If this doesn't work immediately, make sure the streaming works when the services are running on the same machine (perhaps we can test them as native Unix/Linux processes, connected with standard Unix pipes)
7. Write a "dummy" rendering service and include this in the pipeline
8. Change the "dummy" services for "real" services.

-- JonBlower - 01 Apr 2004