Aug 2003, Jorge Arellano Cid, Ferdi Franceschini -Last update: Nov 2009 -----dpid -----------------Nomenclature: ------------dpi: generic

term referring to dillo's plugin system (version1). dpi1: specific term for dillo's plugin spec version 1. at: http://www.dillo.org/dpi1.html dpi program: any plugin program itself. dpi framework: the code base inside and outside dillo that makes dpi1 working possible (it doesn't include dpi programs). dpip: dillo plugin protocol. The HTML/XML like set of command tags and information that goes inside the communication sockets. Note: not yet fully defined, but functional. Note2: it was designed to be extensible. dpid: dillo plugin daemon. server plugin: A plugin that is capable of accepting connections on a socket. Dpid will never run more than one instance of a server plugin at a time. filter plugin: A dpi program that reads from stdin and writes to stdout, and that exits after its task is done (they don't remain as server plugins). Warning, dpid will run multiple instances of filter plugins if requested. ----------About dpid: ----------* dpid is a program which manages dpi connections. * dpid is a daemon that serves dillo using IDS sockets. * dpid launches dpi programs and arranges socket communication between the dpi program and dillo. The concept and motivation is similar to that of inetd. The plugin manager (dpid) listens for a service request on a socket and returns the socket/port pair of a plugin that handles the service. It also watches sockets of inactive plugins and starts them when a connection is requested.

----------------------------------------------------------What's the problem with managing dpi programs inside dillo? ----------------------------------------------------------That's the other way to handle it, but it started to show some problems (briefly outlined here): * When having two or more running instances of Dillo, one should prevail, and take control of dpi managing (but all dillos carry the managing code). * If the managing-dillo exits, it must pass control to another instance, or leave it void if there's no other dillo running! * The need to synchronize all the running instances of dillo arises. * If the controlling instance finishes and quits, all the dpi-program PIDs are lost. * Terminating hanged dpis is hard if it's not done with signals (PIDs) * Forks can be expensive (Dillo had to fork its dpis). * When a managing dillo exits, the new one is no longer the parent of the forked dpis. * If Unix domain sockets for the dpis were to be named randomly, it gets very hard to recover their names if the controlling instance of dillo exits and another must "take over" the managing. * It increments dillo's core size. * ... That's why the managing daemon scheme was chosen. ---------------------What does dpid handle? ---------------------It solves all the above mentioned shortcomings and also can do: * Multiple dillos: dpid can communicate and serve more than one instance of dillo. * Multiple dillo windows: two or more windows of the same dillo instance accessing dpis at the same time. * Different implementations of the same service dpi programs ("dpis") are just an implementation of a service. There's no problem in implementing a different one for the same service (e.g. downloads). * Upgrading a service: to a new version or implementation without requiring patching dillo's core or even bringing down the dpid. And finally, being aware that this design can support the following functions is very helpful: SCHEME Example

-----------------------------------------------------------* "one demand/one response" man, preferences, ... * "resident while working" downloads, mp3, ... * "resident until exit request" bookmarks, ... * "one client only" * "one client per instance" * "multiple clients/one instance" cd burner, ... man, ... downloads, cookies ...

-------Features -------* Dpi programs go in: "EPREFIX/dillo/dpi" or "~/.dillo/dpi". The binaries are named <name>.dpi as "bookmarks.dpi" and <name>.filter.dpi as in "hello.filter.dpi". The ".filter" plugins simply read from stdin and write to stdout. * Register/update/remove dpis from list of available dpis when a 'register_all' command is received. * dpid terminates when it receives a 'DpiBye' command. * dpis can be terminated with a 'DpiBye' command. * dpidc control program for dpid, currently allows register and stop. ----todo: ----These features are already designed, waiting for implementation: * dpidc remove ----------------How does it work? ----------------o o o on startup dpid reads dpidrc for the path to the dpi directory (usually EPREFIX/lib/dillo/dpi). ~/.dillo/dpi is scanned first. both directories are scanned for the list of available plugins. ~/.dillo/dpi overrides system-wide dpis. next it creates internet domain sockets for the available plugins and then listens for service requests on its own socket, and for connections to the sockets of inactive plugins. dpid returns the port of a plugin's socket when a client (dillo) requests a service. if 1) 2) 3) if 1) 2) 3) 4) the requested plugin is a 'server' then dpid stops watching the socket for activity forks and starts the plugin resumes watching the socket when the plugin exits the requested plugin is a 'filter' then dpid accepts the connection maps the socket fd to stdin/stdout (with dup2) forks and starts the plugin continues to watch the socket for new connections // May be not necessary after all...

o o

--------------------------dpi service process diagram --------------------------These drawings should be worth a thousand words! :) (I) .--- s1 s2 s3 ... sn [dpid] '--- srs The dpid is running listening on several sockets. (II) .--- s1 s2 s3 ... sn [dpid] [dillo] [dillo]

'--- srs ------------------' dillo needs a service so it connects to the service request socket of the dpid (srs) and asks for the socket name of the required plugin (using dpip). (III) .--- s1 s2 s3 ... sn [dpid] '--- srs [dillo] '---------------'

then it connects to that socket (s3, still serviced by dpid!) (IV) .--- s1 s2 s3 ... sn .[dpid] [dillo] . . '--- srs '---------------' . .............[dpi program] when s3 has activity (incoming data), dpid forks the dpi program for it... (V) .--- s1 s2 (s3) ... sn [dpid] [dillo]

'--- srs

.---------------' [dpi program]

... and lets it "to take over" the socket. Once there's a socket channel for dpi and dillo, the whole communication process takes place until the task is done. When the dpi program exits, dpid resumes listening on the socket (s3). -------------------------------So, how do I make my own plugin? -------------------------------Maybe the simplest way to get started is to understand a few concepts and then to use the hands-on method by using/modifying the hello dpi. It's designed as an example to get developers started. --------Concepts: --------* Dillo plugins work by communicating two processes: dillo and the dpi. * The underlying protocol (DPIP) has a uniform API which is powerful enough for both blocking and nonblocking IO, and filter or server dpis. * The simplest example is one-request one-answer (for example dillo asks for a URL and the dpi sends it). You'll find this and more complex examples in hello.c First, you should get familiar with the hello dpi as a user: $dillo dpi:/hello/ Once you've played enough with it, start reading the well commented code in hello.c and start making changes! --------------Debugging a dpi --------------The simplest way is to add printf() feedback using the MSG* macros. You can start the dpid by hand on a terminal to force messages to go there. Sometimes more complex dpis need more than MSG*. In this case you can use gdb like this. 1.- Add an sleep(20) statement just after the dpi starts. 2.- Start dillo and issue a request for your dpi. This will get your dpi started. 3.- Standing in the dpi source directory: ps aux grep dpi 4.- Take note of the dpi's PID and start gdb, then: (gdb) attach <PID>

5.- Continue from there... -----------Final Notes: -----------1.- If you already understand the hello dpi and want to try something more advanced: * bookmarks.c is a good example of a blocking server * file.c is an advanced example of a server handling multiple non-blocking connections with select(). 2.- Multiple instances of a filter plugin may be run concurrently, this could be a problem if your plugin records data in a file, however it is safe if you simply write to stdout. Alternatively you could write a 'server' plugin instead as they are guaranteed not to run concurrently. 3.- The hardest part is to try to modify the dpi framework code inside dillo; you have been warned! It already supports a lot of functionality, but if you need to do some very custom stuff, try extending the "chat" command, or asking in dillo-dev.

>>>>>>>>>>>>>>>>>>>>>

<<<<<<<<<<<<<<<<<<<<<