[weboob] Threads in weboob

Romain Bignon romain at peerfuse.org
Mon Apr 26 19:35:20 CEST 2010


Today, I think about threads in Weboob.

The reason is that I've modified the Scheduler class to uses threading.Timer
instead of the sched module, because there are problems with sched to do exactly
what I want, in particular with the PromptSchedular.

Quickly, a new problem appeared. It is obvious that there are concurrency
between threads (for example in the 'havesex' application).

But a new idea comes. What about using threads everywhere in weboob?

For example, currently, when you want to search a video with videoob, it firstly
calls the youtube video, which will firstly go on a page and waits 0.5s (for
example) to get server answer, and it parses data and returns them. Then, it
calls the youporn backend to do the same thing, then the youjizz backend, etc.

So, what about doing this in parallel? It calls either of backends at the same
time, and the frontend get results when they comes.

Now there are new constraints:
— only one backend may be called at the same time. So every backends would have
a global mutex;
— every classes called by backends may also have mutexes;
— the frontend could not get synchronous results. It is the most important
thing, because it is asynchronous.

I think that the solution would be to have a method in the Weboob class which do
the asynchronous calls, which results are stored by callbacks in a shared list,
and the method waits for events in this list, and for example yield them.

So the frontends calls don't change a lot, and they don't need to use callbacks
to get results. For example:

	for result in weboob.call_backends(ICapVideoProvider, search_videos, pattern):
		backend = result.backend
		video = result.data

It would be also possible to get a specific backend, but you need to lock it
before using it.

Well, it is a bit fuzzy, I don't know what do you think about this idea, which
add complexity in Weboob?

It would add also the ability to do non-blockant calls in applications (an
important thing for qvideoob for example).

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 489 bytes
Desc: not available
URL: <https://lists.symlink.me/pipermail/weboob/attachments/20100426/c36873fb/attachment.sig>

More information about the weboob mailing list