[weboob] Threads in weboob

Christophe Benz christophe.benz at gmail.com
Mon Apr 26 19:50:16 CEST 2010


I agree with this idea.

I have no time to answer in details, but let's do the job!

Le Mon, 26 Apr 2010 19:35:20 +0200,
Romain Bignon <romain at peerfuse.org> a écrit :

> Hello,
> Today, I think about threads in Weboob.
> The reason is that I've modified the Scheduler class to uses
> threading.Timer instead of the sched module, because there are
> problems with sched to do exactly what I want, in particular with the
> PromptSchedular.
> Quickly, a new problem appeared. It is obvious that there are
> concurrency between threads (for example in the 'havesex'
> application).
> But a new idea comes. What about using threads everywhere in weboob?
> For example, currently, when you want to search a video with videoob,
> it firstly calls the youtube video, which will firstly go on a page
> and waits 0.5s (for example) to get server answer, and it parses data
> and returns them. Then, it calls the youporn backend to do the same
> thing, then the youjizz backend, etc.
> So, what about doing this in parallel? It calls either of backends at
> the same time, and the frontend get results when they comes.
> Now there are new constraints:
> — only one backend may be called at the same time. So every backends
> would have a global mutex;
> — every classes called by backends may also have mutexes;
> — the frontend could not get synchronous results. It is the most
> important thing, because it is asynchronous.
> I think that the solution would be to have a method in the Weboob
> class which do the asynchronous calls, which results are stored by
> callbacks in a shared list, and the method waits for events in this
> list, and for example yield them.
> So the frontends calls don't change a lot, and they don't need to use
> callbacks to get results. For example:
> 	for result in weboob.call_backends(ICapVideoProvider,
> search_videos, pattern): backend = result.backend
> 		video = result.data
> It would be also possible to get a specific backend, but you need to
> lock it before using it.
> Well, it is a bit fuzzy, I don't know what do you think about this
> idea, which add complexity in Weboob?
> It would add also the ability to do non-blockant calls in
> applications (an important thing for qvideoob for example).
> Romain

Christophe Benz

More information about the weboob mailing list