[Xmltv-devel] A generic 'Augment' package?

[h***@gmail.com]

I was wondering if any thought had been given to extracting the 'fixup' routines from out of tv_grab_uk_rt and making them into a more generic (i.e. grabber non-specific) 'augment' script?

This could run either as an in-flight procedure call, or perhaps better as a post-processor on the generated xml.

There are some excellent ideas in there and I'm sure other grabbers might like to use them if they were generally available.

Just a thought.

[h***@gmail.com]

Has anyone had any thoughts on this please? I'd like to apply some fix-ups to the epg data from my grabbers to fix some of the inconsistencies (and errors!) in the Press Association data.

I think the best place to do this would be in a standalone 'augment' script rather than putting it separately into each of my grabbers.

I was thinking of taking the relevant bits out of Nick's uk_rt grabber (if that's ok Nick?) and adapting them into a standalone augment script.

Is anyone aware if a script like this already exists?

Post by h***@gmail.com
I was wondering if any thought had been given to extracting the 'fixup' routines from out of tv_grab_uk_rt and making them into a more generic (i.e. grabber non-specific) 'augment' script?
This could run either as an in-flight procedure call, or perhaps better as a post-processor on the generated xml.
There are some excellent ideas in there and I'm sure other grabbers might like to use them if they were generally available.
Just a thought.

[Robert Eden]

In addition to the stand-alone script, what about adding a routine to
xmltv.pm? That way a grabber could call it against the programme hash
already n memory. The routine could cache various translate files so
they wouldn't needed to be loaded constantly.

Here's a radical throught, what about making the substitution lookup the
default in XMLTV.pm save programme function? That would instantly make
it available to all grabbers. Since most grabbers wouldn't have any
substitution config files, they won't be affected.

Robert

Post by h***@gmail.com
Has anyone had any thoughts on this please? I'd like to apply some fix-ups to the epg data from my grabbers to fix some of the inconsistencies (and errors!) in the Press Association data.
I think the best place to do this would be in a standalone 'augment' script rather than putting it separately into each of my grabbers.
I was thinking of taking the relevant bits out of Nick's uk_rt grabber (if that's ok Nick?) and adapting them into a standalone augment script.
Is anyone aware if a script like this already exists?

Post by h***@gmail.com
I was wondering if any thought had been given to extracting the 'fixup' routines from out of tv_grab_uk_rt and making them into a more generic (i.e. grabber non-specific) 'augment' script?
This could run either as an in-flight procedure call, or perhaps better as a post-processor on the generated xml.
There are some excellent ideas in there and I'm sure other grabbers might like to use them if they were generally available.
Just a thought.

------------------------------------------------------------------------------
October Webinars: Code for Performance
Free Intel webinars can help you accelerate application performance.
Explore tips for MPI, OpenMP, advanced profiling, and more. Get the most from
the latest Intel processors and coprocessors. See abstracts and register >
http://pubads.g.doubleclick.net/gampad/clk?id=60135031&iu=/4140/ostg.clktrk
_______________________________________________
xmltv-devel mailing list
https://lists.sourceforge.net/lists/listinfo/xmltv-devel

[Karl Dietz]

Post by Robert Eden

Post by h***@gmail.com
I think the best place to do this would be in a standalone 'augment'

script rather than putting it separately into each of my grabbers.

Post by Robert Eden

Post by h***@gmail.com
I was thinking of taking the relevant bits out of Nick's uk_rt

grabber (if that's ok Nick?) and adapting them into a standalone augment
script.

Post by Robert Eden

Post by h***@gmail.com
Is anyone aware if a script like this already exists?

In addition to the stand-alone script, what about adding a routine to
xmltv.pm? That way a grabber could call it against the programme hash
already n memory. The routine could cache various translate files so
they wouldn't needed to be loaded constantly.
Here's a radical throught, what about making the substitution lookup the
default in XMLTV.pm save programme function? That would instantly make
it available to all grabbers. Since most grabbers wouldn't have any
substitution config files, they won't be affected.

I'm with Robert. As every data source will need different fixups its
best to keep them separate. Just extending the writer to take a new
parameter "do-augment" or similar that enables lookup of the
supplementary files sounds good.

Regards,
Karl

[Robert Eden]

Post by Karl Dietz

Post by Robert Eden
In addition to the stand-alone script, what about adding a routine to
xmltv.pm? That way a grabber could call it against the programme hash
already n memory. The routine could cache various translate files so
they wouldn't needed to be loaded constantly.
Here's a radical throught, what about making the substitution lookup the
default in XMLTV.pm save programme function? That would instantly make
it available to all grabbers. Since most grabbers wouldn't have any
substitution config files, they won't be affected.

I'm with Robert. As every data source will need different fixups its
best to keep them separate. Just extending the writer to take a new
parameter "do-augment" or similar that enables lookup of the
supplementary files sounds good.

I was thinking about not having a "du-augment" parameter, but just do it
if the augment files (seperate per-user and supplement). I wouldn't
object to a "don't augment" parameter. My hope is to add it to every
grabber with zero effort by those devs.

Robert

[Karl Dietz]

Post by Robert Eden

Post by Karl Dietz

Post by Robert Eden
In addition to the stand-alone script, what about adding a routine to
xmltv.pm? That way a grabber could call it against the programme hash
already n memory. The routine could cache various translate files so
they wouldn't needed to be loaded constantly.
Here's a radical throught, what about making the substitution lookup the
default in XMLTV.pm save programme function? That would instantly make
it available to all grabbers. Since most grabbers wouldn't have any
substitution config files, they won't be affected.

I'm with Robert. As every data source will need different fixups its
best to keep them separate. Just extending the writer to take a new
parameter "do-augment" or similar that enables lookup of the
supplementary files sounds good.

I was thinking about not having a "du-augment" parameter, but just do it
if the augment files (seperate per-user and supplement). I wouldn't
object to a "don't augment" parameter. My hope is to add it to every
grabber with zero effort by those devs.

I see. My idea was to only add a dependency on the supplemental server
if there is any gain. E.g. for the grabber that does not get its data
of the internet.

Regards,
Karl

[Robert Eden]

Post by Karl Dietz
I see. My idea was to only add a dependency on the supplemental server
if there is any gain. E.g. for the grabber that does not get its data
of the internet.

Hmm excellent point... I didn't think about the load on the supplement
servers. Maybe only check the supplement servers if there is a grabber
correction file. (I envision per-user and grabber managed update files
for each type... so the user can add their own w/o conflict)

Robert

[h***@gmail.com]

Post by Robert Eden

Post by Karl Dietz
I see. My idea was to only add a dependency on the supplemental server
if there is any gain. E.g. for the grabber that does not get its data
of the internet.

Hmm excellent point... I didn't think about the load on the supplement
servers. Maybe only check the supplement servers if there is a grabber
correction file.

There shouldn't be a big load should there? If there isn't a supplemental file then it's just a 404. And using Last-Modified and Etag headers (as per your discussion on the Atlas forum Karl) should keep traffic down.

Post by Robert Eden
(I envision per-user and grabber managed update files
for each type... so the user can add their own w/o conflict)

Yes I think there would need to be separate per-user and per-grabber fixup files. The latter is to fix anomalies in the data source, whereas the per-user is unique to my specific PVR's needs.


My only concern with making it a core function would be the risk of someone generating invalid XML because they had a rubbish rule in their per-user fixup file. But I guess the chances of that are small.

Geoff