[Xmltv-devel] tv_grab_fr maintenance

Discussion:

piratebab

2011-01-17 18:43:54 UTC

Hello all,
I am a team member of the mythtv-fr, the french community of mythtv user.
The french grabber, tv_grab_fr is not working for many months, and
seems not to be maintained anymore.
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.
Best regards,
Piratebab.

Centraladmin

2011-01-17 19:14:38 UTC

Permalink

Hi all,
I am the "official" maintener of the french grabber, and i fully agree
with this proposal.

Rgds, SF

Post by piratebab
Hello all,
I am a team member of the mythtv-fr, the french community of mythtv user.
The french grabber, tv_grab_fr is not working for many months, and
seems not to be maintained anymore.
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.
Best regards,
Piratebab.
------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
xmltv-devel mailing list
https://lists.sourceforge.net/lists/listinfo/xmltv-devel

Karl Dietz

2011-01-17 19:48:09 UTC

Permalink

Hello,

Post by piratebab
I am a team member of the mythtv-fr, the french community of mythtv user.
The french grabber, tv_grab_fr is not working for many months, and
seems not to be maintained anymore.

I'm a bit surprised to hear that as it was succesfully validating all
the time. I just tried myself and it seems to be working fine. (But it
is slow.)

Can you suggest a test that fails on tv_grab_fr as it is currently part
of xmltv? (Maybe some kind of time limit of X minutes per channels
per day?)

Post by piratebab
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.

piratebab

2011-01-17 21:14:10 UTC

Permalink

The actual tv_grab_fr use the "télépoche" web page. This site changes
very often the page arrangement (sometimes every month), and the grabber
need to be adapted after every télépoche change.
This grabber is not working at all now (no information on any channel).

Telerama is providing much reliable information. Our mythtv-fr member,
Zubrick, have adapted the tv_fr grabber for this program source.
The licence of telerama information is clear for a personal use of the
information (in french):
/La licence confère à l'utilisateur un droit d'usage privé, non
collectif et non exclusif, sur le contenu du site. *Elle comprend le
droit de reproduire pour stockage aux fins de représentation sur écran
monoposte* et de reproduction, en un exemplaire, pour copie de
sauvegarde ou tirage sur papier. Toute mise en réseau, toute
rediffusion, sous quelque forme, même partielle, est donc interdite. Ce
droit est personnel, il est réservé à l'usage exclusif et non collectif
du licencié. Il n'est transmissible en aucune manière.
/

The grabber fom zubrick can be downloaded here:
http://mythtv-fr.tuxfamily.org/wiki/lib/exe/fetch.php?hash=1474fc&media=http%3A%2F%2Fdownload.tuxfamily.org%2Fmythtvfr%2Fmiroir%2Fxmltv_grabbers_by_zubrick%2Ftv_grab_fr_telerama

Post by Karl Dietz
Hello,

Post by piratebab
I am a team member of the mythtv-fr, the french community of mythtv user.
The french grabber, tv_grab_fr is not working for many months, and
seems not to be maintained anymore.

I'm a bit surprised to hear that as it was succesfully validating all
the time. I just tried myself and it seems to be working fine. (But it
is slow.)
Can you suggest a test that fails on tv_grab_fr as it is currently part
of xmltv? (Maybe some kind of time limit of X minutes per channels
per day?)

Post by piratebab
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.

There is always room for a faster grabber.
What grabber are you talking about? Is that _fr_telerama?
If the source grants permission to use their data for xmltv the grabber
can easily be added. (just taking a wild guess as the user agent is
not XMLTV related)
Regards,
Karl
------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
xmltv-devel mailing list
https://lists.sourceforge.net/lists/listinfo/xmltv-devel

Karl Dietz

2011-01-18 08:24:08 UTC

Permalink

Hi,

Post by piratebab
The actual tv_grab_fr use the "télépoche" web page. This site changes
very often the page arrangement (sometimes every month), and the grabber
need to be adapted after every télépoche change.
This grabber is not working at all now (no information on any channel).

What I was trying to say is that we run an automated daily test on all
grabbers and tv_grab_fr has been going without error the last ~30 days.
I have tested it myself yesterday and got good data for the configured
test channel. That's what made me wonder. (But it is slow as hell)

See http://wiki.xmltv.org/index.php/XmltvValidation#test_grabbers on
how to validate a grabber. Links to the results of the daily runs are
in the menu on the left.

Post by piratebab
Telerama is providing much reliable information. Our mythtv-fr
member, Zubrick, have adapted the tv_fr grabber for this program
source.
The licence of telerama information is clear for a personal use of

I have taken the new grabber for a test run with the same configuration
(just TF1), it did validate and return some data. So if you would like
to replace _fr with _fr_telerama that's perfectly ok.

There is one major and some minor issues which can be fixed once the
script is imported (but should be taken care of before the next
release):

* The major issue is not using the normal XMLTV user agent but
making up some Telerama App agent for some reason. After changing
the script back to use our implementation get_nice it stopped
working.

* The script has it's own implementation of get_nice, which should
be changed back to the main get_nice.
* Unicode::MapUTF8 gets used, which adds a new dependency. This should
be changed to use Encode
* The description is just "France", I suggest something different from
the current grabber for easier support, like "France (Telerama)"
* What is with the channel logos? They seem to be referred by the
output. (can be downloaded at http://www.number6.ch/tools/ )
* the source-info-url should point at some page with content

And here are some suggestions on possible improvements.
* I don't see program types being returned with the categories. As
you are using MythTV it might be worth to add them.
(e.g. <category lang="en">series</category> for tv series)

Regards,
Karl

Francois Gouget

2011-01-18 23:45:45 UTC

Permalink

I use tv_grab_fr with MythTv and I have not noticed any issue. I
currently have program data until the 26th of January. What are
the symptoms you are noticing?

Post by piratebab
The actual tv_grab_fr use the "télépoche" web page. This site changes
very often the page arrangement (sometimes every month), and the
grabber need to be adapted after every télépoche change.

I feel that's a bit unfair. The web site has not changed since April
2010, that's 8 straight months now. And the grabber has seen 6 patches
since then which means it's not unmaintained.

Post by piratebab
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.

Having another grabber would be nice but I don't see the need to remove
the current one.

--
Francois Gouget <***@free.fr> http://fgouget.free.fr/
We are Pentium of Borg. You will be approximated. Division is futile.

piratebab

2011-01-19 18:14:50 UTC

Permalink

I have made an update of my computer 2 weeks ago. The grabber I used was
tv_grab_fr_telerama. During the update, it has been replaced by the
tv_grab_fr from the xmltv package.
After that, when I try to launch mythfilldatabase, it runs for hours,
and no data was in the database.
I replaced the file tv_grab_fr by tv_grab_fr_telerama, and it works again.
During the last 12 months, when I try to use the file from xlmtv
package, it fails every time to retrieve data.
If you are french speaker, you can join us at
http://mythtv-fr.tuxfamily.org/forums/viewforum.php?id=20, you will have
more information on our problems with the grabber. And you can also
explain us how you succeed to use tv_grab_fr.

regards, piratebab

Post by Francois Gouget

I use tv_grab_fr with MythTv and I have not noticed any issue. I
currently have program data until the 26th of January. What are
the symptoms you are noticing?

Post by piratebab
The actual tv_grab_fr use the "télépoche" web page. This site changes
very often the page arrangement (sometimes every month), and the
grabber need to be adapted after every télépoche change.

I feel that's a bit unfair. The web site has not changed since April
2010, that's 8 straight months now. And the grabber has seen 6 patches
since then which means it's not unmaintained.

Post by piratebab
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.

Having another grabber would be nice but I don't see the need to remove
the current one.
------------------------------------------------------------------------------
Protect Your Site and Customers from Malware Attacks
Learn about various malware tactics and how to avoid them. Understand
malware threats, the impact they can have on your business, and how you
can protect your company and customers by using code signing.
http://p.sf.net/sfu/oracle-sfdevnl
_______________________________________________
xmltv-devel mailing list
https://lists.sourceforge.net/lists/listinfo/xmltv-devel

Karl Dietz

2011-01-19 22:47:02 UTC

Permalink

Hi,

Post by piratebab

Post by Francois Gouget

Post by piratebab
The french grabber, tv_grab_fr is not working for many months, and
seems not to be maintained anymore.

I use tv_grab_fr with MythTv and I have not noticed any issue. I
currently have program data until the 26th of January. What are
the symptoms you are noticing?

This is to be expected because _fr and _fr_telerama, while using a
compatible configuration, use different channel ids which would have
required a configuration run of the grabber (to get the same channels)
and reassigning of the xmltv ids to the mythtv channels. (to map them
to the mythtv channels)

e.g. channel number 1 is TF1 on both grabbers, but has id
C1.telerama.fr on one and C1.telepoche.com on the other grabber.
But MTV is C31.telepoche.com in the established versus C150.telerama.fr
in the new grabber...

Still _fr_telerama sounds like an interesting addition if it works
with a proper useragent.

Regards,
Karl

Ben Bucksch

2011-01-19 22:55:05 UTC

Permalink

This is to be expected because _fr and _fr_telerama ... use different
channel ids
e.g. channel number 1 is TF1 on both grabbers, but has id
C1.telerama.fr on one and C1.telepoche.com on the other grabber.
But MTV is C31.telepoche.com in the established versus C150.telerama.fr
in the new grabber...

That's why the IDs should be independent of the grabber, and the same in
all grabbers.

You could take a look at the _eu_epgdata grabber to see how nice IDs
look like.

piratebab

2011-02-13 19:11:24 UTC

Permalink

I have tested tv_grab_fr included in 0.5.59 xmltv package, and it works
well.
Is there a site where we can download the latest version of the
tv_grab_fr script ? So we can add a link in our mythtv-fr site.
It would be difficult to modify tv_grab_fr_telerama to comply with user
agent and get nice from xmltv.
So we will keep it outside of xmltv .

regards,
piratebab

Post by Francois Gouget

I use tv_grab_fr with MythTv and I have not noticed any issue. I
currently have program data until the 26th of January. What are
the symptoms you are noticing?

Post by piratebab
The actual tv_grab_fr use the "télépoche" web page. This site changes
very often the page arrangement (sometimes every month), and the
grabber need to be adapted after every télépoche change.

I feel that's a bit unfair. The web site has not changed since April
2010, that's 8 straight months now. And the grabber has seen 6 patches
since then which means it's not unmaintained.

Post by piratebab
With have develleped a new grabber, used now by french mythtv user.
If the actual maintener is OK, we propose to use our grabber in the
xmltv package, and to maintain it.

Robert Eden

2011-02-14 06:06:04 UTC

Permalink

I have tested tv_grab_fr included in 0.5.59 xmltv package, and it works well.
Is there a site where we can download the latest version of the tv_grab_fr script ? So we can add a link in our mythtv-fr site.
It would be difficult to modify tv_grab_fr_telerama to comply with user agent and get nice from xmltv.
So we will keep it outside of xmltv .

piratebab

2011-02-14 17:55:45 UTC

Permalink

I need a link, because the telepoche web site change without notice, and
the grabber has to be updated if we want to have a continuity of service.
This update is not synchronised with xmltv packet update. So if we want
to use tv_grab_fr script, we need a link to download it has soon has it
is available.

Regarding tv_grab_fr_telerama, we are compliant with the licence (free
for a personal use). But their webmaster (as many ...) misused the user
agent. For him, iphone means "small screen", like smartphone!
Their aim is not to block any access accept phones from apple, it is
just dirty programming. So nothing illegal.
Many site are in this situation, and requires the plugin user agent
switcher of firefox.

Post by Robert Eden

I have tested tv_grab_fr included in 0.5.59 xmltv package, and it works well.
Is there a site where we can download the latest version of the
tv_grab_fr script ? So we can add a link in our mythtv-fr site.
It would be difficult to modify tv_grab_fr_telerama to comply with
user agent and get nice from xmltv.
So we will keep it outside of xmltv .

I don't understand... if the tv_grab_fr in 0.5.59 works, why do you need a link?
if they are blocking by user-agent, I don't recommend it. I don't
know the laws in France, but they seem to be saying we don't want you
getting our information.
If they are blocking by user agent, development within the xmltv SF
project has ceased. As you noted, there's nothing stopping a less
scrupulous project from hosting it.
Robert
------------------------------------------------------------------------------
Pinpoint memory and threading errors before they happen.
Find and fix more than 250 security defects in the development cycle.
Locate bottlenecks in serial and parallel code that limit performance.
http://p.sf.net/sfu/intel-dev2devfeb
_______________________________________________
xmltv-devel mailing list
https://lists.sourceforge.net/lists/listinfo/xmltv-devel