[Xmltv-devel] Bug in uk_rt for progs starting in BST/GMT switchover

[h***@gmail.com]

Don't know if you're already aware of this or not...


There's a bug in uk_rt for programmes which start in the hour of the BST->GMT switchover. The times are being calculated one-hour out.

E.g E4 on 27th

Incoming RT file =

New Girl~9/25, series 2~Eggs~~Neal Brennan~Jess Day*Zooey Deschanel|Cece*Hannah Simone|Nick Miller*Jake Johnson|Schmidt*Max Greenfield|Winston*Lamorne Morris|Melissa*Kay Cannon|Sadie*June Diane Raphael|Emma*Carla Gugino~false~false~true~true~false~false~false~false~~~Sitcom~Jess and Cece panic [...] starring Zooey Deschanel.~false~27/10/2013~00:40~01:10~30
New Girl~10/25, series 2~Bathtub~~~Jess Day*Zooey Deschanel|Winston*Lamorne Morris|Schmidt*Max Greenfield|Cece*Hannah Simone|Nick Miller*Jake Johnson~false~false~true~true~false~false~false~false~~~Sitcom~Jess is desperate [...] starring Zooey Deschanel.~false~27/10/2013~01:10~01:40~30
Suburgatory~15/22, series 2~Leaving Chatswin~~~George Altman*Jeremy Sisto|Tessa Altman*Jane Levy|Ryan Shay*Parker Young|Noah Werner*Alan Tudyk~false~false~true~true~false~false~false~false~~~Sitcom~George and Noah are left [...] starring Jeremy Sisto, Jane Levy and Parker Young.~false~27/10/2013~01:40~01:05~25
The Cleveland Show~18/23, series 4~Squirt's Honour~~~Cleveland Brown/Rallo Tubbs*Mike Henry|Donna Tubbs*Sanaa Lathan|Cleveland Brown Jr*Kevin Michael Richardson~false~false~true~true~false~false~false~false~~~Sitcom~Rallo joins the Freedom Squirts, [...] to their room.~false~27/10/2013~01:05~01:25~20


i.e. programmes are
New Girl / Eggs 00:40 BST - 01:10 BST
New Girl / Bathtub 01:10 BST - 01:40 BST
Suburgatory 01:40 BST - 01:05 GMT
The Cleveland Show 01:05 GMT - 01:25 GMT


But when parsed by uk_rt this is creating:

Processing programme title 'New Girl'
Episode number/total found: episode 9 of 25 (subtitle: x/y, series z)
Series number found: series 2 (subtitle: x/y, series z)
Pre- processing title/episode details 'New Girl / Eggs'
Post-processing title/episode details 'New Girl / Eggs'
Start time given as '2013/10/27 00:40', duration 30 mins
20131027004000 +0100 - Start time
20131027011000 +0100 - Stop time

----

Processing programme title 'New Girl'
Episode number/total found: episode 10 of 25 (subtitle: x/y, series z)
Series number found: series 2 (subtitle: x/y, series z)
Pre- processing title/episode details 'New Girl / Bathtub'
Post-processing title/episode details 'New Girl / Bathtub'
Start time given as '2013/10/27 01:10', duration 30 mins
20131027011000 +0000 - Start time
20131027014000 +0000 - Stop time

----

Processing programme title 'Suburgatory'
Episode number/total found: episode 15 of 22 (subtitle: x/y, series z)
Series number found: series 2 (subtitle: x/y, series z)
Pre- processing title/episode details 'Suburgatory / Leaving Chatswin'
Post-processing title/episode details 'Suburgatory / Leaving Chatswin'
Start time given as '2013/10/27 01:40', duration 25 mins
20131027014000 +0000 - Start time
20131027020500 +0000 - Stop time

----

Processing programme title 'The Cleveland Show'
Episode number/total found: episode 18 of 23 (subtitle: x/y, series z)
Series number found: series 4 (subtitle: x/y, series z)
Pre- processing title/episode details 'The Cleveland Show / Squirt's Honour'
Post-processing title/episode details 'The Cleveland Show / Squirt's Honour'
Start time given as '2013/10/27 01:05', duration 20 mins
20131027010500 +0000 - Start time
20131027012500 +0000 - Stop time

----


i.e.
New Girl / Eggs 00:40 BST - 01:10 BST
New Girl / Bathtub 01:10 GMT - 01:40 GMT
Suburgatory 01:40 GMT - 02:05 GMT
The Cleveland Show 01:05 GMT - 01:25 GMT


You can see the start & stop times of Bathtub and Suburgatory are wrong.


The problem is caused by lines 2756~
# Determine start time with correct UTC offset
my $start_dt = DateTime->new(
...

This is generating a time in *UTC*. Here's the reason:
http://search.cpan.org/~drolsky/DateTime-1.03/lib/DateTime.pm#Ambiguous_Local_Times

----
Ambiguous Local Times

Because of Daylight Saving Time, it is possible to specify a local time that is ambiguous. For example, in the US in 2003, the transition from to saving to standard time occurred on October 26, at 02:00:00 local time. The local clock changed from 01:59:59 (saving time) to 01:00:00 (standard time). This means that the hour from 01:00:00 through 01:59:59 actually occurs twice, though the UTC time continues to move forward.

If you specify an ambiguous time, then the latest UTC time is always used, in effect always choosing standard time.
----


Since you have not specified a UTC offset to DateTime->new() it is using an offset of 0. (i.e. Setting the TZ to Europe/London is not sufficient to determine BST or GMT - so it becomes an "ambiguous time"). You are then picking this up thinking this is the offset of the yyyymmdd hh:mm:ss you have given it, which of course it's not.

Cheers,
Geoff

[Nick Morrott]

Post by h***@gmail.com
Don't know if you're already aware of this or not...
There's a bug in uk_rt for programmes which start in the hour of the BST->GMT switchover. The times are being calculated one-hour out.

For several years prior to metabroadcast taking over the management of
the xmltv.radiotimes.com data feed, programmes having ambiguous start
times (because they were/are given in localtime, not UTC) during the
BST->GMT changeover were provided with hints. Either (BST) or (GMT)
was prepended to the programme title, and then stripped out during
processing.

These hints were used by the grabber to explicitily set the timezone
to (+0100 or +0000) rather than using local time via the general
Europe/London timezone. This worked perfectly and everyone was happy
(aside from the fact the MythTV couldn't handle the changeover, but
that's another story...).

These hints are clearly now missing, so the grabber is defaulting to
Europe/London local time as intended - but - creating incorrect
timings for some of the programmes with ambiguous start times.

Getting a new XMLTV release out, packaged and installed across
multiple distros is not going to happen by next week, so I'll see if I
can get metabroadcast to restore the hints upstream and get things
working perfectly as before.

Cheers,
Nick

[h***@gmail.com]

Post by h***@gmail.com
There's a bug in uk_rt for programmes which start in the hour of the BST->GMT switchover. The times are being calculated one-hour out.

[...] I'll see if I can get metabroadcast to restore the hints upstream and get things working perfectly as before.

Yes it's worth a try, but don't hold your breath ;) I'll see if I can find the relevant bit of Java code which creates the dat files (assuming such code is in the open source) and try and work up a quick fix; if you can give MB a patch file they just might be more amenable :)

Cheers,
Geoff

[Karl Dietz]

Post by h***@gmail.com

Post by h***@gmail.com
There's a bug in uk_rt for programmes which start in the hour of the BST->GMT switchover. The times are being calculated one-hour out.

[...] I'll see if I can get metabroadcast to restore the hints upstream and get things working perfectly as before.

Yes it's worth a try, but don't hold your breath ;) I'll see if I can find the relevant bit of Java code which creates the dat files (assuming such code is in the open source) and try and work up a quick fix; if you can give MB a patch file they just might be more amenable :)

I think its somewhere around
https://github.com/atlasapi/atlas-feeds/blob/master/src/main/java/org/atlasapi/feeds/xmltv/XmlTvFeedOutputter.java#L54

date(broadcast.getTransmissionTime()),
time(broadcast.getTransmissionTime()),
time(broadcast.getTransmissionEndTime()),
String.valueOf(broadcast.getBroadcastDuration() / 60)

Regards,
Karl

[h***@gmail.com]

Post by Karl Dietz

Post by h***@gmail.com
I'll see if I can find the relevant bit of Java code which creates the dat files (assuming such code is in the open source) and try and work up a quick fix;

I think its somewhere around
https://github.com/atlasapi/atlas-feeds/blob/master/src/main/java/org/atlasapi/feeds/xmltv/XmlTvFeedOutputter.java#L54
date(broadcast.getTransmissionTime()),
time(broadcast.getTransmissionTime()),
time(broadcast.getTransmissionEndTime()),
String.valueOf(broadcast.getBroadcastDuration() / 60)

Oops, I've only just seen this - many thanks Karl.

Geoff

[Nick Morrott]

Post by h***@gmail.com
Don't know if you're already aware of this or not...
There's a bug in uk_rt for programmes which start in the hour of the BST->GMT switchover. The times are being calculated one-hour out.

Thanks to everyone's hard work over the last few days (Geoff and Karl
here, Tom and Liam at metabroadcast), the upstream listings
(regenerated earlier after the patch was committed) now look perfect
and the grabber is producing correct start/stop time, as far as I can
tell from a quick check.

----

Processing programme title '(BST) New Girl'
Episode number/total found: episode 10 of 25 (subtitle: x/y, series z)
Series number found: series 2 (subtitle: x/y, series z)
Pre- processing title/episode details 'New Girl / Bathtub'
Post-processing title/episode details 'New Girl / Bathtub'
Start time given as '2013/10/27 01:10', duration 30 mins
Explicit timezone 'BST' detected in title
Forcing timezone to BST/+0100
20131027011000 +0100 - Start time
20131027014000 +0100 - Stop time

----

Processing programme title '(BST) Suburgatory'
Episode number/total found: episode 15 of 22 (subtitle: x/y, series z)
Series number found: series 2 (subtitle: x/y, series z)
Pre- processing title/episode details 'Suburgatory / Leaving Chatswin'
Post-processing title/episode details 'Suburgatory / Leaving Chatswin'
Start time given as '2013/10/27 01:40', duration 25 mins
Explicit timezone 'BST' detected in title
Forcing timezone to BST/+0100
20131027014000 +0100 - Start time
20131027020500 +0100 - Stop time

----

Processing programme title '(GMT) The Cleveland Show'
Episode number/total found: episode 18 of 23 (subtitle: x/y, series z)
Series number found: series 4 (subtitle: x/y, series z)
Pre- processing title/episode details 'The Cleveland Show / Squirt's Honour'
Assigned title 'The Cleveland Show' to category 'Animation'
Post-processing title/episode details 'The Cleveland Show / Squirt's Honour'
Start time given as '2013/10/27 01:05', duration 20 mins
Explicit timezone 'GMT' detected in title
Forcing timezone to GMT/+0000
20131027010500 +0000 - Start time
20131027012500 +0000 - Stop time

----

Processing programme title '(GMT) How I Met Your Mother'
Episode number/total found: episode 15 of 24 (subtitle: x/y, series z)
Series number found: series 8 (subtitle: x/y, series z)
Pre- processing title/episode details 'How I Met Your Mother / PS
I Love You'
Post-processing title/episode details 'How I Met Your Mother / PS
I Love You'
Start time given as '2013/10/27 01:25', duration 25 mins
Explicit timezone 'GMT' detected in title
Forcing timezone to GMT/+0000
20131027012500 +0000 - Start time
20131027015000 +0000 - Stop time

----

Processing programme title '(GMT) Rules of Engagement'
Episode number/total found: episode 4 of 13 (subtitle: x/y, series z)
Series number found: series 7 (subtitle: x/y, series z)
Pre- processing title/episode details 'Rules of Engagement / Cupcake'
Post-processing title/episode details 'Rules of Engagement / Cupcake'
Start time given as '2013/10/27 01:50', duration 25 mins
Explicit timezone 'GMT' detected in title
Forcing timezone to GMT/+0000
20131027015000 +0000 - Start time
20131027021500 +0000 - Stop time


Cheers,
Nick

[h***@gmail.com]

[...] the upstream listings
(regenerated earlier after the patch was committed) now look perfect
and the grabber is producing correct start/stop time, as far as I can
tell from a quick check.

Well done Nick. Getting this change through against the clock must've taken up a lot of your time. Thanks.


Just one minor thing; I think your stop time always assumes the same DST offset as the start time - this isn't correct if the stop time is after the changeover.

I.e. Suburgatory / Leaving Chatswin should be

<programme start="20131027014000 +0100" stop="20131027010500 +0000" channel="e4.channel4.com">

not

20131027014000 +0100 - Start time
20131027020500 +0100 - Stop time

But I think this bug has always been there, and hopefully most s/w will still make sense of the incorrect value. And at least the start times are correct now so the programmes appear at the correct place in the schedule, which is the important thing! :)

Cheers,
Geoff

[Nick Morrott]

Post by h***@gmail.com

[...] the upstream listings
(regenerated earlier after the patch was committed) now look perfect
and the grabber is producing correct start/stop time, as far as I can
tell from a quick check.

Well done Nick. Getting this change through against the clock must've taken up a lot of your time. Thanks.

It took some time, the bigger issue for me was getting enough
contiguous time to be able to sit down and get the Atlas env up and
running. Thanks again to you and Karl for your suggestions here on
this list and on the Github threads.

Post by h***@gmail.com
Just one minor thing; I think your stop time always assumes the same DST offset as the start time - this isn't correct if the stop time is after the changeover.
I.e. Suburgatory / Leaving Chatswin should be
<programme start="20131027014000 +0100" stop="20131027010500 +0000" channel="e4.channel4.com">
not

20131027014000 +0100 - Start time
20131027020500 +0100 - Stop time

But I think this bug has always been there, and hopefully most s/w will still make sense of the incorrect value. And at least the start times are correct now so the programmes appear at the correct place in the schedule, which is the important thing! :)

Absolutely. IIRC, the XMLTV::DST and Date::Manip based-code was
removed from the grabber when switching over to DateTime (due to the
painfully slow performance of Date::Manip, and the fact its
Europe/London and UTC offset handling broke several releases ago
(6.04/6.05?).

The grabber needs to re-localise the stop times for those times when a
programme straddles the changeover - I'll put it on the todo list.

Cheers,
Nick