[Xmltv-devel] [mythtv-users] mythfilldatabase: Ignoring unknown timestamp format: 20161231235960

[Ian Campbell]

Sure,  but i am I intrigued that your listing provider (who was your 
listing provider?)

It's schedules direct via tv_grab_sd_json. I've CC'd xmltv-devel, just
FtheirI I guess since as I said earlier I don't think 1s here or there
is worth worrying about (but if someone else wants to worry, fine!)

Context for xmltv-devel readers: This thread starts at http://lists.myt
htv.org/pipermail/mythtv-users/2016-December/390066.html and it appears
that something in the SD -> tv_grab_sd_json is mishandling the leap
second i.e. programs crossing midnight this year end 1s early

(independently mythfilldatabase does not accept the actual leap second
as an input, which is what I originally reported, but it seems mostly
moot since it looks like they shouldn't be present in the first place
for the given data).

is ending (and maybe starting) programs at 1sec before 
the hour. Do all your other programs end at xxxxxxxxxx5959 ie the ones not 
around the end of the year? Or xxxxxxxxxx2959 if ending on the half hour? 
Just curious.

Most programs span midnight, one of the "bad" ones from my unadjusted
data (so without the sed I mentioned earlier) was:
<programme start="20161231232500 +0000" stop="20161231235960 +0000" channel="24321">
<programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">

picking a random other channel at a random other time (not over midnight):
<programme start="20161223002000 +0000" stop="20161223005000 +0000" channel="25117">
<programme start="20161223005000 +0000" stop="20161223011500 +0000" channel="25117">

picking another random one with 5959 in it though:
<programme start="20161231170000 +0000" stop="20161231190000 +0000" channel="31786">
<programme start="20161231190000 +0000" stop="20170101005959 +0000" channel="31786">
<programme start="20170101010000 +0000" stop="20170101060000 +0000" channel="31786">
<programme start="20170101060000 +0000" stop="20170101080000 +0000" channel="31786">

and another:
<programme start="20161231210000 +0000" stop="20161231230000 +0000" channel="57747">
<programme start="20161231230000 +0000" stop="20170101005959 +0000" channel="57747">
<programme start="20170101010000 +0000" stop="20170101030000 +0000" channel="57747">
<programme start="20170101030000 +0000" stop="20170101060000 +0000" channel="57747">

WRT the half hour and other offsets they have a similar issue when
spanning the leap second e.g.:
<programme start="20161231210000 +0000" stop="20161231220000 +0000" channel="65161">
<programme start="20161231220000 +0000" stop="20170101002959 +0000" channel="65161">
<programme start="20170101003000 +0000" stop="20170101011500 +0000" channel="65161">
or
<programme start="20161231210000 +0000" stop="20161231233000 +0000" channel="21824">
<programme start="20161231233000 +0000" stop="20170101001459 +0000" channel="21824">
<programme start="20170105011500 +0000" stop="20170105030000 +0000" channel="21824">

So it looks like crossing the leap second has introduced a 1s quirk,
but it seems to resolve itself once programs are fully within 2017.

Ian.

[Kevin Groeneveld]

Post by Ian Campbell
It's schedules direct via tv_grab_sd_json. I've CC'd xmltv-devel, just
<programme start="20161231232500 +0000" stop="20161231235960 +0000" channel="24321">
<programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">

Interesting. Even if it is technically correct (which I don't think it is
in this case) it may be a good idea for the grabber to never output a time
that ends in a value greater than 59.

Between my curiosity and perfectionism I will probably dig into this later
today to figure out what is going and try to fix it. Although I do wonder
if maybe it is already fixed. I have made significant changes to the time
handling in the CVS code that is not in an official release yet. These
changes make the CVS version significantly faster than the release
version. Maybe these changes also fixed the leap second (or maybe it is
okay in the release version and I broke it in the CVS version).


Kevin

[Ian Campbell]

Post by Ian Campbell
It's schedules direct via tv_grab_sd_json. I've CC'd xmltv-devel, just

 

Post by Ian Campbell
    <programme start="20161231232500 +0000" stop="20161231235960
+0000" channel="24321">
    <programme start="20170101000000 +0000" stop="20170101001500
+0000" channel="24321">

Interesting.  Even if it is technically correct (which I don't think
it is in this case) it may be a good idea for the grabber to never
output a time that ends in a value greater than 59.
Between my curiosity and perfectionism I will probably dig into this
later today to figure out what is going and try to fix it.  Although
I do wonder if maybe it is already fixed.  I have made significant
changes to the time handling in the CVS code that is not in an
official release yet.  These changes make the CVS version
significantly faster than the release version.  Maybe these changes
also fixed the leap second (or maybe it is okay in the release
version and I broke it in the CVS version).

Sorry, I should have said: I'm running 0.5.68-1 from the Debian
packages.

Ian.

[Kevin Groeneveld]

Post by Ian Campbell

I do wonder if maybe it is already fixed. I have made significant
changes to the time handling in the CVS code that is not in an
official release yet. These changes make the CVS version
significantly faster than the release version. Maybe these changes
also fixed the leap second (or maybe it is okay in the release
version and I broke it in the CVS version).

Sorry, I should have said: I'm running 0.5.68-1 from the Debian
packages.

I just tried running 0.5.68 (although not the Debian package) and the
latest CVS code and in both cases I could not seem to trigger any output
which includes a time that ends in "60" or "59". Maybe there is something
different about the time setup on my system.

I still think the latest CVS version may fix the problem you are seeing.
It converts all times to unix time internally (which does not include leap
seconds) and uses the gmtime function to convert the times back to strings
for the final output. It seems it should be impossible to get a time
ending in "60". The 0.5.68 code stored and manipulated times using perl
DateTime objects. DateTime does support leap seconds so I could see it
outputting a time ending in "60".

Can you try running the latest CVS version? You can download just the
tv_grab_sd_json grabber from:
http://xmltv.cvs.sourceforge.net/viewvc/xmltv/xmltv/grab/sd_json/tv_grab_sd_json.
You should be able to run that file from pretty much any location on your
system if you have a recent xmltv package installed.


Kevin

[Ian Campbell]

Post by Kevin Groeneveld

Post by Kevin Groeneveld

I do wonder if maybe it is already fixed.  I have made

significant

changes to the time handling in the CVS code that is not in an
official release yet.  These changes make the CVS version
significantly faster than the release version.  Maybe these

changes

also fixed the leap second (or maybe it is okay in the release
version and I broke it in the CVS version).

Sorry, I should have said: I'm running 0.5.68-1 from the Debian
packages.

I just tried running 0.5.68 (although not the Debian package) and the
latest CVS code and in both cases I could not seem to trigger any
output which includes a time that ends in "60" or "59".  Maybe there
is something different about the time setup on my system.

Possibly. FWIW my lineup is GBR-0001194-DEFAULT, I'm running a Debian
Stretch/Sid hybrid system.

Post by Kevin Groeneveld
I still think the latest CVS version may fix the problem you are
seeing.  It converts all times to unix time internally (which does
not include leap seconds) and uses the gmtime function to convert the
times back to strings for the final output.  It seems it should be
impossible to get a time ending in "60".  The 0.5.68 code stored and
manipulated times using perl DateTime objects.  DateTime does support
leap seconds so I could see it outputting a time ending in "60".
Can you try running the latest CVS version?  You can download just
http://xmltv.cvs.sourceforge.net/viewvc/xmltv/xmltv/grab/sd_json/tv_g
rab_sd_json.  You should be able to run that file from pretty much
any location on your system if you have a recent xmltv package
installed.

I tried to install the dev version like I always have but:

$ perl Makefile.PL  PREFIX=/usr/local/stow/xmltv-head/
Can't locate lib/Ask/Term.pm in @INC (you may need to install the lib::Ask::Term module)
(@INC contains: /etc/perl /usr/local/lib/i386-linux-gnu/perl/5.24.1 /usr/local/share/perl/5.24.1
/usr/lib/i386-linux-gnu/perl5/5.24 /usr/share/perl5 /usr/lib/i386-linux-gnu/perl/5.24
/usr/share/perl/5.24 /usr/local/lib/site_perl /usr/lib/i386-linux-gnu/perl-base)
at Makefile.PL line 254.

lib/Ask/Term.pm is in the XMLTV tree AFAICT, not a 3rd party module.

Maybe regular devs don't notice this because they also have a new
enough one installed from a previous iteration to allow them to
iteratively update it?

I set PERL5LIB to my xmltv.git checkout (i.e. "PERL5LIB=`pwd` perl
Makefile.PL ..." and was able to install. The previously bad entry:

<programme start="20161231232500 +0000" stop="20161231235960 +0000" channel="24321">
<programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">
has now become:
<programme start="20161231232500 +0000" stop="20170101000000 +0000" channel="24321">
<programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">

and what was previously:
   <programme start="20161231210000 +0000" stop="20161231220000 +0000" channel="65161">
   <programme start="20161231220000 +0000" stop="20170101002959 +0000" channel="65161">
   <programme start="20170101003000 +0000" stop="20170101011500 +0000" channel="65161">
is now:
<programme start="20161231210000 +0000" stop="20161231220000 +0000" channel="65161">
<programme start="20161231220000 +0000" stop="20170101003000 +0000"
channel="65161">
<programme start="20170101003000 +0000" stop="20170101011500 +0000" channel="65161">

So it's all looking good, thanks!

Ian.

[Ian Campbell]

    <programme start="20161231232500 +0000" stop="20161231235960 +0000" channel="24321">
    <programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">
    <programme start="20161231232500 +0000" stop="20170101000000 +0000" channel="24321">
    <programme start="20170101000000 +0000" stop="20170101001500 +0000" channel="24321">

Just for completeness the SD JSON entries for these were:

      {
        "videoProperties": [
          "letterbox"
        ],
        "new": true,
        "md5": "mOC4USUaW5ASAWno3ii81A",
        "programID": "EP025790950001",
        "liveTapeDelay": "Live",
        "duration": 2100,
        "airDateTime": "2016-12-31T23:25:00Z"
      }
      {
        "programID": "EP018385260004",
        "md5": "Qa/9poKoz0NCykMwVDYAYw",
        "continued": true,
        "videoProperties": [
          "letterbox"
        ],
        "liveTapeDelay": "Live",
        "new": true,
        "airDateTime": "2017-01-01T00:00:00Z",
        "duration": 900
      },

Ian.

[Ian Campbell]

Post by Ian Campbell

 
    <programme start="20161231232500 +0000" stop="20161231235960

+0000" channel="24321">

    <programme start="20170101000000 +0000" stop="20170101001500

+0000" channel="24321">

    <programme start="20161231232500 +0000" stop="20170101000000

+0000" channel="24321">

    <programme start="20170101000000 +0000" stop="20170101001500

+0000" channel="24321">
      {
        "videoProperties": [
          "letterbox"
        ],
        "new": true,
        "md5": "mOC4USUaW5ASAWno3ii81A",
        "programID": "EP025790950001",
        "liveTapeDelay": "Live",
        "duration": 2100,
        "airDateTime": "2016-12-31T23:25:00Z"
      }

Is this correct? A program running from 23:25 until midnight is 35
minutes, which is 2100 seconds, but with the leap second should it not
be 2101?

(IOW is the dev version of sd_json masking an issue with the sd data,
while the released version exposes it?)

Ian.

[Gary Buhrmaster]

On Wed, Dec 21, 2016 at 1:25 AM, Kevin Groeneveld <***@gmail.com> wrote:
...

So, as the documentation indicates, DateTime can account for leap seconds.
However if I change the 2008 to 2016 in the above code the leap second is
not recognized. I suspect a newer version of Ubuntu or DateTime would fix
this.

The DateTime module version 1.34 was the first
(according the the upstream change log) to support
this Decembers leap second(*). I have no idea how
Canonical decides which items are important
enough to push and update for a LTS version, but if
DateTime is not one of them, they may need to
reconsider. Someone running Ubuntu might want
to open a support ticket on the issue to insure
that the DateTime sheppard/packager is informed
for the next time.



(*) On or about July 6th. Which is about as early
as possible, since IERS only made the formal
announcement that day.

[Ian Campbell]

If Gracenote sent 2101 seconds and that was used to compute a
stop
times, all stop times post leap second would be off as the system
clock
was updated.  I think 2100 is the best answer as it only effects
shows
that stop at exactly midnight.

Minor nit: It appears to affect any show which crosses over
midnight
(or more accurately, a leap second), e.g. up thread I reported this
     <programme start="20161231170000 +0000" stop="20161231190000
+0000" channel="31786">
     <programme start="20161231190000 +0000" stop="20170101005959
+0000" channel="31786">
     <programme start="20170101010000 +0000" stop="20170101060000
+0000" channel="31786">
     <programme start="20170101060000 +0000" stop="20170101080000
+0000" channel="31786">
which has the blip between 20170101005959 and 20170101010000.
But I don't think that invalidates the thrust of your argument.

It's interesting that your time APIs seem to be aware of the
upcoming 
leap second.  I didn't test with perl DateTime, but I did try PHP...
I 
expected both used the OS API and didn't seem to know about the leap
second.
What OS are you using?

Debian Stretch (current testing) although not completely up to
date/sporadically updated. I'm currently running with tzdata 2016j-2.
When I first noticed this I was running 2016i-1 but upgraded and reprod
before I posted to mythtv-users. Both are newer than 2016g which Nick
identified downthread as the point where the leap second was added.

FWIW I also upgraded libdatetime-perl from 2:1.36-1+b1 to 2:1.41-1 when
I first noticed the issue.

Ian.

[Robert Eden]

Looks like it's the Perl DateTime module that is taking the leap second
into account, not the OS.

I don't think they'll be a fix.. for our use, it would be better if they
didn't take it into account!

Robert

[Nick Morrott]

I am a little surprised that a still in support LTS version of Ubuntu doesn't
seem to recognize the leap second that is happening in a few days.

The installed tzdata package version needs to be at least version
2016g to have details of the leap second at the end of the month.
Updated packages for supported versions of Ubuntu are available:

https://launchpad.net/ubuntu/+source/tzdata

Cheers,
Nick