Discussion:
[Xmltv-devel] Duplicate <rating> tags in sd_json output
Ian Campbell
2016-05-23 08:55:39 UTC
Permalink
My mythfilldatabase runs are complaining:

2016-05-23 06:35:03.532635 E  DB Error (programrating insert):
Query was:
INSERT INTO programrating        ( chanid, starttime, system, rating) VALUES (?, ?,    ?,  ?)
Bindings were:
:CHANID=8173, :RATING="TVG", :START=2016-05-26T15:30:00Z,
:SYS="USA Parental Rating"
Driver error was [2/1062]:
QMYSQL3: Unable to execute statement
Database error was:
Duplicate entry '8173-2016-05-26 15:30:00-USA Pare-TVG' for key 'chanid'

chaid=8173 is "BBC ONE East W" AKA xmltvid 24321 and indeed I see there
are two identical <rating> tags in the XML (and two identical
<presenter> FWIW):

  <programme start="20160526153000 +0000" stop="20160526161500 +0000" channel="24321">
    <title>Flog It!</title>
    <sub-title>Sandon Hall 42</sub-title>
    <desc>"Flog It!" [...].</desc>
    <credits>
      <presenter>Paul Martin</presenter>
      <presenter>Paul Martin</presenter>
    </credits>
    <category>Collectibles</category>
    <category>Auction</category>
    <category>Arts/crafts</category>
    <category>Series</category>
    <category>series</category>
    <episode-num system="xmltv_ns">13.44.</episode-num>
    <episode-num system="dd_progid">EP012584290968</episode-num>
    <audio>
      <stereo>stereo</stereo>
    </audio>
    <previously-shown start="20160526000000 +0100" />
    <new />
    <subtitles type="teletext" />
    <rating system="USA Parental Rating">
      <value>TVG</value>
    </rating>
    <rating system="USA Parental Rating">
      <value>TVG</value>
    </rating>
  </programme>

which is reflected in the JSON

[
   {
      "originalAirDate" : "2016-05-26",
      "titles" : [
         {
            "title120" : "Flog It!"
         }
      ],
      "contentRating" : [
         {
            "code" : "TVG",
            "body" : "USA Parental Rating"
         },
         {
            "body" : "USA Parental Rating",
            "code" : "TVG"
         }
      ],
      "programID" : "EP012584290968"

I'm unsure if this is expected/allowed at any level and therefore who
(mythtv, xmltv, sd) is expected to consolidate/tolerate them so I'll st
art by reporting here.

Ian.
Robert Kulagowski
2016-05-23 14:40:38 UTC
Permalink
Post by Ian Campbell
which is reflected in the JSON
[
{
"originalAirDate" : "2016-05-26",
"titles" : [
{
"title120" : "Flog It!"
}
],
"contentRating" : [
{
"code" : "TVG",
"body" : "USA Parental Rating"
},
{
"body" : "USA Parental Rating",
"code" : "TVG"
}
],
"programID" : "EP012584290968"
This is an upstream error; I've contacted Gracenote.
Ian Campbell
2016-05-28 08:42:46 UTC
Permalink
Post by Robert Kulagowski
Post by Ian Campbell
which is reflected in the JSON
[
    {
       "originalAirDate" : "2016-05-26",
       "titles" : [
          {
             "title120" : "Flog It!"
          }
       ],
       "contentRating" : [
          {
             "code" : "TVG",
             "body" : "USA Parental Rating"
          },
          {
             "body" : "USA Parental Rating",
             "code" : "TVG"
          }
       ],
       "programID" : "EP012584290968"
This is an upstream error; I've contacted Gracenote.
FYI there is a bunch more of them now:

Duplicate entry '12384-2016-06-04 22:45:00-Motion P-PG-13' for key 'chanid'
Duplicate entry '12325-2016-06-06 19:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-07 19:00:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-07 19:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-07 23:45:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-08 00:15:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-08 19:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-09 19:00:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-09 23:40:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12325-2016-06-10 19:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-06 20:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-07 20:00:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-07 20:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-08 00:45:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-08 01:15:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-08 20:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-09 20:00:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-10 00:40:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '4027-2016-06-10 20:30:00-USA Pare-TV14' for key 'chanid'
Duplicate entry '12452-2016-06-04 23:45:00-Motion P-PG-13' for key 'chanid'

I suspect we should make xmltv de-dup them.

Ian.
Robert Kulagowski
2016-05-28 12:47:08 UTC
Permalink
Post by Ian Campbell
I suspect we should make xmltv de-dup them.
This is an upstream bug at Gracenote, which I've already notified them
about. But I will be adding de-dupe code to the servers tonight,
because it affects multiple clients.
Ian Campbell
2016-05-29 15:10:21 UTC
Permalink
Post by Robert Kulagowski
Post by Ian Campbell
I suspect we should make xmltv de-dup them.
This is an upstream bug at Gracenote, which I've already notified them
about. But I will be adding de-dupe code to the servers tonight,
because it affects multiple clients.
My last mythfilldatabase run had no duplicates, thanks!

Ian.

Nick Morrott
2016-05-24 00:31:45 UTC
Permalink
INSERT INTO programrating ( chanid, starttime, system, rating) VALUES (?, ?, ?, ?)
:CHANID=8173, :RATING="TVG", :START=2016-05-26T15:30:00Z,
:SYS="USA Parental Rating"
QMYSQL3: Unable to execute statement
Duplicate entry '8173-2016-05-26 15:30:00-USA Pare-TVG' for key 'chanid'
chaid=8173 is "BBC ONE East W" AKA xmltvid 24321 and indeed I see there
are two identical <rating> tags in the XML (and two identical
<programme start="20160526153000 +0000" stop="20160526161500 +0000" channel="24321">
<title>Flog It!</title>
<sub-title>Sandon Hall 42</sub-title>
<desc>"Flog It!" [...].</desc>
<credits>
<presenter>Paul Martin</presenter>
<presenter>Paul Martin</presenter>
</credits>
<category>Collectibles</category>
<category>Auction</category>
<category>Arts/crafts</category>
<category>Series</category>
<category>series</category>
<episode-num system="xmltv_ns">13.44.</episode-num>
<episode-num system="dd_progid">EP012584290968</episode-num>
<audio>
<stereo>stereo</stereo>
</audio>
<previously-shown start="20160526000000 +0100" />
<new />
<subtitles type="teletext" />
<rating system="USA Parental Rating">
<value>TVG</value>
</rating>
<rating system="USA Parental Rating">
<value>TVG</value>
</rating>
</programme>
which is reflected in the JSON
[
{
"originalAirDate" : "2016-05-26",
"titles" : [
{
"title120" : "Flog It!"
}
],
"contentRating" : [
{
"code" : "TVG",
"body" : "USA Parental Rating"
},
{
"body" : "USA Parental Rating",
"code" : "TVG"
}
],
"programID" : "EP012584290968"
I'm unsure if this is expected/allowed at any level and therefore who
(mythtv, xmltv, sd) is expected to consolidate/tolerate them so I'll st
art by reporting here.
Zero or more <rating> elements are permitted for a <programme> element
per the XMLTV DTD.

There's nothing in XMLTV.pm (which is the last place grabber data is
checked before writing to XML) to prevent writing identical (and
useless) <rating> elements. This could (and should) be improved to
ensure that no duplicate elements are output for a programme during
writing and can reach a consuming application like MythTV.

Robert K has stated this issue was an upstream error, so it appears no
software in the entire output chain was programmed to remove such
duplicates from the data output path :)

Cheers,
Nick
Loading...