[Xmltv-devel] Hosting for supplement.xmltv.org

[Mattias Holmlund]

Hi everyone!

As some of you might be aware, there is a service called
supplement.xmltv.org that supplies metadata files to the following grabbers:

tv_grab_be
tv_grab_ch_search
tv_grab_eu_epgdata
tv_grab_fr_kazer
tv_grab_huro
tv_grab_it
tv_grab_it_dvb
tv_grab_uk_bleb
tv_grab_uk_rt

The metadata files are updated automatically from CVS (by a cronjob that
I control). This means that we can push out metadata changes to grabbers
automatically without forcing users to upgrade to a new version of xmltv.

The supplement service is provided by two different servers
(supplement1.xmltv.org and supplement2.xmltv.org) run by Robert Eden and
Marc Feist. The reason that we have two servers is for load-balancing
and redundancy. The servers are plain http servers carefully setup to
make sure that they both respond in the exact same way so that
http-caching works even when users are automatically switched between
servers.

Yesterday Marc sent me an email saying that he will loose control over
the server running supplement2.xmltv.org. I think we need to find a
replacement server for it or maybe switch to another hosting setup.

Our options are:

- Find someone else who can host supplement2.xmltv.org
- Find someone who can reliably host the entire supplement.xmltv.org
with high availability. Maybe someone wants to pay for hosting with
Amazon S3 or a similar service?

I don't have access to logs on either of the supplement servers, so I
can't tell you how much traffic we are talking about. Maybe Robert or
Marc knows?

I can keep my cronjob running and switch it to a different host, but if
someone wants to take over that as well it would be more than welcome.

/Mattias

[Ben Bucksch]

Post by Mattias Holmlund
- Find someone else who can host supplement2.xmltv.org

If there are no elborate / dangerous scripts to run to get the content,
and there's no other solution, I can host it.

Ben

[Carsten Aulbert]

Hi

Post by Ben Bucksch
If there are no elborate / dangerous scripts to run to get the content,
and there's no other solution, I can host it.

Also, if the initial setup is relatively straight -forward and traffic
is not too much, I can host this also (so far I was more on the winning
end of xmltv, so feel free to ask to give back :))

Do you have more info on traffic, number of requests per day, etc?

Cheers

Carsten

[Robert Eden]

Post by Carsten Aulbert

Post by Ben Bucksch
If there are no elborate / dangerous scripts to run to get the content,
and there's no other solution, I can host it.

Also, if the initial setup is relatively straight -forward and traffic
is not too much, I can host this also (so far I was more on the winning
end of xmltv, so feel free to ask to give back :))
Do you have more info on traffic, number of requests per day, etc?

I'm pretty sure the sync method is just a rsync over ssh. It doesn't
take a lot of server resources.. Heck, all of XMLTV.ORG is one of many
domain's on my brothers small VPS.

It looks like I didn't have specific supplement statistics turned on,
but all of XMLTV.ORG used 2G of transfer in May.

I can probably host the second supplement on schedulesdirect.org too.

Mattias can name the winner. :) We could even have 3!

Robert

[Nick Morrott]

Post by Carsten Aulbert

Post by Ben Bucksch
If there are no elborate / dangerous scripts to run to get the content,
and there's no other solution, I can host it.

Also, if the initial setup is relatively straight -forward and traffic
is not too much, I can host this also (so far I was more on the winning
end of xmltv, so feel free to ask to give back :))
Do you have more info on traffic, number of requests per day, etc?

I'm pretty sure the sync method is just a rsync over ssh.  It doesn't
take a lot of server resources.. Heck, all of XMLTV.ORG is one of many
domain's on my brothers small VPS.
It looks like I didn't have specific supplement statistics turned on,
but all of XMLTV.ORG used 2G of transfer in May.
I can probably host the second supplement on schedulesdirect.org too.
Mattias can name the winner. :)  We could even have 3!

Any chance we could increase the CVS sync frequency (hourly or even
half-hourly)?

With more users and more grabbers using the supplement service, the
more important it is to get the updates to CVS to end-users in as
timely manner as possible.

Cheers,
Nick

[Mattias Holmlund]

Post by Nick Morrott
Any chance we could increase the CVS sync frequency (hourly or even
half-hourly)?

I've changed it to half-hourly. I just hope that CVS will not start
taking too much time causing my cronjob to bite itself in the tail. (Did
I mention that I don't trust CVS...)

/Mattias

[Nick Morrott]

Post by Mattias Holmlund

Post by Nick Morrott
Any chance we could increase the CVS sync frequency (hourly or even
half-hourly)?

I've changed it to half-hourly. I just hope that CVS will not start

Thank you!

Post by Mattias Holmlund
taking too much time causing my cronjob to bite itself in the tail. (Did
I mention that I don't trust CVS...)

Fingers crossed the change doesn't cause any issues for any of our
kind supplement hosts. If it is a problem, I'm sure hourly would be
fine.

Cheers,
Nick

[Carsten Aulbert]

Hi Robert et al.

Post by Robert Eden
I'm pretty sure the sync method is just a rsync over ssh. It doesn't
take a lot of server resources.. Heck, all of XMLTV.ORG is one of many
domain's on my brothers small VPS.
It looks like I didn't have specific supplement statistics turned on,
but all of XMLTV.ORG used 2G of transfer in May.
I can probably host the second supplement on schedulesdirect.org too.

OK, that looks like piece of cake then. I was more worried if it was in
the high 100GB range per month.

So count me in if you want me (dedicated rented root server in Germany).
Do you run GeoIP or something similar for world-wide load balancing?

Cheers

Carsten

[Robert Eden]

Post by Carsten Aulbert
OK, that looks like piece of cake then. I was more worried if it was in
the high 100GB range per month.
So count me in if you want me (dedicated rented root server in Germany).
Do you run GeoIP or something similar for world-wide load balancing?

Nothing that fancy. just multiple DNS entries for the name.

Mattias, what do you think about instead of using rsync against a master
server, each server runs your script to update from cvs (at increased
frequency)? That way each server is totally independent.

Robert

[Karl Dietz]

Post by Robert Eden
Mattias, what do you think about instead of using rsync against a master
server, each server runs your script to update from cvs (at increased
frequency)? That way each server is totally independent.

How does that fit in with moving the repository from CVS to GIT? I'd
rather not let people setup a CVS sync job at their servers and then
pull the plug on the CVS repo :)

Searching for the procedure at sourceforge lead me to this
http://sourceforge.net/apps/trac/sourceforge/ticket/22146

[Robert Eden]

Post by Karl Dietz
How does that fit in with moving the repository from CVS to GIT? I'd
rather not let people setup a CVS sync job at their servers and then
pull the plug on the CVS repo :)

Personally, I'm not a fan of moving to git... it would be a lot of work,
and I don't see much benefit to us. A more active project? Maybe... but
not us.

Should SF shut down CVS, I would prefer SVN.... I'm more familiar with
it. I use git a little at work, and I find the commit/pull/push thing
annoying. I can see the benefit for projects with lots of forks, but
it's not needed for the stuff I work on.

I don't do a lot of the commits here, so if the active folks really
want git their opinions matter more than mine... I just don't see the point.

If we did move off CVS, the sync job probably wouldn't be difficult to
redo. MKDIST would be the biggest change.

Robert

[Jan Schneider]

Post by Robert Eden

Post by Karl Dietz
How does that fit in with moving the repository from CVS to GIT? I'd
rather not let people setup a CVS sync job at their servers and then
pull the plug on the CVS repo :)

Personally, I'm not a fan of moving to git... it would be a lot of work,
and I don't see much benefit to us. A more active project? Maybe... but
not us.
Should SF shut down CVS, I would prefer SVN.... I'm more familiar with
it. I use git a little at work, and I find the commit/pull/push thing
annoying. I can see the benefit for projects with lots of forks, but
it's not needed for the stuff I work on.
I don't do a lot of the commits here, so if the active folks really
want git their opinions matter more than mine... I just don't see the point.
If we did move off CVS, the sync job probably wouldn't be difficult to
redo. MKDIST would be the biggest change.

Actually, I have set up a local Git mirror of XMLTV CVS, because it's
much easier to develop with Git, once you get to it. Completely
independent from forking features, which is just about collaboration.

OTOH this setup works well for me, I don't do much XMLTV development
anyway, and XMLT is a sufficiently small project so that it could keep
working fine with CVS for many more years. Just my 2 cents.

[Mattias Holmlund]

Post by Robert Eden

Post by Carsten Aulbert
OK, that looks like piece of cake then. I was more worried if it was in
the high 100GB range per month.
So count me in if you want me (dedicated rented root server in Germany).
Do you run GeoIP or something similar for world-wide load balancing?

Nothing that fancy. just multiple DNS entries for the name.
Mattias, what do you think about instead of using rsync against a master
server, each server runs your script to update from cvs (at increased
frequency)? That way each server is totally independent.

That won't work with the current setup. Since the DNS returns multipe
entries for supplment.xmltv.org, it is up to the client to choose an IP
address at random. This means that the first time the client may downlod
from supplement1 and the second time from supplement2. The http-caching
mechanism relies on last-modified timestamps to know if content has
changed and needs to be downloaded. This means that the timestamps on
the files need to be the same on all servers. Therefore, I must generate
the files in one place and rsync them to all servers.

/Mattias

[Mattias Holmlund]

Post by Carsten Aulbert
Hi Robert et al.

Post by Robert Eden
I'm pretty sure the sync method is just a rsync over ssh. It doesn't
take a lot of server resources.. Heck, all of XMLTV.ORG is one of many
domain's on my brothers small VPS.
It looks like I didn't have specific supplement statistics turned on,
but all of XMLTV.ORG used 2G of transfer in May.
I can probably host the second supplement on schedulesdirect.org too.

OK, that looks like piece of cake then. I was more worried if it was in
the high 100GB range per month.
So count me in if you want me (dedicated rented root server in Germany).
Do you run GeoIP or something similar for world-wide load balancing?

Germany sounds good since I assume that Roberts server is in the US.

Could you configure a virtual host that responds to supplement.xmltv.org
and supplement3.xmltv.org (makes testing easier), configure it with no
ETag header ("FileETag None" if you are using apache) and allow
directory listings. I will send you a public ssh-key that I will use to
upload data via rsync.

/Mattias

[Ben Bucksch]

Post by Robert Eden
I'm pretty sure the sync method is just a rsync over ssh.

Perfect, works for me. Traffic is no issue for me, either.

Contact me off-list with what you need and what I need to do.

[Mattias Holmlund]

The new supplement server, kindly hosted by Carsten Aulbert, is now up
and running. Traffic is load-balanced between that server and the server
run by Robert Eden. Please report any problems to the mailing-list.

I want to thank Marc Feist for running a supplement server for us for
over three years. Marc, you can now take your server off-line.

/Mattias

Post by Mattias Holmlund
Hi everyone!
As some of you might be aware, there is a service called
tv_grab_be
tv_grab_ch_search
tv_grab_eu_epgdata
tv_grab_fr_kazer
tv_grab_huro
tv_grab_it
tv_grab_it_dvb
tv_grab_uk_bleb
tv_grab_uk_rt
The metadata files are updated automatically from CVS (by a cronjob
that I control). This means that we can push out metadata changes to
grabbers automatically without forcing users to upgrade to a new
version of xmltv.
The supplement service is provided by two different servers
(supplement1.xmltv.org and supplement2.xmltv.org) run by Robert Eden
and Marc Feist. The reason that we have two servers is for
load-balancing and redundancy. The servers are plain http servers
carefully setup to make sure that they both respond in the exact same
way so that http-caching works even when users are automatically
switched between servers.
Yesterday Marc sent me an email saying that he will loose control over
the server running supplement2.xmltv.org. I think we need to find a
replacement server for it or maybe switch to another hosting setup.
- Find someone else who can host supplement2.xmltv.org
- Find someone who can reliably host the entire supplement.xmltv.org
with high availability. Maybe someone wants to pay for hosting with
Amazon S3 or a similar service?
I don't have access to logs on either of the supplement servers, so I
can't tell you how much traffic we are talking about. Maybe Robert or
Marc knows?
I can keep my cronjob running and switch it to a different host, but
if someone wants to take over that as well it would be more than welcome.
/Mattias