FOK! Audioscrobbler / Last.fm Group XLVIII

quote:

Op maandag 18 juli 2011 10:16 schreef Solique het volgende:

[..]

last.fm kaput.

quote:

Op maandag 18 juli 2011 13:07 schreef Desunoto het volgende:
Komt wel weer goed.

quote:

Op maandag 18 juli 2011 13:18 schreef willemwaus het volgende:
Hoop 't, had mijne eind maart al per ongeluk gereset, nu weer hard bezig richting die 150k plays waar ik op stond

quote:

Op maandag 18 juli 2011 13:37 schreef Solique het volgende:

[..]

Niemand sinds een tijdje

quote:

Library and streaming services outage
Monday, 18 July 2011

Since 04:00 GMT on Sunday morning, the primary Radb service has been exhibiting intermittent problems meeting acceptable service levels. This means that libraries, scrobble counts and the services associated with them (stats, radio stations etc) appear to be broken when you use Last.fm.

We’re working very hard to fix this as soon as possible, and we’ve had engineers on it throughout yesterday and last night, but we wanted to keep you posted here with what’s happened and what we know so far.

All of our services here at Last fm are predicated around keeping track of your accumulated music plays – your scrobbles – and using statistical methods on the historical data to build awesome things.

Because scrobbles power everything dynamic and wonderful on the site, we need fast realtime access to the scrobbles and associated summary data ( library, charts, neighbourhoods, etc. ). And with milions of people scrobbling, and the size of the historical scrobbles set, doing this fast in realtime, while updating the same sets with new plays as fast as can be managed is a significant challenge. We have a custom, in memory database service that we designed and implemented to address these needs. It is called Radb. It is usually pretty awesome at answering these sorts of questions in a predictably constant time.

But since 4am on Sunday Radb has been failing. The reasons for this are unclear. Failing over to the redundant service providers helps a little, but not enough. The reasons for this are also unclear. We have engineers working flat out to diagnose this problem.

To remove as much stress as we can from the database service layer, we have additionally disabled most of our radio, library and recommendation services, temporarily. We have also stopped accepting new scrobbles for the meantime, for a similar reason. Scrobbling clients will detect this state and will cache your scrobbles until the submission service is reactivated, so no need to panic about scrobbles being lost.

Keep an eye on the status page, and the appropriate forum posts. We’ll update with more information and estimates as we have them.