LJFM rides again

[feren]

Okay, does anybody else remember the wonderful piece of work that skx put together and called the LiveJournal Friends Monitor? If you do, you might remember that it was forced offline by LiveJournal around October 1st of 2003. I really missed LJFM because it automated a number of tasks for me and simplified how I kept track of everyone who was friending and unfriending me on LiveJournal. Ultimately it was sort of one of those "set it and forget it" systems, I told it to alert me of changes and never heard from it again unless there was something to report.

Well, to paraphrase Milhouse, LJFM is back.... in pog form. Thanks to Steve's generosity I was able to retrieve the source code to the LJFM and, after a bit of tinkering, I got the database built out and had the scripts up and running on my new server. So here's the deal: if you like the features of marnanel's Joule the friends-list analyser but hate having to manually run the Joule script every day you should go over to the newly reincarnated LJFM at http://ljfm.black-panther.us and get signed up. The system will take two days to start tracking and notifying you of changes -- this is because the system has to pull down the list of who has friended you on the first day, and has to pull down a fresh list on the second day to compare against for changes.

All of you are welcome to use the reincarnated LJFM, but please don't pass it around too much because I want to keep the load it generates on LJ to a minimum. I think when the site was running on Steve's server he had something like 6,000 users and it definately got LiveJournal's attention in a bad way. I'd like to avoid a similar situation so my server doesn't get banned like Steve's did.

Speaking of generosity, if you like the LJFM I suggest you throw a couple of dollar's Steve's way as a "thank you" for putting it all together originally. He didn't have to share the source code after he got shut down, but he did.

Calling all angels

Comments

[youngvanwinkle.livejournal.com]

There's an obvious way to minimize the load this puts on LJ, though I don't know whether you'd have the knowledge of the source to rig something like this up.

When someone signs up, have the service automatically assign them a minute of the day that no other user is assigned. Then it checks that user's info on that minute every day.

This way, each user gets checked on a different minute, instead of checking all of them at the same time. It distributes the load across the entire day, which would mean it'd take a lot more users for the service to initiate something resembling a DoS attack.

[frysco.livejournal.com]

When someone signs up, have the service automatically assign them a minute of the day that no other user is assigned. Then it checks that user's info on that minute every day.

Might actually want to make this a particular second of a day rather than minute. There are 1440 minutes in a day, and with 6000 users of a system, you would get overlaps. If someone was really log watching, then seeing multiple requests from the same server - even if it's just 4 at a time - on the same minute to within just a few seconds could raise eyebrows.

By going to seconds, you have 86400 slots to play around with, and spreading around the slots could mean that a request goes in once every 14 seconds. Which is less noticeable.

Of course, it only scales so far before you get into the same problem as using minutes, but based on what Q@ said about skx having 6000 users, that might not be for a while.

[feren.livejournal.com]

I'm thinking that if I ever get to a number of subscribers that I have to start worrying about drawing undo attention to my server I will rework the code (it's pretty straightforward and hackable, Steve documented the hell out of its internals like a good coder should) so that it runs through users in alphabetical order, with two letters each hour. So the run schedule would be like:

0000 to 0100 hours: Livejournal accounts beginning with A or B

0100 to 0200 hours: Livejournal accounts beginning with C or D

0300 to 0400 hours: Livejournal accounts beginning with E or F

... and so on.

[skx.livejournal.com]

That's a stunningly simple idea; which would distribute the load perfectly.

(OK it's not perfect, there do seem to be clear frequency differences.. (http://www.steve.org.uk/Software/Livejournal/Usernames/index.php))

Thanks for the suggestion, I'll remember that.