LJFM rides again

[feren]

Okay, does anybody else remember the wonderful piece of work that skx put together and called the LiveJournal Friends Monitor? If you do, you might remember that it was forced offline by LiveJournal around October 1st of 2003. I really missed LJFM because it automated a number of tasks for me and simplified how I kept track of everyone who was friending and unfriending me on LiveJournal. Ultimately it was sort of one of those "set it and forget it" systems, I told it to alert me of changes and never heard from it again unless there was something to report.

Well, to paraphrase Milhouse, LJFM is back.... in pog form. Thanks to Steve's generosity I was able to retrieve the source code to the LJFM and, after a bit of tinkering, I got the database built out and had the scripts up and running on my new server. So here's the deal: if you like the features of marnanel's Joule the friends-list analyser but hate having to manually run the Joule script every day you should go over to the newly reincarnated LJFM at http://ljfm.black-panther.us and get signed up. The system will take two days to start tracking and notifying you of changes -- this is because the system has to pull down the list of who has friended you on the first day, and has to pull down a fresh list on the second day to compare against for changes.

All of you are welcome to use the reincarnated LJFM, but please don't pass it around too much because I want to keep the load it generates on LJ to a minimum. I think when the site was running on Steve's server he had something like 6,000 users and it definately got LiveJournal's attention in a bad way. I'd like to avoid a similar situation so my server doesn't get banned like Steve's did.

Speaking of generosity, if you like the LJFM I suggest you throw a couple of dollar's Steve's way as a "thank you" for putting it all together originally. He didn't have to share the source code after he got shut down, but he did.

Calling all angels

Comments

[youngvanwinkle.livejournal.com]

There's an obvious way to minimize the load this puts on LJ, though I don't know whether you'd have the knowledge of the source to rig something like this up.

When someone signs up, have the service automatically assign them a minute of the day that no other user is assigned. Then it checks that user's info on that minute every day.

This way, each user gets checked on a different minute, instead of checking all of them at the same time. It distributes the load across the entire day, which would mean it'd take a lot more users for the service to initiate something resembling a DoS attack.

[frysco.livejournal.com]

When someone signs up, have the service automatically assign them a minute of the day that no other user is assigned. Then it checks that user's info on that minute every day.

Might actually want to make this a particular second of a day rather than minute. There are 1440 minutes in a day, and with 6000 users of a system, you would get overlaps. If someone was really log watching, then seeing multiple requests from the same server - even if it's just 4 at a time - on the same minute to within just a few seconds could raise eyebrows.

By going to seconds, you have 86400 slots to play around with, and spreading around the slots could mean that a request goes in once every 14 seconds. Which is less noticeable.

Of course, it only scales so far before you get into the same problem as using minutes, but based on what Q@ said about skx having 6000 users, that might not be for a while.

[feren.livejournal.com]

I'm thinking that if I ever get to a number of subscribers that I have to start worrying about drawing undo attention to my server I will rework the code (it's pretty straightforward and hackable, Steve documented the hell out of its internals like a good coder should) so that it runs through users in alphabetical order, with two letters each hour. So the run schedule would be like:

0000 to 0100 hours: Livejournal accounts beginning with A or B

0100 to 0200 hours: Livejournal accounts beginning with C or D

0300 to 0400 hours: Livejournal accounts beginning with E or F

... and so on.

[skx.livejournal.com]

That's a stunningly simple idea; which would distribute the load perfectly.

(OK it's not perfect, there do seem to be clear frequency differences.. (http://www.steve.org.uk/Software/Livejournal/Usernames/index.php))

Thanks for the suggestion, I'll remember that.

[jadine.livejournal.com]

I'm trying to unsubscribe from LJFM, because my client doesn't parse the HTML emails correctly. However, the remove link in the email doesn't work! (I've tried cutting and pasting it, and it just can't find that page.) Can you unsubscribe me?

Additionally, there's no contact info on the webpage. I had to go find Steve's post about it, which linked to here. You might want to add an email address or something.

[feren.livejournal.com]

[However, the remove link in the email doesn't work! (I've tried cutting and pasting it, and it just can't find that page.)]

It worked just fine for me when I ran it through testing earlier this week. That's really odd! Can you do me an immense favor and e-mail me (via feren -at- livejournal-dot-com) the link the e-mail updates were giving you? I'd really appreciate it because it would help me further dissect this issue and determine where things went awry.

[Can you unsubscribe me?]

No problem! You're officially out of the system as of now.

[You might want to add an email address or something.]

Mmm, good point. I generally dislike putting email addresses on the web, it makes life so much easier for spammers when they can send a bot out to harvest innocent addresses from the web. I thought that if anybody had an issue they'd just mail me back at the address the updates are sent from -- but I forget I don't think like everyone else. I appreciate the feedback and will work on those issues.

Incidentally, if I built the system out to send non-HTML e-mails would you be interested in rejoining?

[jadine.livejournal.com]

Emailed the link.

Some of us just assume that automatically generated email won't include a reply-to address that's seen by a human :)

I would be interested if you set it up for plaintext emails. For now, I'll just keep checking Joule.

[skx.livejournal.com]

I spotted a problem in the past with some people saying that they couldn't get either the "validate" or the "remove" links working.

In the end I tracked this down to the behaviour of their web browsers - as you can see the way these scripts work is that they generate a request to a file that isn't found; which would normally generate a 404 error.

Instead the .htaccess file invokes the validate/remove script and all is well.

It appears that some browsers will get a 404 error some of the time; I believe this is related to the IE setting "show friendly HTTP error messages", but some people seemed to see this because they had strange IE plugins loaded.

With regard to the email address; I used a real address for a long time, but then switched to a dedicated account so that I didn't get bothered with bounces. (I recieved 20-50 a day on average).

I think it'd be nice to extend the code to handle this magically, and set the from address to be "bounce-user-host@sending-domain.com", but I didn't have time.