Shutdown Server Again!

Yesterday in the wee ours of the AM, the server had to be temporarily shutdown due to high CPU usage.

Then later in the day, Yahoo Crawler activated the 60-second throttle 3 times (so at least that is working properly).

With the above, I have made more changes. These are significant and "slick" if I do say so myself.

BEFORE:

• Guest clicks on review link that activates the rvwbuild.cgi script
  
• rvwbuild.cgi would then check the dat file, update file, and shtml file
  
• If any condition merited it, the shtml file was built
  
• If no changes, then rvwbuild.cgi jumped to the existing shtml file
  

The number of links through rvwbuild.cgi were tremendous!!! All searchs, all archives, all reviews (through next and previous). That meant this script was running constantly.

NOW:

• Guest clicks on review link and that takes them to the shtml file PERIOD
  

All the links from searches, archives, and reviews were changed to directly link to the shtml cached file. FAST FOR YOU... please tell me if you notice the speed.

So how do I update the reviews?

Each time someone adds a flight log, or adjusts it in the Flight Log Management page, adds an opinion or rocket specific tip a notice is added to a "refresh" list.

At midnight, the refresh list is read in and all the files in that list are "rebuilt".

Box O Parts Deadline Extended to March 1

We really expected better participation, however, we'll not delay the start too long. Just one more week. Declare your parts and participation.

CLICKERS WANTED: On March 1st, we'll be doing our final preps for the next EMRR on CD. We will need some volunteers to help us click through some old scripts to update the files. E-mail if you can help.

Well, since switching over to our new server, EMRR has dumped them twice. This time they called me!

Bottomline is that EMRR is stealing all of the CPU time. You can read about the fixes on the Blog

Dumped the Server (again!)

Well, since switching over to our new server, EMRR has dumped them twice. This time they called me!

Bottomline is that EMRR is stealing all of the CPU time.

We believe this is from bots/spiders and it looks like despite my robots.txt blocker, they like to get into many areas.

Here were two suspect areas:

1) I had an shtml file that had links to a bunch of CPU intensive site maintenance files accessable only through my myEMRR page. The bots were able to go into that page due to review caching that leaves the link back to myEMRR. I have killed that file from the server and will generate it manually through a script.

2) The bots seem to follow the little "refresh" links in the reviews so that they show the review, follow the refresh, show the review and follow the refresh again and again!

* Numbers after + are successful hits on "robots.txt" files

So, last night I:

1) eliminated the "refresh" link on the reviews

2) changed the self-refresh check on the reviews to only refresh if:
a) there is a new guest opinion - done upon validation
b) there is a new rocket specific tip - done upon validation
c) if a new flight log has been added - this is done at midnight
d) if it is more than 15 days old (catch all; will pickup RockSIM, Rec Motors)

Today I:

1) changed the arcbuild to create cached shtml files and to only refresh upon changes to the master database. This will improve the speed of the site for you and reduce CPU time (for me)

Tomorrow I will:

1) Implement a bot detection throttle which will prevent the two main scripts from running more than 10-15 requests from the same IP within 1 minute

If anyone has any experience in reducing CPU time via perl scripting, please let me know.

2/22/09 Update:

I spent a lot of time doing the following:

1) eliminated the "refresh" link in the pages to prevent bot cycles
2) adjusted the arcbuild script to create cache pages that only refresh upon changes (faster for you)
3) eliminated a double load of a who's on counter file
4) implemented a 60-second pause page for any IP that hits the site > 10 within 1 minute
5) improved robots.txt using a professional to ID false bots to block them
6) eliminated an admin file that allowed the BOT to run some very intensive scripts that are used periodically to check the sites database/integrity

Stories and Tips

Within a month of starting EMRR, I had initiated Guest Experiences (February 1998). This was a collection of comments about various rockets and experiences in rocketry. The introduction of this feature on EMRR was simply a text-based page that folks could read, as is.

In November 1998, Mike Kent said he’d help with a project of separating the Guest Experiences into two separate databases: Guest Stories and Hints/Tips. This allowed EMRR to introduced the Hints/Tips database by the end of November 1998. EMRR has no knowledge of where Mike is or what he is up to, but his initial separating work has lasted more than 10 years on EMRR.

hint (hint) an indirect, covert, or helpful suggestion;

tip (tip) a useful hint or idea; a basic, practical

By April 1999, EMRR had flipped the text files into databases and implemented searching and a form that allowed EMRR guests to submit input into the Stories and Tips databases.

The next set of updates came in 2002 when EMRR had amassed 256 Stories and 253 Hints/Tips. Starting in March, EMRR further split the growing Hints/Tips database into generic Tips and Rocket Specific Tips. These Rocket Specific Tips were then placed at the bottom of each of the dynamically built review pages. In April, just for some visual content, EMRR collected a bunch of pictures for the various story categories and added a random picture selection to the Stories display pages. (I regret not asking for pictures from the very beginning.)

Specific Rocket Tips are observations, hints, tips and even some modifications that guests have made to specific rocket kits. Guests share their advice and this is shown along with the review, so that when folks are reading the review or building the same rocket they can have some extra guidance from others that have experience with the kit. These have been a real success and we continue to see growth.



The next evolution in Stories and Tips came in December 2004 and January 2006, respectively. At these times, EMRR introduced the first Featured Story and Featured Tip. What makes them featured? More content and a series of pictures.

fea-tured (fee'chuhrd) made a feature or highlight; given prominence

There are presently 29 Featured Stories. The most recent additions are stories that support a Youth Participation/Outreach Contest that EMRR held in 2008.

In June 2007, EMRR updated the Featured Tips page to present similar tips from the database at the bottom of the Featured Tip being displayed. In addition, there were a number of scratch articles that were really tips, so EMRR moved them over to the tips database. Presently there are 57 Featured Tips.

All-in-all, Stories and Hints/Tip (especially Rocket Specific Tips) have been a great added feature of EMRR.

Next Up: Flight Logs

EMRR Script Directory Pointer Changes

If you point to reviews and other items on EMRR you may need to make the following changes. To make global changes, you should consider using a global search and replace tool. Here's a free one http://www.digitalvolcano.co.uk/textcrawler.html

Change cgi-bin/rocksim/ to cgi-bin/resources/

Change cgi-bin/rvwbuild/ to cgi-bin/reviews/

Change cgi-bin/arcbuild/ to cgi-bin/reviews/

The old directories will go away on 3/1

For example: The old directory pointer to a review was:

"cgi-bin/rvwbuild/rvwbuild.cgi?data...."

Only the directory changed to "reviews" so it will become:

"cgi-bin/reviews/rvwbuild.cgi?data..."

Box O Parts Deadline Approaching

If you are going to play you need to declare your participation and parts...tick-tock!

General update of a nice selection of reviews.

Some directory simplification taking place which may effect your links into EMRR (read the Blog and eList Newsletter).

Elevate Eleven Idea: Drag race with 11 rockets...

A General Update (for you!)

It is time to make your declarations for the new Box O' Parts contest. Outside of a typical update, I've been busy writing the scripts needed for the next EMRR on CD.