Posted on Leave a comment

What we did the last couple weeks, technical updates

For about a month now we have been doing big behind the scenes changes – you might’ve noticed when RepArchive was down.

Today we want to clarify what we’ve been doing. This post probably isn’t worth reading for the average user, but for big fans that want to know how RepArchive works and why it was down one or the other time you will learn everything here.

Clean image library & regenerate images

We haven’t cleaned taobao images since 2017. What images would’ve needed to be cleaned? Mainly it’s dead items, links that don’t work anymore because the item got taken down by the seller or taobao.

The reason was a) that it’s a lot of work and b) that when we have we or our users were wondering what the dead link was it would’ve been possible for us to show the product image. But this never got used in reality.

The main advantage we saw was that it would trim the database. All the images are the reason RepArchive is a little slower than Toy-Links.com that only uses a limited set of images.

A second advantage was that when we started in 2017 we didn’t have the image server we have today, the other sites like repladies.co and hwsource.com, and we also didn’t have support for marketplaces other than taobao . Nowadays images are prefixed with tb_ra_ for instance, to indicate that it’s an image from Taobao on Reparchive. For legacy support we ran two variants of our code, one for images with the appropriate prefix and one for one without.

The same thing applied to Imgur QC pics, which included images that weren’t used anymore and images without the appropriate prefix as well.

The cleanest way to make sure unused images are cleaned and the prefix issue is fixed was deleting all images and regenerating them. The majority was done rather quickly, but it took about a week to make sure we got every last image.

The result is that we now have cleaned the database which speeds up our site. All taobao images are up-to-date and tons of broken links were fixed or removed.

Updates limited to a certain time of the day

We noticed that while the items are updating on the site, which takes a few hours, the site is responding considerably slower.

To make it more user friendly we took a good look at our analytics over the last 3 months and determined the time when we have the fewest users on the site. We found out that for 5 hours of the day, the site’s active users are 50% below what they are for the remaining 19 hours.

The updates on the site now only happen during those 5 hours to provide you with a responsive browsing experience.

This does not mean that the remaining hours of the day aren’t used to update the off-page items list with data from taobao, weidian and other sources.

Due to less traffic our other sites, like toy-links, repladies and hwsource update all day.

Thanks for your continuous support and everyone who tells us when something doesn’t work.

Leave a Reply