Website file limit

[Pete Eeles]

Hi all,

I was reminded during the UKB social event to take another look at just how close UKB is to the hard limit of 300,000 files imposed by the service provider (and which is, I believe, a software limit). I thought we had at least another year to go before having to take action, but found that we were at 295,732 files. In other words, we were going to hit the limit within a month. The only saving grace is that I was wearing brown trousers as this figure flashed up on my screen ... And so, I've had to bring forward the inevitable and have spent the last few days thinking about options and implementing a resolution.

First and foremost, NO FILES HAVE BEEN LOST. They are all backed up on my machine at home, on two external hard drives, and in the cloud.

The vast majority of files are a result of attachments to posts (usually images). Based on various discussions I've had, I believe that the long-term solution is for files to be hosted on the cloud (using, for example, Amazon S3) where the 300,000 limit won't apply, for UKB to save post attachments there, and for UKB to link to these from posts. However, implementing that will take some time.

In the meantime, I've archived (removed from the server) all post attachments up until the end of 2012. No posts have been deleted, however, since we still want these to be searchable for interesting info. when needed. The result is that you will find images missing if they are associated with posts before 2013. Following this "intervention", we are now at 249533 files, a total reduction of around 45,000 files. I will, of course, be monitoring things more closely moving forward. I'm now thinking about a mechanism for reinstating (even if temporarily) missing images; watch this space.

To ensure that we still have access to the personal diary content, with images, I've created a PDF file for all members with a personal diary for each year (2010, 2011, 2012) and a summary of these PDFs can be found at: http://www.ukbutterflies.co.uk/forum_diaries.php. This link is accessible under the "Community" menu, named "Personal Diaries (archive)".

As usual, all comments/thoughts welcome.

Cheers,

- Pete

[Neil Freeman]

Just spotted this post Pete.

Strangely enough, I was wondering recently if there was a simple way to convert my PD to a format that I could save on a hard drive...It would have to be simple because I am not that IT savvy.
Do you have any plans to convert any other years into PDFs?

Cheers,

Neil.

[Pete Eeles]

Do you have any plans to convert any other years into PDFs?

Hi Neil,

I've written a simple mechanism that supports a yearly summary of a PD for any member. I've not made it generally available since it isn't optimised and causes a performance problem when too many members try and use it at the same time.

However, given your comment, it might be worth me completing the exercise for all years, even if they're still available online! It just takes a bit of time. Would this be of interest?

Cheers,

- Pete

[Neil Freeman]

It would certainly be of interest to me Pete, and I would imagine a few other members too.

Cheers,

Neil.

[David M]

Yes, I agree. This would be a welcome option.

What about removing images from some of the historic 'Sightings' threads? I doubt many people refer back to these and it would free up some space.

[Pete Eeles]

Will look into it!

Thanks for the thoughts, David - the problem is that 90% of the files are in personal diaries and I feel that focusing on other areas would be "dancing around the edges"!

Cheers,

- Pete

[David M]

Ah, I see (although I'm surprised that the proportion is SO high).

Looks like PD redaction is the only real option then. Any moves to preserve earlier pages in any format would be useful.

[Pete Eeles]

Hi all,

Just to let you know that we hit the 300,000 file limit on the server last week, and so I've now archived the 2013 and 2014 personal diaries by creating PDFs of the content. They can be found under the "Community -> Personal Diaries (archive)" menu, which will take you here:

http://www.ukbutterflies.co.uk/forum_diaries.php

I have all files backed up and, at some point, hope to put them all back once I figure out how to do that by using a separate service. Sorting that out will take time, but is something on my list for the end of this year.

Any questions, please shout!

Cheers,

- Pete

[NickMorgan]

Pete,
I hadn't realised you had had to do this until I e-mailed you when I discovered that my pictures had disappeared out of the early pages on my Personal Diary. I hadn't even found the "Website Comments" section until just now.
I just wanted to thank you for the enormous amount of work you put in to keep this forum up and running.
Nick

[Pete Eeles]

Thanks Nick - all part of the service

Cheers,

- Pete

[PhilBJohnson]

Thanks for the info.
I must have missed a conversation in 2017 !
I thought that images scaled, reduced in size to save space (pixels closer together, were thought to remain more detailed, than picture files with fewer pixels to save space). Thumbnail size images were considered, to help diary text flow.

[Pete Eeles]

This is not related to the size of any file, but the NUMBER of files, and we have a hard limit of 300,000 files. Around 25,000 images are in the species albums and, of the remainder, the vast majority are in personal diaries - hence their 'culling'. And, despite my pleas for contributors to restrict their posts so that they don't post similar images, where a subject has moved 1mm (!!!), this has been completely ignored by some of the more prolific posters, which is frustrating to say the least.

Cheers,

- Pete

[PhilBJohnson]

I was reminded during the UKB social event to take another look at just how close UKB is to the hard limit of 300,000 files imposed by the service provider (and which is, I believe, a software limit).

Thanks Pete for being "in-house" with your service (or software) provider.
Another competing service provider might have allowed the number of photographic images, based on a calculation of average camera picture file size.
As "we all" wanted to win that photographic image prize, or have our images enlarged enough to hang on a wall, camera technology "evolved" to capture more data (pixels closer together) with larger file sizes.
As an example, this led to someone's iCloud housing more data (including slow motion videos at 240 frames a second).
The tech. advancers for remote storage might have thought how much they could compress a picture file for storing it remotely (upload), to save space, then be able to reproduce it accurately again on downloaded from "cloud" on demand (which ever cloud that might have been).
Kind Regards and
best wishes for the future.