Website file limit

Comments on the website and discussion forums, including enhancement requests and bug reports.
User avatar
Pete Eeles
Administrator & Stock Contributor
Administrator & Stock Contributor
Posts: 6164
Joined: Tue Jan 17, 2006 6:10 pm
Location: Thatcham, Berkshire
Contact:

Website file limit

Postby Pete Eeles » Tue Nov 28, 2017 1:20 pm

Hi all,

I was reminded during the UKB social event to take another look at just how close UKB is to the hard limit of 300,000 files imposed by the service provider (and which is, I believe, a software limit). I thought we had at least another year to go before having to take action, but found that we were at 295,732 files. In other words, we were going to hit the limit within a month. The only saving grace is that I was wearing brown trousers as this figure flashed up on my screen ... And so, I've had to bring forward the inevitable and have spent the last few days thinking about options and implementing a resolution.

First and foremost, NO FILES HAVE BEEN LOST. They are all backed up on my machine at home, on two external hard drives, and in the cloud.

The vast majority of files are a result of attachments to posts (usually images). Based on various discussions I've had, I believe that the long-term solution is for files to be hosted on the cloud (using, for example, Amazon S3) where the 300,000 limit won't apply, for UKB to save post attachments there, and for UKB to link to these from posts. However, implementing that will take some time.

In the meantime, I've archived (removed from the server) all post attachments up until the end of 2012. No posts have been deleted, however, since we still want these to be searchable for interesting info. when needed. The result is that you will find images missing if they are associated with posts before 2013. Following this "intervention", we are now at 249533 files, a total reduction of around 45,000 files. I will, of course, be monitoring things more closely moving forward. I'm now thinking about a mechanism for reinstating (even if temporarily) missing images; watch this space.

To ensure that we still have access to the personal diary content, with images, I've created a PDF file for all members with a personal diary for each year (2010, 2011, 2012) and a summary of these PDFs can be found at: http://www.ukbutterflies.co.uk/forum_diaries.php. This link is accessible under the "Community" menu, named "Personal Diaries (archive)".

As usual, all comments/thoughts welcome.

Cheers,

- Pete

User avatar
Neil Freeman
Posts: 2550
Joined: Fri Jul 23, 2010 6:25 pm
Location: Solihull, West Midlands

Re: Website file limit

Postby Neil Freeman » Fri Dec 01, 2017 7:53 pm

Just spotted this post Pete.

Strangely enough, I was wondering recently if there was a simple way to convert my PD to a format that I could save on a hard drive...It would have to be simple because I am not that IT savvy.
Do you have any plans to convert any other years into PDFs?

Cheers,

Neil.

User avatar
Pete Eeles
Administrator & Stock Contributor
Administrator & Stock Contributor
Posts: 6164
Joined: Tue Jan 17, 2006 6:10 pm
Location: Thatcham, Berkshire
Contact:

Re: Website file limit

Postby Pete Eeles » Sat Dec 02, 2017 12:12 am

Neil Freeman wrote:Do you have any plans to convert any other years into PDFs?


Hi Neil,

I've written a simple mechanism that supports a yearly summary of a PD for any member. I've not made it generally available since it isn't optimised and causes a performance problem when too many members try and use it at the same time.

However, given your comment, it might be worth me completing the exercise for all years, even if they're still available online! It just takes a bit of time. Would this be of interest?

Cheers,

- Pete

User avatar
Neil Freeman
Posts: 2550
Joined: Fri Jul 23, 2010 6:25 pm
Location: Solihull, West Midlands

Re: Website file limit

Postby Neil Freeman » Sun Dec 03, 2017 7:59 pm

It would certainly be of interest to me Pete, and I would imagine a few other members too.

Cheers,

Neil.

User avatar
David M
Posts: 7807
Joined: Tue Aug 11, 2009 8:17 pm
Location: South Wales

Re: Website file limit

Postby David M » Tue Dec 05, 2017 7:10 pm

Yes, I agree. This would be a welcome option.

What about removing images from some of the historic 'Sightings' threads? I doubt many people refer back to these and it would free up some space.

User avatar
Pete Eeles
Administrator & Stock Contributor
Administrator & Stock Contributor
Posts: 6164
Joined: Tue Jan 17, 2006 6:10 pm
Location: Thatcham, Berkshire
Contact:

Re: Website file limit

Postby Pete Eeles » Wed Dec 06, 2017 4:34 pm

Will look into it!

Thanks for the thoughts, David - the problem is that 90% of the files are in personal diaries and I feel that focusing on other areas would be "dancing around the edges"!

Cheers,

- Pete

User avatar
David M
Posts: 7807
Joined: Tue Aug 11, 2009 8:17 pm
Location: South Wales

Re: Website file limit

Postby David M » Sat Dec 09, 2017 12:37 pm

Ah, I see (although I'm surprised that the proportion is SO high).

Looks like PD redaction is the only real option then. Any moves to preserve earlier pages in any format would be useful.


Return to “Website Comments”

Who is online

Users browsing this forum: No registered users and 0 guests