 Notice. New forum software under development. It's going to miss a few functions and look a bit ugly for a while, but I'm working on it full time now as the old forum was too unstable. Couple days, all good. If you notice any issues, please contact me. |
Forum Index : Microcontroller and PC projects : The Archive problem....
Page 1 of 2   |
|||||
| Author | Message | ||||
Grogster Admin Group  Joined: 31/12/2012 Location: New ZealandPosts: 9610 |
Hi all.  This is not about backup - there are a ton of options there. This is about ARCHIVE of important data. Flash is not a good choice, as the cell charge dissipates give a little time - Data is gone. DVD/CD/Blu-Ray is only good for about ten years if you look after it, as the dye degrades over time - Data is gone. These two points alone, mean that for the most part, SD cards, USB flash-drive's, CF cards, SSD's and pretty much any CD/DVD media are all not a good choice for archive, as they WILL be unreadable if you park them on a shelf for 20-25 years. I hear good things about M-disc , so am considering this format, but tape drives seem to be the favourite for serious archiving, as tape is still quite robust long term. The tapes are cheap-ish, but can hold 6TB or more per tape, which is WAY more then any of the disc mediums can touch. The problem is that tape DRIVES are hideously expensive bits of kit. I don't really know why, they just are.  We shall not mention that come time to read back the media, you have to have a reader of some kind. Having a huge archive is useless if you can't actually read it cos the reading devices no longer exist.  It's something of a problem, as no one media is perfect for this. Even hard-drives have issues if used as long-term storage. Cloud-based storage like Dropbox or OneDrive are popular and quite cheap, but they don't really work when you have about 50TB of data to archive, as I currently have spread across various server machines. It's a bit of a problem, if you will pardon the pun. At the moment, I have all my redundant backups on hard-drives, stored in a cupboard at the workshop, so that if anything happens here at the house(Gawd forbid), I can copy back from that set - provided nothing happens to the workshop!  EDIT: I also have a Dropbox backup of all the important smaller files, but none of the big stuff like audio or video filesDo any members here have any suggestions as to how archiving should be done? Remember - this is NOT just backups or mulipule-redundant-backups(such as the triple-backup method), we are talking long-term archive storage of data here - 25 years AT LEAST. EDIT: Here is a link to ten reasons why tape is still the best way to backup data Smoke makes things work. When the smoke gets out, it stops! |
||||
| JohnS Guru  Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
I'm under the impression tape is not reliable long term (bits bleed through, oxide flakes off, etc). Sorry. Best I can suggest is NOT to put on a shelf for 25 years but instead plan to re-copy the entire set every few years onto whatever the currently reasonably affordable storage is. (Multiple copies, of course.) John |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9610 |
That is logical advise. But 50TB every few years could start to get expensive over time....  Has anyone used M-disc or have any option of it? I have all the original DVD's and BluRay discs I have bought over the years, but we are talking hundreds of authentic discs here, and to re-rip them all if the rips all failed.....Ahhhh, no thank you. EDIT: Perhaps, I could just consider the original DVD's and BluRay discs to be the archive masters? How long do professional silver DVD's and BluRay discs last do you think? Not something I would enjoy doing, but as an archive master..... Smoke makes things work. When the smoke gets out, it stops! |
||||
CaptainBoing Guru Joined: 07/09/2016 Location: United KingdomPosts: 2170 |
have a look at Amazon Glacier. HD based archival. US$0.004 per gigmonth and 10 Gb/month retrieval without charge. I use this but not 50Tb! Will cost you $200 a month at that rate but there is a scale. With Glacier, your stuff is stored on disk which is then powered down and left dark. Subject to the usual resilient storage methods. There is a lead time while your data is located and mounted, so not always instant access. I Glacier just the stuff I really don't want to lose. besides cloud storage, on my desk I use a RAID5 NAS filer (as my general connected storage) with secondary backup of the NAS to single disc. Everything is in at least two places, the first is resilient and the single disc (last chance) is powered down most of the time - comes up once a day to receive the NAS backup then back to sleep. |
||||
| Volhout Guru Joined: 05/03/2018 Location: NetherlandsPosts: 5089 |
People always think about archiving large amounts of data, as if it where important to anyone...after they die... No, it is not. Best way is to sort the data into 2 groups. Really important data (and that is only little) there are very good ways to make it survive time...  Not important data (maybe only for yourself, as long as you are interested in it).  It is the sorting that one generally wants to avoid (hard to say goodbye...) Trust me, all my (and my wife's) parents are dead, and we had to clean out their collection of posessions (empty the house) one by one. And it is remarkable how much people collect that is absolutely without any value for others.... Even the photo albums full of polaroid (and black and white photo's) we kept, where never opened afterwards. They collect dust in our attic now. That also goes for data. PicomiteVGA PETSCII ROBOTS |
||||
| lizby Guru Joined: 17/05/2016 Location: United StatesPosts: 3378 |
>collect dust +++ What would 50TB compress down to? You could get 6 of these: https://www.amazon.com/Seagate-Desktop-External-Drive-STGY8000400/dp/B07CQJBSQL or howevermany needed for your compressed archiving. In 8 years, buy again what you need (much cheaper then) and copy it all. If you do that a 3rd time, you have 24 years. (Are you expecting to live another 24 years, and still want this stuff--if so, tell us what your secret is?) ;-) PicoMite, Armmite F4, SensorKits, MMBasic Hardware, Games, etc. on fruitoftheshed |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
Production CDs and DVDs are photo copied with nice clean round holes for the data bits. CDs burned on your machine have nasty little pits from the laser. The US NBS does not consider laser burned CDs or DVDs as archive media. I have been rotating hard drives through an external adapter for years. The bare disk is cheaper than the one in a case with a wall wart and a usb cable. For archival purposes you don't need to buy a really fast drive, stick with the cheaper ones. Compression takes quite a bit of time. Be prepared to walk away and have a beer. Think about making image copies of an entire disk. Macrium Reflect free does a good job and it compresses very quickly. Paul in NY |
||||
| JohnS Guru Joined: 18/11/2011 Location: United KingdomPosts: 4044 |
I also do image copies. I use dd and bzip2 (or whatever compression is trendy). John |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9610 |
Thank you chums, useful information. I have Macrium Reflect - I used it to upgrade the SSD in this Intel NUC, which only has one internal SATA port. The archive is for me only. The thought of having to re-rip my entire DVD, CD and BluRay collection all over again makes me close my eyes and shudder....  Smoke makes things work. When the smoke gets out, it stops! |
||||
bigmik Guru Joined: 20/06/2011 Location: AustraliaPosts: 2950 |
Hi Grogs, All, I thought I was being really smart and cool running two NAS raid 5 systems and a Bluray backup of really important stuff like photos.. 2 weeks ago I received a message that disk 3 was faulty on one of my NAS boxes and needs replacing.. I immediately replaced it (no problems went as planned).. Then I started wondering about backups.. I have about 50GB of digital photos on 2 bluray disks (also backed up on my two NAS setups and my main PC.. BUT What if the house burned down??? I have lost the lot.. I can probably contact my software people for replacement keys for the software I purchased but my memories are irreplaceable.. The more booze I drink the less I remember and I want to save these for my daughter and her children (when she has them)... I will be reading this with great interest.. In my case it is mostly the pictures and documents to backup.. Regards, Mick Mick's uMite Stuff can be found >>> HERE (Kindly hosted by Dontronics) <<< |
||||
| bigmik Guru Joined: 20/06/2011 Location: AustraliaPosts: 2950 |
GDay Grogs, I just read that article on MDISK and when I read RITEK were producing them. I instantly turned off that idea as they make the absolute worst DVD media... The dark/deep purple colour is very appealing but within 6-12 months I doubt you could read your data successfully... So I wont be touching MDISK.. I also am not sure any tape media is a good idea as tape is known to flake as it ages.. I think online storage is probably the best option but not cheap but then you need someway of recording the login details for those that follow after you are gone so that they can retrieve the data.. Regards, Mick Mick's uMite Stuff can be found >>> HERE (Kindly hosted by Dontronics) <<< |
||||
TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
I used to have my backup NAS in the shed. It is far enough away from the house that, apart from a bush-fire, loosing both in the one fire is unlikely. I now have the NAS in the house and store CD/DVDs in the shed along with a big pile of negatives. Jim VK7JH MMedit |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9610 |
Mik - Dropbox has a 1TB plan which is very affordable. I have one of those for all the local stuff such as the PCB patterns I design etc. This is my 'Off-site' storage. I don't think it is much - US$100 per year or something. That would allow you to store all your photos in the cloud. 50GB is well within the 1TB plan. HERE is a link to the 1TB plan. 8.95 per month, or $99 per year. For 1TB of cloud storage, I think that is a pretty good deal. It all comes down to if you are prepared to pay that every year. But it is then their responsibility to keep your data safe, so hassle-free, but it would add up to a bit of cost over several years. Smoke makes things work. When the smoke gets out, it stops! |
||||
| Grogster Admin Group Joined: 31/12/2012 Location: New ZealandPosts: 9610 |
Yes, that is EXACTLY how I have one of my servers setup at the moment. Great minds think alike. I also have in-house backups on other Linux machines, but they would be destroyed if there was a fire or something. (Gawd forbid) Smoke makes things work. When the smoke gets out, it stops! |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
Part of my "fire plan" is grab the NAS and put it in the tub with the passports etc. (and the laptop) VK7JH MMedit |
||||
| erbp Senior Member  Joined: 03/05/2016 Location: AustraliaPosts: 195 |
@Grogs - An alternative solution??? From the 47L16 datasheet referenced on the an alternative to VAR SAVE thread: Now at 2KB per chip, how many chips would you need to store your 50TB of data??? |
||||
| Paul_L Guru Joined: 03/03/2016 Location: United StatesPosts: 769 |
Trusting a cloud data storage provider to securely store your data for many years is a questionable practice. Who is dropbox anyway. It was founded in 2007 by Drew Houston and Arash Ferdowsi, two students at MIT. It must be good ... it has been censored in China. It has a NASDAQ ticker sysmbol "DBX". As of now it has never earned a profit. They started out using Amazon's S3 storage system but they are now using their own database written in CoffeeScript. They have major offices in San Francisco, Austin TX, and Dublin Ireland. Sounds pretty good, doesn't it. But not as good as Control Data, Digital Equipment Corporation, Remington Rand Univac, Data General, Honeywell, Philips, Philco-Ford, Radio Shack, Wang Labs, or Zenith Data Systems, all of which have gone bye bye. Dropbox stands a better chance of disappearing than any of these. Google does appear to be a lot sounder. Don't trust a single cloud storage company! Pair Dropbox with Google Drive, Box, One Drive, iCloud, or Amazon Drive. Zoolz provides free cold storage with delayed access. Google Photos provides unlimited free photo storage. Flickr provides 1TB of free photo storage. Paul in NY |
||||
OA47 Guru Joined: 11/04/2012 Location: AustraliaPosts: 986 |
I might be an old cynic but what is the definition of FREE? OA47 |
||||
| TassyJim Guru Joined: 07/08/2011 Location: AustraliaPosts: 6283 |
You just have to sell your soul. VK7JH MMedit |
||||
panky Guru Joined: 02/10/2012 Location: AustraliaPosts: 1114 |
Hi G, 50TB seems a huge amount to have in perpetual archive? Anyway, DLT tapes can store up to 1.6TB but the drives are not cheap and the tapes have the long term issues with cross magnetization and/or degradation. The tapes themselves are not real cheap either. Lastly, tapes are slow, comparatively. An alternative solution would be a Western Digital NAS with 20TB storage. At around a $1000 AUD, it works out considerably cheaper than the tape option and considerable faster. With 2 10TB drives, you could partition one as storage and have some form of script to progressively archive to the second. If the archive drive fills, store it and plug in a new drive. Also very easy to duplicate if you need off-site archiving. Lastly, storage is only half the problem - you need to have a system that can restore the data for use if and when needed. I have a pile of 8" dsdd floppies I can't read, an even bigger pile of 5",3.5", DLT tapes etc. none of which I can read due to the hardware being lost, broken or (sob), thrown during house moves. Good luck panky. PS. I agree with someone earlier - plan on moving the data to a new medium every fives years or so. ... almost all of the Maximites, the MicromMites, the MM Extremes, the ArmMites, the PicoMite and loving it! |
||||
| Page 1 of 2 |
|||||
 Print this page |
 |
The Back Shed's forum code is written, and hosted, in Australia. | © JAQ Software 2025 |
