Open Photography Forums  
HOME FORUMS NEWS FAQ SEARCH

Go Back   Open Photography Forums > The Gear > Storage - Memory

Storage - Memory All devices that are used to store image data.

Reply
 
Thread Tools Display Modes
  #1  
Old January 25th, 2009, 05:23 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default External hard drive for mirror/backup

My current "safety" situation for files comprises the following (all the systems discussed run Windows XP):

1. I mirror most of the data from the two hard drives on my machine (one for programs, one for data) onto space on one of the hard drives on Carla's machine. I mirror all the date from her two hard drives (except of course for the mirror of mine!) onto a separate partition on one of my hard drives. Because this operates on a "mirror" basis, it provides no protection should I accidentally "lose" (or screw up) a file (unless I realize it before nightfall).

2. I backup all the data onto DVD's using a "full-incremental scheme".

The amount of data has become so stupendous that the backup to DVD plan is just impractical. A "full" backup is overdue, and I face the prospect of recoding onto 20 or more disks!

I think I need to look into an external hard drive. This of course raises a lot of questions, including:

1. What interface? USB2 rigs are readily available. But I guess there is also the prospect of accessing it over our LAN (100baseT). What are the pros and cons of both? One matter would be just that of transfer speed. Another is that LAN-capable external drives seem harder to find; maybe I mean "harder to find inexpensively". If I used a USB2 rig, I would host it on my machine, and for mirroring the data from Carla's machine the software would read it over the LAN (as happens now for both mirror and backup to DVD) and write it out over the USB port to the external HD.

2. What data management strategy? A mirror strategy is perhaps the simplest, but gives me no "recall" of accidentally deleted or accidentally badly bungled files. But of course some version management system will consume more space. A compromise would be a scheme in which all files are written to the "safety" drive, so that a bungled one would displace the earlier "OK" one, but files that were deleted on my system would not be killed on the safety drive (as happens under the backup software I currently use for mirroring.) And what software packages would be good for administering this?

I'd appreciate any observations you might have on this matter.

Thanks so much.
Reply With Quote
  #2  
Old January 26th, 2009, 04:21 AM
Michael Fontana Michael Fontana is offline
Senior Member
 
Join Date: Dec 2006
Location: Switzerland
Posts: 1,557
Default

Hi Doug

I can' really talk about the best option on yours OS, but here's what I use to have alwith every data twice, ending up with totally 13 harddisks, from 300 GB to one TB, each:

- Mirror-RAID for instant/shorttime backup within the computer.

- excremental autobackup of it on a daily base, on a exernal disk, softwarebased. The box is a S-ATA-based Firm/Seritek, with a hand full of drive trays.

This covers my needs against potential electric/lithning hazards (destroing the computer) as well as accidentally deleted files. With the external autobackup I alwith can go back, within a days period.


- when projects are finished, its entire data is migrated - with drag and drop!! - but I din't found a better solution yet, to at least 2 different harddisks, beeing at the studio and at home.
__________________
http://www.proimago.net
Reply With Quote
  #3  
Old January 26th, 2009, 04:52 AM
Jack_Flesher Jack_Flesher is offline
Senior Member
 
Join Date: Jun 2006
Posts: 549
Default

You might want to take a look at the Drobo...
__________________
Jack
www.getdpi.com
Reply With Quote
  #4  
Old January 26th, 2009, 05:21 AM
Cem_Usakligil Cem_Usakligil is offline
Senior Member
 
Join Date: Jun 2006
Posts: 5,946
Default

Quote:
Originally Posted by Doug_Kerr View Post
1. What interface? USB2 rigs are readily available. But I guess there is also the prospect of accessing it over our LAN (100baseT). What are the pros and cons of both? One matter would be just that of transfer speed. Another is that LAN-capable external drives seem harder to find; maybe I mean "harder to find inexpensively". If I used a USB2 rig, I would host it on my machine, and for mirroring the data from Carla's machine the software would read it over the LAN (as happens now for both mirror and backup to DVD) and write it out over the USB port to the external HD.

2. What data management strategy? A mirror strategy is perhaps the simplest, but gives me no "recall" of accidentally deleted or accidentally badly bungled files. But of course some version management system will consume more space. A compromise would be a scheme in which all files are written to the "safety" drive, so that a bungled one would displace the earlier "OK" one, but files that were deleted on my system would not be killed on the safety drive (as happens under the backup software I currently use for mirroring.) And what software packages would be good for administering this?

I'd appreciate any observations you might have on this matter.
Hi Doug,

I don't have much time for a longish answer right now, but I can go on writing pages later if you want me to ;-)

1) Interface: USB is more than fine, but it can be an issue accessing it from various PCs. So I'd recommend a SATA II enclosure (a so called NAS) which has an GB Ethernet interface and room for 2 SATA II 3.5" drives. Most of these enclosures sell for between 100-300$. You have to install your own disk drives yourself (1 or 2 disks). I recommend using 2x 1TB disks from Samsung, Seagate or WD. If you use two disks, you can configure the NAS to use Raid 0 (not recommended for a NAS) or as Raid 1 (I am neutral on this but I do not recommend it per se). In order to get any decent read/write speeds over the Ethernet, you have to get hold of one of the better NAS enclosures, such as the Synology Disk Station DS207+, the D-Link DNS-323 or the QNAP TS-209 Pro II. These units achieve a throughput of some 40-50 MB/s when connected using a 1000Mb/s (Gigabit) Ethernet infrastructure. 100 Mb/s Ethernet speeds are around 10-12 MB/s max. I don't know anything about the Drobo units, they seem to be popular in the states. If you buy one, check the real life transfer speeds published by independent reviewers on the net. USB speeds are usually between 25-35 MB/s, BTW.

2) Data management / backups: I'd, in your case, mirror the files from the PCs to the NAS. Use a synchronization program such as the MS SyncToy, Backer, MirrorFolder or any other equivalent. If you don't set-up Raid 1 in the NAS (no need to), then you can use one of the HDs in the NAS for this purpose. I'd then use a backup program to create full and incremental backups from the NAS HD which contains the mirrored data to the 2nd HD in the NAS. I'd also back up the 2nd NAS disk to an external USB drive (use a set of two) and keep one copy locally and keep the other one at an off-site location. You can rotate these weekly or more/less often.

HTH,

Cheers,
__________________
Kind Regards, Cem

flickr
website

Last edited by Cem_Usakligil; January 26th, 2009 at 06:52 AM. Reason: typos
Reply With Quote
  #5  
Old January 26th, 2009, 06:35 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Michael, Jack, Cem,

Thanks so much for your insights. I may have to press for more details as my thinking develops.

Best regards,

Doug
Reply With Quote
  #6  
Old January 26th, 2009, 06:39 AM
Cem_Usakligil Cem_Usakligil is offline
Senior Member
 
Join Date: Jun 2006
Posts: 5,946
Default

Quote:
Originally Posted by Doug_Kerr View Post
Michael, Jack, Cem,

Thanks so much for your insights. I may have to press for more details as my thinking develops.

Best regards,

Doug
Please do so :-)
PS: I see that you don't have a 1000 Mb/s Ethernet infrastructure in place yet. So maybe you can start with 2x USB2 drives plus one for off-site until you can upgrade the PCs and your switch to GB Ethernet. 10-12 MB/s throughput over the 100 Mb/s Ethernet can be painfully slow, albeit manageable for incremental backups overnight.

Cheers,
__________________
Kind Regards, Cem

flickr
website
Reply With Quote
  #7  
Old January 26th, 2009, 11:28 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 32,467
Default

Quote:
Originally Posted by Michael Fontana View Post
Hi Doug

I can' really talk about the best option on yours OS, but here's what I use to have alwith every data twice, ending up with totally 13 harddisks, from 300 GB to one TB, each:

- Mirror-RAID for instant/shorttime backup within the computer.

- excremental autobackup of it on a daily base, on a exernal disk, softwarebased. The box is a S-ATA-based Firm/Seritek, with a hand full of drive trays.

This covers my needs against potential electric/lithning hazards (destroing the computer) as well as accidentally deleted files. With the external autobackup I alwith can go back, within a days period.


- when projects are finished, its entire data is migrated - with drag and drop!! - but I din't found a better solution yet, to at least 2 different harddisks, beeing at the studio and at home.
I ROTWL!

The word to use, I believe might be "incremental" as excremental implies that the material being stored you never want to see again! source of definition.

However, it's really a novel way of thinking of all the data we keep pumping out!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #8  
Old January 26th, 2009, 12:58 PM
Michael Fontana Michael Fontana is offline
Senior Member
 
Join Date: Dec 2006
Location: Switzerland
Posts: 1,557
Default

Off course, you right Asher, LOL ;-)
__________________
http://www.proimago.net
Reply With Quote
  #9  
Old January 26th, 2009, 01:09 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Hi, Michael,

Quote:
Originally Posted by Michael Fontana View Post
Off course, you right Asher, LOL ;-)
But it's a great Freudian slip, and we are trying here to find some technical concept to which the word can be legitimately applied in other than its normal meaning.

Best regards,

Doug
Reply With Quote
  #10  
Old January 26th, 2009, 01:18 PM
Nicolas Claris Nicolas Claris is offline
Administrator/Moderator
 
Join Date: Apr 2006
Posts: 5,331
Default

Quote:
Originally Posted by Asher Kelman View Post
I ROTWL!…
…
…
LoL

As I didn't know the meaning of ROWTL, I checked http://www.acronymfinder.com

and got this only answer:

What does ROTWL stand for?
Regionalna Organizacja Turystyczna Wojewodztwa Lodzkiego (Polish: Lodz Region Tourist Organization; Poland)

Now the discussion about data storage may bounce!
Reply With Quote
  #11  
Old January 26th, 2009, 01:37 PM
Daniel Buck Daniel Buck is offline
Member
 
Join Date: Sep 2008
Location: Culver City, CA
Posts: 374
Default

I've been using a drobo, I don't work with huge volumes of images, but for me it seems to be doing pretty good. I use it as one of my main backups (but not my only)
__________________
Daniel Buck - Photographer and 3d artist
photography: 404Photography.net - BuckshotsBlog.com
3d work: DanielBuck.net
Reply With Quote
  #12  
Old January 26th, 2009, 04:05 PM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 32,467
Default

Nicolas,

No need to go into Polish! ROTFWL = "Roll on the floor with laughter"! ROTFWL!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #13  
Old January 26th, 2009, 04:19 PM
Michael Fontana Michael Fontana is offline
Senior Member
 
Join Date: Dec 2006
Location: Switzerland
Posts: 1,557
Default

hmm Asher

I know the floor-version as ROFL

meanwhile ROTWL - I had to go through Poland as well - Rolling Out The Window Laughing, which is new to me too.

Ah ok, I see that you Asher used "I ROTWL!" and not "ROTFWL " first, therefore the difference.
__________________
http://www.proimago.net
Reply With Quote
  #14  
Old January 26th, 2009, 06:27 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Hi, Michael,

Quote:
Originally Posted by Michael Fontana View Post
hmm Asher

I know the floor-version as ROFL

meanwhile ROTWL - I had to go through Poland as well - Rolling Out The Window Laughing, which is new to me too.

Ah ok, I see that you Asher used "I ROTWL!" and not "ROTFWL " first, therefore the difference.
I roll
You roll
He rolls/she rolls/it rolls
We roll
Y'all roll (youse roll in the Northeast) (reseparating the collapsed second person plural)
They roll

I just thought it would be nice to do something conjugal.

Best regards,

Doug
Reply With Quote
  #15  
Old January 27th, 2009, 03:55 AM
Mike Bailey Mike Bailey is offline
Member
 
Join Date: May 2006
Location: Wisconsin, United States
Posts: 114
Default

Doug,

The simplest solution - without change to your current network/computers - might be to just buy external USB drives. I've found the Seagate 500g drives to be reliable and fast (about 1.2 gig a minute transfer rate, depending). If you had two, you could put one on each machine, so no need to worry about the slower network. Currently they're selling for about $90 at places like Best Buy and some office supply stores. From your 20 DVD, that probably means you're talking a total of 100 - 120 gig per full backup, so that would allow you a number of backups before you had to start deleting the oldest ones.

The more expensive external Seagates have eSata; the cheaper just USB 2, which it sounds like you already have on your computers.

Keeping the size of the external drive down seems sensible since you aren't stockpiling all your data on one even huger drive. With the increasing size of drives it seems when failure happens too much more (i.e. ALL) is lost. Always a compromise, but these 'medium' sized 500g drives should manage nicely.

Mike
Reply With Quote
  #16  
Old January 27th, 2009, 05:17 AM
Michael Fontana Michael Fontana is offline
Senior Member
 
Join Date: Dec 2006
Location: Switzerland
Posts: 1,557
Default

Watch out for the Seagate Barracuda's and Diamond Max at the moment - they've some problems:


__________________
http://www.proimago.net
Reply With Quote
  #17  
Old January 27th, 2009, 05:20 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Hi, Mike,

Quote:
Originally Posted by Mike Bailey View Post
Doug,

The simplest solution - without change to your current network/computers - might be to just buy external USB drives. I've found the Seagate 500g drives to be reliable and fast (about 1.2 gig a minute transfer rate, depending).
Actually (and I didn't get a chance to report to the gang on the development of my thinking) I ordered a 1TB Western Digital USB2 rig yesterday, and it is reported to be on the truck headed here at present (of course there is a bad storm here, so who knows what that will do!).

I haven't yet figured out exactly what strategy I will use for exploiting it, and I expect a second unit will be in the picture (but I wanted to make sure I liked the particular machine I got before ordering a second one).

Thanks a lot for all your insight into this matter. I'll keep you up-to-date.

Best regards,

Doug
Reply With Quote
  #18  
Old January 27th, 2009, 05:23 AM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Hi, Michael,

Quote:
Originally Posted by Michael Fontana View Post
Wach out for the Seagate Barracuda's and Diamond Max at the moment - they've some problems
Thanks for the heads up.

We've pretty well concentrated on Western Digital drives here (although I did in fact have one go sour recently!), and as you will see from other my recent note here we have ordered a 1TB WD USB2 external. So we'll see what develops.

Thanks again for the heads up.

Best regards,

Doug
Reply With Quote
  #19  
Old January 27th, 2009, 07:19 PM
Jay Hoss Jay Hoss is offline
Member
 
Join Date: Jun 2006
Location: Midwest
Posts: 38
Default

Doug,

Another option would be to use one of these. I have two of them at work and use it as a back-up solution using Microsoft's free software SyncToy. I find that full size internal SATA drives are much more affordable per gigabyte then other external enclosure solutions.
__________________
--
jason
network photography
Reply With Quote
  #20  
Old January 27th, 2009, 08:21 PM
Doug Kerr Doug Kerr is offline
Senior Member
 
Join Date: May 2006
Location: Alamogordo, New Mexico, USA
Posts: 8,133
Default

Hi, Jay,

Quote:
Originally Posted by Jay Hoss View Post
Doug,

Another option would be to use one of these
Interesting. Thanks for the lead.

My 1TB WD external came this morning! It's on the air, and I am now making the initial load (using the same software I formerly used for mirror backup from each of our machines to the other.

That's not the final arrangement - just wanted to get back on the air.

Thanks again.

Best regards,

Doug
Reply With Quote
  #21  
Old January 28th, 2009, 05:12 PM
Jeremy Lawrence Jeremy Lawrence is offline
New Member
 
Join Date: Jan 2009
Posts: 9
Default

Doug - you can probably buy a decent HD enclosure for the same amount of money as one of those gadgets. They are designed for people who swap HDS around a lot or computers shops doing repairs/upgrades, not for the sort of thing you need.

A handy programme I've used for Windows backing up is Clone 2.1 http://newtonsoftware.co.uk/clone/
Very simple, but very good. I use it to keep the data synced on the 2 external HDs I use on on my window's laptop, whilst travelling. You tell Clone what folders/HDs to to keep synched, how you want it syncing [replace with newer usually] and how often to do it and that's it really.
I have fancier software on the Mac machine, but nothing quite as good as this little gem.


I'm surprised no-one has mentioned WHS [WIndows Home Server], this was produced by the good + clever coders at MS whilst all the newbies made Vista! ;-) .
WHS is a safer bet than Drobo too - Drobo is a proprietry and simplified version of WHS really, even though they got to market first as WHS was delayed for some time. I have a HP WHS, nice little machine which sits in cellar head behind a fire door and connects via the Home network+ethernet.


To help you decide what to do.
Mirroring safeguards current data on a single computer against the one drive failing.
Copying all this data to another disk incementally, is backing up.
Copying all new Data as it is created up and simply adding it all to another disk is archiving.
Archiving and backup are two different things. If you get a corrupt file and it is backed up, the back up is corrupted. The Archive however should be untouched as data is only added to the Archive with no replacing.
Archiving is a complete PITA, as files worked on need archiving just as much as the original RAWs, but how do you add them easily into your normal file structure?
__________________
Tradition is the backbone of the spineless.
www.futtfuttfuttphotography.com
Reply With Quote
  #22  
Old January 29th, 2009, 01:34 AM
Asher Kelman Asher Kelman is offline
OPF Owner/Editor-in-Chief
 
Join Date: Apr 2006
Posts: 32,467
Default

This is good Jeremy.

What software do you use for your Mac? There's need for backup of files, finding duplicates and making sure one doesn't corrupt the archives.

I've just boiled down 27 drives to 5 TB of archived data. Cataloging is an issue. Iview Media Pro crashes when the more than 10,000-30,000 files are stored. My one Drobo will make a dynamic incremental update of what's new for me beyond (i.e. that's not represented as yet on the archived file. I still need to get another set of drives to put off site. Ideally this would be a second Drobo or similar set up to work automatically at the other end of a modem far away.

I met a guy in the big photo store in S.F. who painstakingly scanned all his film to digital files and made DVD backups too and before he arranged an offsite place, a fire destroyed all his work, everything film and digital of 15 years creative work! Of course this was a big blow to him and he has not recovered and is now selling gear instead!

So having a mirror is not good enough! Need something off site too!

Asher
__________________
Follow us on Twitter at @opfweb

Our purpose is getting to an impressive photograph. So we encourage browsing and then feedback. Consider a link to your galleries annotated, C&C welcomed. Images posted within OPF are assumed to be for Comment & Critique, unless otherwise designated.
Reply With Quote
  #23  
Old January 29th, 2009, 03:38 AM
Jeremy Lawrence Jeremy Lawrence is offline
New Member
 
Join Date: Jan 2009
Posts: 9
Default

I use Chronosync on the Mac.
It's like a graphically fancier verion of Clone2.1 for the PC.
Though there have been some issues when people upgraded to 4 from 3, but if starting from scratch.....


I'm just trying to work out how to do off site storage that can be updated frequently and easily. It has to be easy or it won't get done.
I'm thinking two external HD enclosures with 4 Sata drives RAIDed. One is at a friend's house and one incementing during the week at my place. If I swap enclosures once a week, that limits me to only one weeks data loss.
Not cheap though. Another eSATA card for Mac, £120, 2x eSATA 4drive enclosures £350, 8x 1.5Tb HDs £800 and it's still not big enough for all my data either as with RAID5 that's only 4.5TB of total backup. Plus a UPS to make sure data isn't corrupted whilst writing due to power glitches.

If Broadband was actually fast, then I could use online storage, but upload speeds are pitifully slow for this sort of thing.
__________________
Tradition is the backbone of the spineless.
www.futtfuttfuttphotography.com
Reply With Quote
Reply

Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are Off
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -7. The time now is 08:48 AM.


Posting images or text grants license to OPF, yet İ of such remain with its creator. Still, all assembled discussion İ 2006-2017 Asher Kelman (all rights reserved) Posts with new theme or unusual image might be moved/copied to a new thread!