Answers.Body: <p>Personally, I use a 12 TB RAID array composed of 5 3TB drives.  If any one of them fails, I don't lose any data, but if 2 or more fail at the same time, I am up a creak.</p>
  
  <p>Other plans can consist of RAID 6, which allows for 2 drives to fail, but at the cost of two drives worth of storage capacity rather than one (so 5 3TB drives would only give you 9TB of storage rather than 12).</p>
  
  <p>Alternately, some people prefer either RAID 1, where files are stored on two different drives in full or simply drive mirroring where files are kept on two drives manually and one is moved off site.</p>
  
  <p>Historically, I used to use optical media, and I still occasionally use BD-R disks for some specific backup scenarios, however, in general, the expectation is that HDDs should last longer than burnable optical media (though both do degrade over time.)</p>
  
  <p>It ultimately comes down to how much risk you are willing to live with and how much you can afford to spend on backups.  Really complex setups can include things like dual RAID 6 arrays, one kept on site, the other kept off site.</p>
  
  <p>Cloud archive storage is also an option where you pay to have your data massively replicated over diverse geographic regions so that no one disaster can wipe out your data, but this generally has a relatively high cost and has high bandwidth needs as well, rather than being a one time cost.</p>
  
  <p>As for connections, you can use eSATA, USB 3.0 or Thunderbolt for quick connections.  These will all keep up with the speed of most HDDs that you would want to use for archival purposes.  Size wise, smaller drives are likely more stable as they utilize larger size blocks, but you still want to make sure you go for high quality drives that have a focus on data longevity.  Most HDD manufacturers have drives intended for long term backups, though the amount of benefit you get from these drives is somewhat debated as quality of basic model HDDs has risen significantly in recent years compared to the low point.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T00:13:55.337Z
Answers.Text: How do you deal with disk fragmentation on the video storage drives? Does moving the files as a group inherit the fragmentation when moving to a new drive, or is it better to move the video files individually so that fragmentation is not inherited?
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T18:29:21.617Z
Answers.Text: @user610620 no, copying files doesn't carry over fragmentation.  You have to worry about new fragmentation a bit if you're removing things from your archive drive, but fragmentation is a storage block layer thing, not a file thing.  Also, if you are mostly worried about archival storage, fragmentation is not really an issue even if present.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-05T18:29:33.760Z
Answers.Text: Thanks for saying that copying *individual files* don't carry over fragmentation. Can you say the same for cloning fragmented hard drives?
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-06T18:58:36.043Z
Answers.Text: @user610620 if you do a block level copy the fragmentation will remain, but I'm not sure why you'd ever want to do a block level copy when moving a project from working drives to archives.  It would be horribly space inefficient.  I've only ever seen people copy the folders they need to move.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-07T17:39:11.930Z
Answers.Text: I just searched what "block level copy" means, and still don't know how anyone would implement that in Windows. So yes, I must be talking about "copying folders" then when I say cutting and pasting *individual files*.
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T18:11:07.283Z
Answers.Text: @user610620 block level copies are used in image backups on windows where you are backing up the entire drive rather than specific projects.  That doesn't make sense in the context of video projects since you want to move files and folders around rather than disk images.  Only disk images maintain fragmentation because fragmentation is simply the fact that a file may be written to multiple spots on a physical disk, based on where there is room.  You might get different fragmentation after a copy if the drive you are copying to doesn't have contiguous storage, but that's different.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-11T18:12:45.620Z
Answers.Text: Additionally, fragmentation is far less of an issue on anything SSD based as they don't really have seek times (there is a slight random access penalty, but not normally worth mentioning, especially since the SSD controller normally intentionally fragments storage to spread the wear and tear on an SSD.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-18T19:43:44.217Z
Answers.Text: @user610620 correct, that will copy individual files in to new spots on the other drive.  Even copying from one spot on the same drive to another spot and deleting as a separate activity will change how fragmented the files are.  The only file operation that won't change fragmentation is a rename (cut/paste) on the same drive.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-26T16:28:34.670Z
Answers.Text: So cutting and pasting to a new drive will not inherit the fragmentation of the file on the old drive?
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T16:03:01.777Z
Answers.Text: @user610620 correct, fragmentation refers to difference between how a file is physically stored vs logically stored.  To the user, it's a single file, but the disk storing it may not have a big enough contiguous section of free space or may prefer to write it in different sections for lifespan reasons.  In these cases, the file is fragmented for storage at multiple smaller physical locations.  You copy logical files though, not physical ones.  The destination has its own physical constraints and will map the logical file to physical storage in its own way because it's a different physical spot
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-01T19:55:32.687Z
Answers.Text: Please insert this in, and re-work, your answer
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-02T19:41:10.633Z
Answers.Text: @user610620 it really is tangential to the question though.  Fragmentation has nothing to do with my answer, I was just taking the time to explain to you why it wasn't relevant.  If anything, if you don't object, I will clear this comment thread since it's not really relevant to the answer here.  I can nuke all the comments at once.
Answers.UserDisplayName: AJ Henderson
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-03T02:07:53.453Z
Answers.LastActivityDate: 2014-12-03T02:07:53.453Z
Answers.UserDisplayName: AJ Henderson
Answers.Body: <p>Don't trust any hard drive to be reliable! Always backup your footage, you can not trust any HDD, doesn't matter how reputable the manufacturer is. There are always monday batches. HDDs WILL fail at some point, so ALWAYS make backups.
  Bear in mind that you might find a RAID solution nice to use but that's not a real backup, backup drives are disconnected and are not running with your "live" drive, otherwise you might end up with all your drives failing together, including your backups.</p>
  
  <p>I recommend going for at least 2x3TB drives, they have a lot better price per GB than 1TB drives. Though 1TB drives tend to last a tad longer on average but thats not worth the much higher price in my opinion. If you want to invest more for higher product life, don't buy standard desktop drives but instead a workstation or server HDD. For Western Digital (WD) that would be the Red/Black/Purple series. Seagate has a few 24/7 Surveilance HDDs that are also more reliable.</p>
  
  <p>In terms of speed. I'm guessing you are using USB 2.0. An alternative is USB3.0, the speed of that exceeds the read and write speeds of any HDD. If you are a Mac user you also have Thunderbolt in most newer products but external HDD cases for that are a hell lot more expensive than USB 3.0 cases but they have the advantage of daisy chaining, which means you can connect many external drives to one port without a hub if you need multiple drives connected at the same time.</p>
  
  <p>In terms of what is better, external or internal. For HDDs that doesn't really matter anymore in terms of speed, as long as you get a case with a quality controller and dont use USB 2.0 or Firewire. It depends on what suits you better. A case is of course more portable but there are also hot swap cages for backups that are installed like DVD drives.
  Choose what you prefer.</p>
  
  <p>And a general advice, never buy a complete solution where you buy a hard drive already enclosed in an external case. You do not know what you get and they are barely cheaper than getting your own case and installing the HDD yourself in a few seconds.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-04T00:13:18.540Z
Answers.Text: How do you deal with disk fragmentation on the video storage drives? Does moving the files as a group inherit the fragmentation when moving to a new drive, or is it better to move the video files individually so that fragmentation is not inherited?
Answers.UserDisplayName: user610620
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2014-12-02T23:41:39.593Z
Answers.LastActivityDate: 2015-01-19T12:25:25.450Z
Answers.LastEditDate: 2015-01-19T12:25:25.450Z
Answers.UserDisplayName: timonsku
Body: <p>I've got a bunch of usb external hard drives, but I don't trust them for storing digitised footage. I will have potentially up to 3 TB of footage after digitising. USB can be slow for transferring large files. Is there a faster alternative? and how do I know what I buy is reliable? E.g. is it better to have 3 1 TB disks or 1 3 TB disk? Is it better to go for an internal hard drive in a PC or an external disk drive?</p>
  
Tags: <storage><digitisation>
Title: What kind of storage for digital footage?
UserDisplayName: Andrew Welch
ThreadText: # What kind of storage for digital footage?
  
  (Asked by: username_0 on 2014-12-02)
  
  I've got a bunch of usb external hard drives, but I don't trust them for storing digitised footage. I will have potentially up to 3 TB of footage after digitising. USB can be slow for transferring large files. Is there a faster alternative? and how do I know what I buy is reliable? E.g. is it better to have 3 1 TB disks or 1 3 TB disk? Is it better to go for an internal hard drive in a PC or an external disk drive?
  
  **Tags:** storage, digitisation
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_3 on 2014-12-02. Score: 4 | Accepted answer
  
  Don't trust any hard drive to be reliable! Always backup your footage, you can not trust any HDD, doesn't matter how reputable the manufacturer is. There are always monday batches. HDDs WILL fail at some point, so ALWAYS make backups.
  Bear in mind that you might find a RAID solution nice to use but that's not a real backup, backup drives are disconnected and are not running with your "live" drive, otherwise you might end up with all your drives failing together, including your backups.
  
  I recommend going for at least 2x3TB drives, they have a lot better price per GB than 1TB drives. Though 1TB drives tend to last a tad longer on average but thats not worth the much higher price in my opinion. If you want to invest more for higher product life, don't buy standard desktop drives but instead a workstation or server HDD. For Western Digital (WD) that would be the Red/Black/Purple series. Seagate has a few 24/7 Surveilance HDDs that are also more reliable.
  
  In terms of speed. I'm guessing you are using USB 2.0. An alternative is USB3.0, the speed of that exceeds the read and write speeds of any HDD. If you are a Mac user you also have Thunderbolt in most newer products but external HDD cases for that are a hell lot more expensive than USB 3.0 cases but they have the advantage of daisy chaining, which means you can connect many external drives to one port without a hub if you need multiple drives connected at the same time.
  
  In terms of what is better, external or internal. For HDDs that doesn't really matter anymore in terms of speed, as long as you get a case with a quality controller and dont use USB 2.0 or Firewire. It depends on what suits you better. A case is of course more portable but there are also hot swap cages for backups that are installed like DVD drives.
  Choose what you prefer.
  
  And a general advice, never buy a complete solution where you buy a hard drive already enclosed in an external case. You do not know what you get and they are barely cheaper than getting your own case and installing the HDD yourself in a few seconds.
  
  > **Comments:**
  > **username_2**: How do you deal with disk fragmentation on the video storage drives? Does moving the files as a group inherit the fragmentation when moving to a new drive, or is it better to move the video files individually so that fragmentation is not inherited?  
  
  ---
  
  ### Answer by username_1 on 2014-12-03. Score: 2
  
  Personally, I use a 12 TB RAID array composed of 5 3TB drives. If any one of them fails, I don't lose any data, but if 2 or more fail at the same time, I am up a creak.
  
  Other plans can consist of RAID 6, which allows for 2 drives to fail, but at the cost of two drives worth of storage capacity rather than one (so 5 3TB drives would only give you 9TB of storage rather than 12).
  
  Alternately, some people prefer either RAID 1, where files are stored on two different drives in full or simply drive mirroring where files are kept on two drives manually and one is moved off site.
  
  Historically, I used to use optical media, and I still occasionally use BD-R disks for some specific backup scenarios, however, in general, the expectation is that HDDs should last longer than burnable optical media (though both do degrade over time.)
  
  It ultimately comes down to how much risk you are willing to live with and how much you can afford to spend on backups. Really complex setups can include things like dual RAID 6 arrays, one kept on site, the other kept off site.
  
  Cloud archive storage is also an option where you pay to have your data massively replicated over diverse geographic regions so that no one disaster can wipe out your data, but this generally has a relatively high cost and has high bandwidth needs as well, rather than being a one time cost.
  
  As for connections, you can use eSATA, USB 3.0 or Thunderbolt for quick connections. These will all keep up with the speed of most HDDs that you would want to use for archival purposes. Size wise, smaller drives are likely more stable as they utilize larger size blocks, but you still want to make sure you go for high quality drives that have a focus on data longevity. Most HDD manufacturers have drives intended for long term backups, though the amount of benefit you get from these drives is somewhat debated as quality of basic model HDDs has risen significantly in recent years compared to the low point.
  
  > **Comments:**
  > **username_2**: How do you deal with disk fragmentation on the video storage drives? Does moving the files as a group inherit the fragmentation when moving to a new drive, or is it better to move the video files individually so that fragmentation is not inherited?  
  > **username_1**: @username_2 no, copying files doesn't carry over fragmentation.  You have to worry about new fragmentation a bit if you're removing things from your archive drive, but fragmentation is a storage block layer thing, not a file thing.  Also, if you are mostly worried about archival storage, fragmentation is not really an issue even if present.  
  > **username_2**: Thanks for saying that copying *individual files* don't carry over fragmentation. Can you say the same for cloning fragmented hard drives?  
  > **username_1**: @username_2 if you do a block level copy the fragmentation will remain, but I'm not sure why you'd ever want to do a block level copy when moving a project from working drives to archives.  It would be horribly space inefficient.  I've only ever seen people copy the folders they need to move.  
  > **username_2**: I just searched what "block level copy" means, and still don't know how anyone would implement that in Windows. So yes, I must be talking about "copying folders" then when I say cutting and pasting *individual files*.  
  > **username_1**: @username_2 block level copies are used in image backups on windows where you are backing up the entire drive rather than specific projects.  That doesn't make sense in the context of video projects since you want to move files and folders around rather than disk images.  Only disk images maintain fragmentation because fragmentation is simply the fact that a file may be written to multiple spots on a physical disk, based on where there is room.  You might get different fragmentation after a copy if the drive you are copying to doesn't have contiguous storage, but that's different.  
  > **username_1**: Additionally, fragmentation is far less of an issue on anything SSD based as they don't really have seek times (there is a slight random access penalty, but not normally worth mentioning, especially since the SSD controller normally intentionally fragments storage to spread the wear and tear on an SSD.  
  > **username_1**: @username_2 correct, that will copy individual files in to new spots on the other drive.  Even copying from one spot on the same drive to another spot and deleting as a separate activity will change how fragmented the files are.  The only file operation that won't change fragmentation is a rename (cut/paste) on the same drive.  
  > **username_2**: So cutting and pasting to a new drive will not inherit the fragmentation of the file on the old drive?  
  > **username_1**: @username_2 correct, fragmentation refers to difference between how a file is physically stored vs logically stored.  To the user, it's a single file, but the disk storing it may not have a big enough contiguous section of free space or may prefer to write it in different sections for lifespan reasons.  In these cases, the file is fragmented for storage at multiple smaller physical locations.  You copy logical files though, not physical ones.  The destination has its own physical constraints and will map the logical file to physical storage in its own way because it's a different physical spot  
  > **username_2**: Please insert this in, and re-work, your answer  
  > **username_1**: @username_2 it really is tangential to the question though.  Fragmentation has nothing to do with my answer, I was just taking the time to explain to you why it wasn't relevant.  If anything, if you don't object, I will clear this comment thread since it's not really relevant to the answer here.  I can nuke all the comments at once.  
  
  ---
  
  