Request #26935

Missing files in the restore.

felix
Aug 07 23:05

Madam, sir,

I had a serious disk crash a month ago, and after buying a new one, I am restoring my backup from Backblaze with .zip . Some restores went fine, but I have noticed that some files are missing in the restored folder (while they are visible on your site).

Exemple, on your web site, I can see that the Tango-Big > Backup > Photo-Video> Movies Home Made > Ano Nuevo > Media folder has a bunch of files (13), while in the restored folder it only has 6. You can see for example that Clip 1 is missing.

The files have been backed up (as I can see them on your site) but for some reasons they are missing in the .zip. I thjink they are missing in the .zip, because the .zip size is much smaller than the reported size by your View/Restore Files interface

Overall, the Movies Home Made folder is reported to be 86,9 GB on your site, while the restored version is only 30,27 GB... a lot of files are missing.

Can you please check this, and what I am doing wrong (if anything).

Thanks in advance

PS: see the screen dump attached

 

Comments

User photo
felix
August 08, 2012 01:33
User photo
kc
Backblaze Inc
  • Hello Felix,
    I am getting hold of the backend guy to see what is happening.

    Thanks,
    Casey

    The Backblaze Team

August 08, 2012 12:08
User photo
kc
Backblaze Inc
  • Hello Felix,
    It looks like there is an issue with the HGUID's mismatching on some of the data.
    Since you can see the data the point is there, however, the hguid on the files are from machine that was transfered from in November.
    So, it is showing the files with the incorrect ID, but it can not decrypt the files since it is a different ID and is no longer associated with a host.

    I am sorry but the files that you are missing are non-accessible.

    You should also, remove the host and re-upload all of the data to clean this issue up.

    Thanks,
    Casey

    The Backblaze Team

August 09, 2012 11:40
User photo
felix
  • Casey,

    I hope this is a joke... A backup service which is not able to recover some of the files... is just not a reliable backup system. You call this an issue? wait... this is a major flaw in your system.

    I have no idea what is an HGUID and as a final user, I do not think I should even know about this. You are telling me that the files which were backup before November were moved to another machine and now you cannot decrypt them? I have been a BackBlaze customer for almost 2 years, and I do not care about how you manage my files internally, but I certainly expect all my backed up files to be recoverable, independently of where you store them.

    As you must have read in my first message, I am trying to restore a complete backup of my hard drive which crashed last month. To make my point cleat, I DO NOT HAVE THE ORIGINAL VERSION OF ALL THESE FILES, AND I WAS RELYING ON Backblaze backup. Hence, I certainly expect you to be abIe to recover them and make them available to me.

    Please let me know how you intend to successfully proceed.

    Thanks in advance,

    PS: sorry if the tone of my message seems to be upset, but I can guarantee you that if I am not able to recover my backup, I will write this story on the web and make sure that every single review of your service with user comment will point to it.

August 09, 2012 13:20
User photo
felix
  • Casey,

    Just to add one more thing. Indeed last november, my old Mac Mini died, and I transferred my backblaze backup account to the new machine (using the procedure proposed on your site). I consider that it is a major flaw in your procedure if backblaze does not update the HGUID during this transfer process... This must be happening every day, if people now realize that the files which were on the old machine are not recoverable anymore (and have not been backup again automatically by your system) then you are in trouble... trust me.

    Regards,

August 09, 2012 13:38
User photo
kc
Backblaze Inc
  • Hello Felix,
    This type of behavior should be gone from any new installation.
    There have been measures put in place to stop this from happening, however, at the time when the transfer was done the safety freeze was not in place. It has been recently implemented but it can not correct issues the happened prior to it inception.

    I am sorry for files appearing in the restore, they should not have been displaying.

    Thanks,
    Casey

    The Backblaze Team

August 09, 2012 14:29
User photo
felix
  • Casey,

    I think there is a confusion here. The problem is not that the files show up or not in the restore interface... The problem is that YOU DID NOT TRANSFER THEM OR PROPERLY BACKED THEM UP after the change of machines with the new HGUID. Again, this is a serious flaw. Moreover, when you fixed the problem, you did not consider changing the backup program to make sure that these files had to be backed up again? Because these files have been on my disk for a long time, before the machine change and also after (this was in fact the same disk which I moved from the dead machine to a new one). So they should have been backed up again (or transferred from the old one) with the new HGUID. To make my point clear, I am not trying to restore files which were backed up before last november and removed ever since from disk. These files have always been on the disk ever since I used Backblaze, and they should have been properly backed up all along (before and after the changes of machine).

    Another thing which does not make any sense is that some files are accessible and not others while clearly they were in the same folder (e.g. for the media folder mentioned above, I get 6 files restored out of 13)... why?

    To be franc, I am now really upset by this whole thing, and I will not accept "sorry" as a response. I am sure you personally have nothing to do with this, but your engineer better get their act together and find a solution. Because this is a very serious issue, and if I do not get my files back, I will make sure that it is publicize.

    Here is my phone number if somebody wants to call me and discuss this (I am currently in France UT+2) +33 6 87 90 51 80.

    Regards,

    Felix

August 09, 2012 15:07
User photo
kc
Backblaze Inc
  • Hello Felix,
    Again, the flaws that you are referring to have been addressed, in all of the newest clients.
    However, they are not backwards encompassing.
    > you did not consider changing the backup program to make sure that these files had to be backed up again?<
    The issue with this is that the data that tells the data center what has been uploaded is on your computer, and the data on your computer was telling the data center it was uploaded.

    So the issue isn't as simplistic as you are making it out to be.

    The data is likely there, but it has a different HGUID associated with the data, so it can not be accessed, but it is uploaded and there are pointers to the data.

    This doesn't happen on all users, or even a lot of users, this happens to a few users. That being said, I am going to go forth to the developers and put forward that anyone that exhibits files that are not accessible in there back, have the backup state removed, and then re-upload the entire data set.

    >Another thing which does not make any sense is that some files are accessible and not others while clearly they were in the same folder (e.g. for the media folder mentioned above, I get 6 files restored out of 13)... why?<

    The data is likely there, but it has a different HGUID associated with the data, so it can not be accessed, but it is uploaded and there are pointers to the data.

    >To be franc, I am now really upset by this whole thing, and I will not accept "sorry" as a response. I am sure you personally have nothing to do with this, but your engineer better get their act together and find a solution. Because this is a very serious issue, and if I do not get my files back, I will make sure that it is publicize.<

    I understand where you are coming from.
    However, in this case, if you want a backup that you do not have this issue with, you will need to remove the backup state and re-upload the data. Make sure you have the newest client:
    Available here http://files.backblaze.com
    There safe guard to do what you stated above are in this build but the data will have to be re-uploaded.

    I will speak to the devs and try to push for a removal of all backups with this issue.

    Thanks,
    Casey

    The Backblaze Team

August 10, 2012 15:22
User photo
felix
  • Casey,

    I think this discussion is not really focussing on my issue. I understand that you recognize that there has been a major issue with backup of people migrating their backup from one machine to another. To sum up:
    - from at least last november until recently, some people migrating their backup states did not have their files properly backed up again because of bugs in YOUR software wrongly assuming that the files were already on the server (they were indeed, but with a presumably wrong HGUID)
    - this issue has been know from Backblaze from sometimes (how long?)
    - you are telling me that it has now been fixed, but not soon enough for my files to be backed up again with the new HGUID
    - even still now, there are people with inconsistent backup state ("I will speak to the devs and try to push for a removal of all backups with this issue"), I am sure your customer will be happy to learn about this.

    From my point of view...
    - I have been a customer for a few years
    - Last november, I used YOUR migrating procedure (from my dead MacMini to my iMac),
    - I have always been UP TO DATE on the backup program,
    - All the files we are talking about here have been on my disk Tango-big all along, (movies of my kids I did 10 years ago)
    - I really trusted your company as to fulfill its part of the contractual deal (please, do not tell me that your license agreement says that you take no responsibility for loss of data, that would be laughable for a backup company).

    I still think that for a presumably serious backup company, taking more than 6 month to realize that such a serious flaw is present and to take complete corrective action is not serious. There are plenty of success happy restore stories on your web site. Can you please publish my terrible experience for which I think you have FULL responsibility? Can you tell me just ONE think I did wrong in all this? Because I can at least list FOUR things you did wrong:
    - your buggy migrating software did not properly migrate my files with the new HGUID,
    - your buggy backup program (installed and up to date on my iMac) did not consider backing up again most of my files because it considered that they were already on the server (yes, they were, but with a presumably useless HGUID)
    - your buggy server program did not consider "removing" the backed up state on these files with the old HGUID
    - your buggy restore/files interface was showing me all my files (with old and new HGUID) leading me to believe that everything was neat and clean and my data safe.

    That's a awful lot of bug for a "backup" company... and please, do not tell me I am making this too simplisitic. I am a computer scientist myself (check my name on the web if you want) and I can make the difference with a "simple bug" and a major design flaw... and clearly in your case, we fall in the second category.

    Let's try to move forward from here because I am loosing patience, and the time is ticking.

    As I told you from the beginning, I still get some of my files back (6 out of 13 for the example above). Why are you able to decrypt these 6 and not the other 7? Please, respond with a clear answer, I think I deserve that instead of telling me that this is now fixed for new customers.

    You tell me you do not have the HGUID for these files I cannot recover... But how do you manage decrypting the backed up files when a customer machine die? you must have the HGUID stored somewhere, so they can recover the files and put them on their new machine. Why cannot you do this again for the files which were presumably saved with the old HGUID? This does not make sense to me. Again, please give a specific precise response.

    And please, do not tell me again and again that if I want a good backup I should remove the backup state and re-upload the data. I AM TRYING TO RESTORE MY FILES after a complete disk crash, if I still had them, I would not try to restore them and we would not have this discussion.

    Regards,

August 12, 2012 02:01
User photo
kc
Backblaze Inc
  • Hello Felix,
    "As I told you from the beginning, I still get some of my files back (6 out of 13 for the example above). Why are you able to decrypt these 6 and not the other 7? Please, respond with a clear answer, I think I deserve that instead of telling me that this is now fixed for new customers."

    The answer is we can not decrypt this data, since, it has a different ID. You will not get the data that isn't showing up in the restore back.

    "You tell me you do not have the HGUID for these files I cannot recover... But how do you manage decrypting the backed up files when a customer machine die? you must have the HGUID stored somewhere, so they can recover the files and put them on their new machine. Why cannot you do this again for the files which were presumably saved with the old HGUID? This does not make sense to me. Again, please give a specific precise response."

    No, the HGUID is always there once it created. What has likely happened, is that a process did not die during the transfer backup state, so a bztransmit executable was uploading data with a previous HGUID to the New HGUID. Since, the computer, after the transfer, is under the new HGUID, all of the decryption take place from the ID, not the old ID. We can not mix and match ID's to decrypt the data.

    "And please, do not tell me again and again that if I want a good backup I should remove the backup state and re-upload the data. I AM TRYING TO RESTORE MY FILES after a complete disk crash, if I still had them, I would not try to restore them and we would not have this discussion."

    When you create a restore in full, these are the files that you can access. These are the only files that are available.

    If you wish you can speak with the dev, he may be able to shed a little more light on the situation but I believe you will largely get the information that you have previously been stated.

    Thanks,
    Casey

    The Backblaze Team

August 13, 2012 17:12
User photo
felix
  • Casey,

    "The answer is we can not decrypt this data, since, it has a different ID. You will not get the data that isn't showing up in the restore back."

    I just checked again today, and all my files are STILL showing up (for example in the Media folder of Ano Nuevo, I can see 13 files) . So if there is a bug, it is still here and your dev have NOT fixed it.

    " is that a process did not die during the transfer backup state, so a bztransmit executable was uploading data with a previous HGUID to the New HGUID"

    How can a process survive a machine change? and still transmit with the old machine HGUID on the new one?

    Yes, I would appreciate having a dev looking into this and explaining me how many bugs I have been encountering .

    Regards,

    Felix

August 13, 2012 23:57
User photo
felix
August 14, 2012 00:00
User photo
kc
Backblaze Inc
  • Hello Felix,
    Yes, you can see them, and you shouldn't be able too.
    I will get this over to the dev.

    Thanks,
    Casey

    The Backblaze Team

August 14, 2012 11:34
User photo
brianwonline452
Backblaze Inc
  • Hi Felix-

    My name is Brian Wilson, I'm one of the founders of Backblaze (you can see my picture on this web page: http://www.backblaze.com/team.html ) and I'm the software engineer that wrote most of the underlying backup system. I'm also the guy who created the problem you (and a few other people) encountered where you cannot recover some of your files. It won't help you, but I'm really sorry this happened, and we have since fixed this problem so it won't happen to any other customers. Again, this doesn't change the reality that you cannot restore some of your files from our system.

    Let me explain the flaw/bug/bad design that I put into Backblaze (since fixed). What Casey has been telling you is correct, and here is the high level summary: each computer that backs up with Backblaze has a globally unique identifier. We call this an HGUID (Host's Globally Unique Identifier). I cannot stress highly enough how profoundly important it is that this is "unique". What happened to your backup (and happened to several other customers) is that for a period of time, what we thought was a "unique" identifier was being shared by more than one of your computers (several different tiny corner cases cause this in less than 1 out of a thousand customers). Having a shared "HGUID" initiated a chain of events ending with some of your files being deleted from our datacenter.

    You were not the first person to report this issue, although you may very well be the last customer ever affected. This doesn't help you, but it was a set of small corner cases where this bug manifested itself. When we realized it was possible, we implemented a sledgehammer fix where instead of deleting some of your files from the Backblaze datacenter, your backup would become "Safety Frozen" - put into a protected state to prevent any damage. Here is some info: http://www.backblaze.com/safety_frozen.html

    So in summary, here are the two most important things I can tell you:

    1) Some of your files were deleted from our datacenter. My software caused this, and if I could get those back for you I would, but the fact is some of your files are gone. And not just gone, now the disk space where your files used to be has been over-written with new files from another backup from another user. There is no recovery here, no possible way to get your files back. I swear if there was a way I would do it.

    2) If you or any other customer creates a new backup starting today, this will NOT occur again. To clear out your old bad backup and bad situation, you should reboot (frees up any open files), uninstall Backblaze, and re-install Backblaze and re-push all your files from scratch with the latest version of Backblaze.

    I know you depended on Backblaze being rock solid and flawless, and I know we failed you. This kind of responsibility keeps me awake at night, none of us here takes it lightly.

    I also have one correction/addition to Casey's answer to your question:

    > and all my files are STILL showing up...

    Your backup consists of a LIST of the files you have backed up and where to go find them in our datacenter, plus the ACTUAL FILES sitting on hundreds of different "data pods" in our datacenter. When you "View/Restore" files, you are seeing the *LIST* of files. But the restore fails when we go to fetch the ACTUAL FILES as they are either missing from the expected location.

    Ask any other questions, I can explain anything else you would like to know.

    -- Brian Wilson
    CTO and Founder, Backblaze
    brianw@backblaze.com

    The Backblaze Team

August 14, 2012 15:25
User photo
felix

Awaiting your response

Attach file »

Please consider this request as resolved