Data Reload After 6.0.12 Update

Bugs
1

Hi Chris,

It’s probably not fair to post this as a bug, but I wanted to be sure you would see it soon. And since we’re in different time zones, there’s bound to be delay in responses anyway.

Today, my three devices running Moviebase received the update to version 6.0.12. It appears their data was cleared and they are reloading the data from the database. I’m guessing that you made corrections in the database itself, and that is the reason for forcing the reload of the data into the app from there. The updates happened hours ago, and the apps are no where near having all the appropriate data. My concern is that the loading of the data seems to be stalling after some of it is loaded. I’m not sure how long the reloading should take, and when I should assume it’s not going to work. I’m going to let the apps continue overnight and see where they are in the morning. I don’t want to hurry to reload from a backup if they are eventually going to get all the data they should. I’m willing to be patient if it is eventually going to finish properly,

I think I know why you asked me if my data had been loaded from a backup earlier when I sent the bug about the personal list items being listed with 0 item counts when there are items in the lists. I looked at some backup files, and see that they have bad (zero) counts for several personal list entries, even though all the appropriate entries are listed below the zero count.

In fact, I had decided to correct the counts in a backup and reload it to try to fix that problem. Before I did, I received the update. I’m thinking maybe you corrected such items in the database and maybe other data to be reloaded into our apps. Am I wrong about that?

Anyway, I will see where things stand tomorrow and whether the data reloads finish, or at least get further along.

I would like to be sure I am right to be giving this a lot of time before giving up on the automatic reload.

I also saw that version 6 has a new format for the backup data. I guess the new app can process backups in both formats, so version 5 backups still work for version 6. Is that correct?

Thanks

2

Thanks for the feedback. No data will be deleted on the old database, nor the server side (only marked as deleted).

There is an option in the settings → data → reload old database, which will force the restoration of the complete old data.

Both JSON format should work.

Because there is a large amount of data, it can take some time for the loading. What exactly is missing in your case?

3

It doesn’t look like any more data loaded overnight.

What is missing varies by device. On my primary phone, I am missing a few hundred movies from my history, maybe 300 TV shows from my history, over a thousand episodes from my history. I have nothing in my watchlist. I am missing a few movies from collection. All my TV sows are missing from collection. All my personal lists are missing. Hidden items are all missing.

On my older phone, over a thousand movies are missing from my history, along with a couple hundred TV shows and several hundred episodes. All my watchlist movies and TV shows are missing. All my collection entries are missing. All my personal lists are missing. All my hidden items are missing.

On my tablet, everything is missing.

Would it help for me to force a sync from the profile page?

I have a backup from my version 6 system that is recent and complete. And I have corrected the 0 item personal list entries in it. Should I just go ahead and restore that?

It doesn’t look like any more data is going to load automatically.

4

The new release also includes an export of the old database.

You can trigger the reloading via the settings → data → reload old database.

If you want, you can send me the JSON backup file via help@moivebase.app, so I can double check the content.

I guess you always have only the Moviebase Google account and not the trakt one?

5

Yes, I only use the Google account. I don’t like the complications Trakt introduces because of the inconsistencies between Trakt and TMDb.

The restore from the old data option is nice. I did it on my three devices, and it mostly worked. It was pretty fast. It looks like there is a problem when it is automatically triggered by a new update that it doesn’t work properly. It worked well when I did it manually. Most of the data looks good, but there were slight inconsistencies in the results among the three devices.

I appreciate your offer to check my backup file, but there is no need. I am very familiar with the format because I have developed a program to do extracts from them. I did try downloading my modified backup to two devices, and it seemed to work fine. The data looks good. BUT now both devices have the problem of my not being able to make updates. This is a very serious issue. It really makes the app unusable, and I have no idea how to fix it.

Could you give us an option to clear all our app data before doing the data download from the old data? It would be consistent with the restore option that does the same thing. l would like to try that and see if it makes a difference with the issue of not being able to make changes. I am suspicious that restoring from a backup is triggering the problem of not being able to further update the data.

6

Chris,

I’ve been experimenting for quite a while.

After I restored my modified backup file to my two phones, the data looked fine, but I couldn’t do updates on either phone for a while.

However, after some time, my older phone started allowing changes, and it now seems to be fine.

My new phone, however, continued to not allow any updating, at least not very much. The only thing I seemed to be able to do was add an item to a personal list, but then I could not remove it.

I tried a number of things to recover, but nothing worked.

Then I decided to delete all the data for the app on that phone and start over. After that I could do updates as normal. Then I used the data - restore the old database option to recover the data from the old database. That worked, and I could still do updates just fine. Then I tried restoring from a backup with the delete existing data option. The restore worked fine, but then I could not do any changes again. I have tried this with multiple backup files to this phone. I always get the same result.

It appears that if I restore any backup on this phone, then I can no longer change anything.

Right now, I have deleted may app data again and started over. I am using the data - restore the old database option again, and it is restoring.

It is a pretty bad problem if I can never restore from a backup again on this phone. I have no idea what could be causing this, but it is very consistent. It doesn’t seem to matter which backup file I use - it always locks me out of making additional changes. Weird.

P.S.

I have now cleared the data for the app several times and restarted. No matter what i do, I have not been able to restore my data and still be able to change anything like marking an episode watched. I can get my data reloaded from a backup, but then I am locked out of changes.

I have tried many different backups, including a backup created from the old database with the new option under Data to create it. I have restored from both pre version 6 and version 6 backups. They all restore the data, but then I am locked out of making changes

If I load the data from the old database, I end up with a lot of missing data. Unacceptable.

I have also tried just letting the data re-sync after I reset the app, but so far that seems to stop with much data missing, I am trying that again and am going to leave the app overnight to see if it does any better. This is so frustrating.

My other two devices are OK running the app and have all or most of the correct data. They seem to work as expected after restores and with data loads from the old database. But they are old, frozen at Android version 10 and 11. I am having terrible difficulty with my newest device, running Android 16. It seems odd that all the problems happen with the newest device. Maybe there is something about Android 16 or newer security causing an issue. I am quickly running out of ideas on how to get around this. I hope you can help. I have spent most of my day trying to solve it…

7

Thanks. I have tested the code with a Google Pixel, but maybe I need to check the Samsung too.

The database is the original one from Android, but it looks like the issue is not the code. It’s more than the database seems to be locked for a few users (2-3 from around 1000).

If you want to send me some backup, I can test that too. I will also add more logging and try to understand the entire flow.

8

Thanks, Chris.

I’m glad I’m not the only one.

Also, all three of my devices are Samsung.

I will send you a sample of my backup files.

Also, trying to just let the app sync the data didn’t work. Lots of history data is missing.

I have now sent the backup file samples.

And, of course, the database doesn’t seem to be locked for my older devices running under the same ID.

10

Hi San, what do you mean by the date/time?

Do you mean inside the file?

12

Chris,

  1. I received the 6.1.0 update on my 3 devices today. After the update, the sync I had started yesterday on my primary phone picked back up and seemed to finish. The result was not perfect, but it is close enough that I can manually correct what is wrong. Also, I can now do updates to my data there. Yay!!! I suspect that if I restored from a backup, I would lose that ability again though. But at least the data is not in terrible shape, and I can update now. That is a huge step.

  2. Please, please, please do not ever trigger an automatic total reload of our data from the old database with an update again, at least for me. I think that would really set me back, and make me have to redo a tremendous amount of work.

  3. I just noticed that on all my devices I have hundreds of movies and a few TV shows marked as Favorites. I never mark anything as a Favorite. That’s just not a feature I need or use. On the TV shows it is all the shows in my Collection that are marked. All the movies marked as Favorites are also from my Collection, but it’s not nearly all of them. I don’t know why this happened, but I would definitely rather clean it up manually than start over again. All my devices are at least at a place where I can reasonably clean up the discrepancies myself.

13

Thanks for the update.

Slow adding:
I added a new version with an improved database transaction. It also seems that Samsung Knox can block some database writes.

About the missing items:
Some TMDB ID resources are not valid anymore, because they were deleted from the TMDB database. In this case, it will not transfer this data.
If something is missing, please let me know the items so I can verify that it works with your backup.

About the lists:
The Favorite and Collection lists come in the mix with the Trakt list. Moviebase was Favorites and Trakt Collection.

I guess with the multiple updates and sync, it has mixed it a bit. We have two solutions:

  • I rename the lists on the server side and delete one of the other
  • Export the JSON file and rename the list manually FAVORITES to COLLECTION

Sorry for the issues. This was a very large and complex task. This refactoring will improve the app a lot in the future.

14

Chris,

Thanks for all the very quick responses.

First of all, I fully understand the complexities of what you are doing. I have been a programmer my entire adult life, and I’m old. I think Moviebase is an incredible product. What you have done with it makes it far superior to any of the competing products in my view. I never mean to be critical of your work. Your responsiveness is amazing, and your speed at solving problems is equally astounding. So thank you for what you do, and if I ever seem too frustrated or critical, that is my fault, and I don’t mean it that way.

Now for the specific items:

  1. Yes, the new version obviously vastly improved my synchronization experience. There were still quite a few history items missing. Quite a few of the show episodes I had previously marked as viewed were no longer marked. I could not see any pattern, There were random episodes spread throughout multiple seasons. I have already fixed all of that. There were also some missing movie entries. It’s not a lot. I am working on identifying those, but it’s times consuming because I have so many entries. When I watch a movie, I mark it watched in the history, and I add it to my Movies Watched personal list. Some of my movies in the Movies Watched list were missing form the history, and some of the items in the history were missing from my personal list. I have already identified almost all of those and have corrected them. I have to update my extract program to handle the new backup format. When I have that, it will be much easier to identify discrepancies between large lists. That’s the main reason I wrote it. It also makes it much easier to identify discrepancies among my three devices running Moviebase. It is possible my synchronization would have worked better if I had started it under version 6.1.0, instead of the earlier version, but I don’t wan to start over to find out. My devices are all in a very usable state now. I also understand that some TMDb entries get removed. I have seen that multiple times before. They show up in my extract as items in my history that no longer have the detail entries in the backup from TMDb. I flag those for investigation.

  2. I didn’t realize that you meant for Favorites in Moviebase to correspond to Collections in Trakt. I don’t view them as the same thing. The way I use Collection is to list items in my library of movies and shows. By Favorites I would mean I just particularly like that movie or TV show or episode or whatever. Even though I don’t use Trakt, I would still prefer to use Collection the way I do. But if that doesn’t work out, I can always create a personal list for that purpose. Anyway, I have already removed all of the Favorites that had been marked, so I don’t have the problem at this point. I just wanted to let you know about it, so you can make any needed correction.

  3. My biggest remaining concern is being able to restore a backup on my primary phone and still update the data afterward. Regarding your comment about Knox, is there something you can do to solve this, or am I stuck? I think all my Samsung devices have Knox, but only this one is having the problem. Do you think if I removed Knox from the device, it would fix my problem? I could try to do that. I don’t really care about having Knox for my purposes. it’s just always installed on Samsung devices automatically. I am of course worried about doing a restore and once again being locked out and having to start over.

  4. I really like the new app’s Upcoming listing because it shows all upcoming episodes of a show and not just the next one. That’s a very nice improvement. Could you also make the calendar option in your widget work that way? I have always wished it would.

  5. I notice that I no longer see the cloud icon displayed with a check mark if an item has been synced. Is that something you removed?

Thanks again for all your hard work.

15

Chris,

I don’t know a lot about Knox, but since my phone is only for personal use, not part of an institutional security program, I think the types of security it offers would not block the writes, unless Moviebase were trying to access a restricted area of storage or something by mistake. From what I read about it, I’m not seeing why in my case it should block any of your writes. I guess it cannot really be removed in any reasonable way. Portions can be disabled though.

Steve

16

1 I need to do some research on a better way to handle TMDB deletes. Do you see this issue often?
2 There was a bug in my code, and also fixed it in Version 6, which separates both.
3 I know that Knox can slow down the Database transactions. Not blocking completely.
4 Thanks, there will come a new episode page where you will see all missing episodes. The new database loads all episodes into the DB to make it more offline and consistent.
5 Yes, the icon was confusing the users. I will make the sync more robust instead

Do you still have the issue with adding/removing? I made the database schema easier and faster.

17

Chris,

  1. It doesn’t happen too often. Sometimes, for instance, TMDb will decide that a show with maybe 2 parts will be changed to a movie instead, and sometimes multiple shows have been combined into a single show… There are probably some other reasons too. I have been using the app a long time. In that time I have seen several instances, and the problems accumulate in my data…

  2. OK, I assume I will still be able to use Collection the same. I have already cleaned up (removed) the Favorites that were set, so I’m OK now, if it doesn’t happen again.

  3. Well, what I see when it happens is that the transaction never happens. I have given it a very long time, and it just ignores my changes. I have tried many changes and many types of changes that are just ignored. I never noticed the changes being too slow on any of my Samsung devices.

  4. Thank you, I like the change a lot.

  5. OK, that makes sense to me.

Right now, all my devices are working well. I spent last night cleaning up missing items and fixing discrepancies between devices. I used your CVS extract to help with that, and it sped up the process quite a bit.

However for my primary phone - A Samsung Galaxy S22 on Android 16, the way I made it work was to clear the app’s data completely and then let it sync itself from the database. That’s when it was stuck partly restored after running all night. Then when I received your 6.1.0 update, the app unstuck and completed the sync from the database. After that finished, I could do updates just fine. I have been cleaning up remaining problems manually. The big concern is that any time I have restored from a backup on that phone, and only on that device, not my others, then after the restore, I can’t change anything. Obviously, I want to be able to backup and restore any of my devices, but right now I would be afraid to try a restore on the this one device. Again, I have done restores on my older Samsung devices with no problem updating data afterward. I would have to do a restore on the S22 to see if it is still locked out of updates afterward with your most recent updates. If it still fails, then I’m back to starting over with that device again. Do you think I should test it? i have done a lot of work to clean it up after clearing its data and letting it just sync from the database.

18

Chris,

Also, concerning the number 1 point above, another thing that happens fairly often is changes in the episodes within a show, even shows that have ended, Episodes are sometimes removed (especially in specials), or two episodes might be combined into one or one might be split into two, etc. When this happens, if you select the show for detail viewing, the show will be updated with the changed information, but the old information is left in the watched episode history. I don’t know yet if this is still happening in version 6, but if it is, it would be a nice thing to fix.

19

Thanks for the update. I thought the latest update must fix something with the database. But it seems like there are still some issues, even though it’s not related to the code itself only. So there must be something between the Android version on Android devices and how the library is running.

20

Chris,

After spending a lot of time fixing any remaining errors on my primary phone, I backed it up. As an experiment I copied the backup JSON to my Samsung tablet that had been working and restored the backup file there. It restored fine, and the data looks pretty good. BUT now I cannot update anything there. So we have our answer, It is not only happening on my primary phone and it is definitely not fixed, as a restore on the table left it without the ability to make any changes. I did it there because that is the device where I use the app least. I am not going to do a restore anywhere else until I do one successfully on the tablet and can still make updates. I hope you can find an answer for this. I have been using the app on these Samsung devices for a long time with no problem until the version 6 update with the new database.

Do you think maybe the reason not many people are seeing this problem is that not many are restoring from backups? Just wondering.

OK, this strange. After quite a while, I went back to the tablet, and now I can do updates on it. This is long after I did the restore. After I did the restore, it said it finished, and then I could not update. Now, after waiting quite a while longer, I can. Do you think maybe there is a setting that says the restore is in progress that is not being reset promptly after the restore finishes. Or maybe the restore is taking a long time, and the message that says it is finished is coming much too soon. I know on my primary phone, I waited quite a long time and could never update after it stopped being able to update.

Something else that doesn’t look right, is that when i backup a device and immediately use that backup to restore to another device, even though I turn on the switch to clear the data before restoring, the second device’s counts don’t exactly match what I backed up. Several of the counts are off by a little. It seems to me that the data on the second device should exactly match what I backed up, but it doesn’t. That makes me suspicious that the restore is not doing what it should.

21

Something else that doesn’t look right, is that when i backup a device and immediately use that backup to restore to another device, even though I turn on the switch to Clear Previous Items before restoring, the second device’s counts don’t exactly match what I backed up. Several of the counts are off by a little. It seems to me that the data on the second device should exactly match what I backed up, but it doesn’t. That makes me suspicious that the restore is not doing what it should. It looks like the differences are in the history data. Does the “Clear Previous Items” option not clear the history?

22

The clear previous item should always clear the completed data. But I guess I will extract it out into a separate function to really only clear all the data so that you can see it.

So that’s actually a good hint. Maybe when I restore the data, it’s maybe blocking somewhere, so it’s maybe stuck somewhere.