-
-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Messages Import Export issue #88
Comments
Duplicate of #6 |
@biopsin From the descriptions, these two are completely different. Here the file is exported, but without extension, so it can't be imported. |
There's no crash or any visible error in this case, export is always successful (with or without extension). And even with proper extension, the same file can not be imported unless it is exported to "Downloads" folder in home directory, and if its unmoved, untouched. If I move same file to another location and try to import it, it is greyed out. If i again move it to Downloads and try to import it, it still stays greyed out. So, both issues are very different ones in my opinion. |
But what sizes is the exported file? |
The file which gets exported is usually of few MBs (around 5 MB), I had 2-3 years of SMS which i was backing up. |
There are two separate issues:
|
Very similar to FossifyOrg/Notes#14 (which has a more detailed commit message). Basically, some file pickers don't automatically append the extension, so we should do it ourselves. It doesn't cause problems. Reported in FossifyOrg#88.
Sometimes Android reports the MIME type as "application/octet-stream" even if the file has a .json extension (specifically, on Android 8 and 9 for backups exported outside the Download folder). Previously, such files could not be selected in the file picker when importing. As a workaround, allow files with that MIME type. Fixes FossifyOrg#88.
Very similar to FossifyOrg/Notes#14 (which has a more detailed commit message). Basically, some file pickers don't automatically append the extension, so we should do it ourselves. It doesn't cause problems. Reported in FossifyOrg#88.
@Abcd1234-dot Do you want to check if #136 and #137 fix this issue for you? They do for me. (I'm not a project member so you shouldn't trust me, but if you don't want to build from source / wait for a project member to build, then I've attached an APK file that I built myself: unofficial-messages-2-foss-debug-for-issue-88.apk.zip.) |
I don't know how to compile from source code. the apk file you attached is compiled one from the source code of fossify messages and includes your fix? if yes then i am willing to give it a try. And yeah probably you identified actual the problem. |
Yes. (I had to put it in a .zip file because GitHub limits attachment file types.) @naveensingh Are you okay with me building and uploading APKs like that for testing? |
@tom93 Yeah, it's okay. As these things can be risky in general, there will be some guidelines later. I'll be setting up workflows to build APKs automatically so PR authors don't have to do it themselves but it'll require a PR to have |
@tom93 I tried the attached apk. It fixes one issue. Now I am able to select .json file stored anywhere in internal storage. But when I select it, it prompts the "importing messages" window with tick/untick sms and mms messages and ok and cancel buttons and when i click on ok, "importing" appears for few seconds and then disappear and nothing happens!! the "importing messages" window stay there as before. I guess import is not successful or some other kind of error (no visible error tho)?? I tried my old backup and new backup but same results. |
I think this issue exists in all apps which allows import export backups, so it might need to port to all such apps if the PR is successfully merged. |
I'm glad it fixed the file selection issue. I don't know why the import isn't working, I tried reproducing in an Android 8 emulator but it worked fine for me. I'm not sure how to debug the new issue. All I can think of is to open the .json file in a text editor (might need to rename to .txt) and check that it actually contains the messages you expect. And the adb logs might be useful if you are able to get them. Update from the future: The issue was that the import is really slow. I hope the devs can help more... For reference, here are my steps:
The messages reappear correctly. The only difference compared to what you described is that the "Import messages" popup disappears as soon as I click "OK". I also tested with MMS.
Do you mean the missing .json extension and file selection issue? If so, I've mentioned porting those fixes (#136 (comment), #137 (comment)) and I plan to follow up on that once approved. |
I cross verified the exported file by opening it on my desktop, it contains messages as expected. my old backup contains too much content that made my desktop with low ram frozen (last 2 years of SMS). But my new backup contains very less messages. there's no visible error and exporting is always successfull in my case. i guess logs might be useful if someone tells me how to produce logs. I would provide them to @tom93 personally or devs and won't post them publicly for security reasons. |
Getting the Android logs is a bit of a hassle*. But since you're okay with running debug builds, and since the import code is fairly short, I can add some custom debugging code that will log what it's doing to a text file in the Download folder. That might be enough to identify the issue. *To get the Android logs, you need to download the platform tools, enabled developer mode and USB debugging on your phone, and then run adb logcat from the command line. P.S. For future reference, if you're having trouble opening large text files on a PC, I recommend trying Notepad, WordPad (included with Windows until Windows 11), or installing Notepad++. But since the issue also happens with the new small backup, we should focus on that. |
I have logs now. I am not sure how to share them with you privately as there seems no way to send personal message on github. you may ping me on [email protected] and i can share it in email. |
Great. You can view my email by appending ".patch" to any of my commits (example, from #137). If you want to limit the logs, you can try to identify the relevant parts by searching for words such as "fossify", "fatal", "exception", and "error". Stack traces from crashes are typically the jackpot, they look like this:
The short names in lines such as "at kb.m.a(SourceFile:12)" are not sensitive, they refer to locations in the source code and can be converted to the full names (see #6 (comment)). I just noticed that you only get the "Importing..." message. There should also be an "Importing successful" message after that. I'm pretty certain that you are not seeing the second message (and that the dialog is not closing) due to an uncaught exception (as in the above example), which is very promising. |
Yes exactly, I am not getting "import successful" message as it appears while exporting like "export successful", that's why i guess some underlying error? By the way I sent you logs via email and I literally don't understand a single word in the log file. You may find crashing errors for Play services, please ignore them. |
Hmm. I had a look at the log that you sent, there were no relevant crashes, but based on the debug messages of the form "D/MmsSmsProvider( 2504): getSingleAddressId: insert new canonical_address for xxxxxx, _id=58" it looks like it is importing, just doing it really slowly (the timestamps show it was importing for ~4 minutes and then it seem like you killed the app). I suspect the calls to If you're up for an experiment, I'm curious what would happen if you try importing the smaller backup again and just leave it running for a couple of hours (e.g. overnight when you're not using your phone). I predict that the import will succeed eventually. Alternatively, just wait until I write a PR. |
I could try new back-up, but a silly question, what would happen if I try to restore backup while already having those messages in database? thats question for my new backup. (I am already messing with my important sms which is risky :') ) (I might try backing up messages again and deleted all existing messages then try to import again in mean time) and I have to admit when I opened .json file directly on my desktop (old backup) (not .txt file) it got frozen by saying too much data on json file (file is still about 5 MB but it contains last 2-3 years of sms which i expect to be very large number) but theoretically import should not be that slow as export was very quick when i backed up those sms! (took about a minute as I remember). And yeah, I tried couple of times to import before getting logs and i fairly waited around 5 minutes to 10 minutes. |
The importer is supposed to detect whether the messages already exists, and skip importing ones that already exist. Please don't do anything risky though (the ideal situation would be to import on a test device; I don't advise deleting the messages on your main device, even though in theory it should be possible to restore from the backup). The import won't necessarily take the same amount of time as the export, because the code is different and might be slower. One idea is to try only importing SMS or only importing MMS (it's unclear which would be faster -- MMS are large but I assume there are fewer of them). Edit: And again, just leave the import running for as long as you can. You can keep the logcat open to get an idea of what's happening (the MmsSmsProvider messages are the ones to look for, I think you will get one per conversation; if you see new ones popping up in the log, that means the import is chugging along). I've thought some more about making the import faster. It should be fairly easy to implement bulk insert & query, so feel free to just wait a couple of days until I put that together. (Implementing proper error handling that tries to import all the messages even if some of them cause errors is trickier, but that part isn't required for testing.) Re opening the .json file, I'm not sure which program you are using to open it, but you can definitely open it with Notepad/WordPad (even though the extension is .json rather than .txt), you just need to use right click > "Open with", or temporarily rename the file. It just won't look pretty (and it might be too large for Notepad; but WordPad should be able to handle it). |
I do not use MMS tho, only sms so i can test for that only. Alright, I will wait for your PR. |
I wrote a patch to do bulk insert (branch "import-bulk", commit) but unfortunately it only reduces the time by about 30% (when processing new messages; existing messages get skipped really quickly with the patch). Edit: That was on Android 8. On Android 14 the reduction is ~60%. Both were in an emulator. I don't know if the import can be done faster, because most of the time is spent in Android's Telephony provider and that's not code that we can control. I guess the best approach is to provide better progress indication (example). If you want to test it out, here is an APK: unofficial-messages-for-issue-88-fast-import-with-debug.apk.zip (usual disclaimer). In addition to the bulk insert patch, this APK also contains another patch (branch "import-bulk-debug", commit) that adds debugging messages to give some indication of progress. It also includes the patch from earlier to allow importing files with incorrect MIME type. If you import using that APK, you should see a sequence of debug messages (they will appear on the screen and also be written to the log -- look for lines that contain "XXX"). Quick background: The patch divides the backup into "chunks", each with 999 messages, and imports one chunk at a time. The debug messages look like this:
If the messages in the chunk have already been imported, then the debug message will instead say |
Your fix works. Import was successful this time and took around 5 minutes to import my old backup. all the previous sms are here since 2021.
there was no progress indication in notification tho. but it introduced a new issue. all the imported messages are showing up randomly in the app! and not sorted by date i.e. latest ones on top and oldest ones on bottom. on home page of app, it just shows time under the sms thread (which should be the time when it actually received but instead it shows the time of successful import) and when i open that thread it shows correct date/time. see example below. Below picture is top message on home page of app and you can see the time under the thread is the time when the backup was successfully imported. the following picture is the same sms thread opened and you can see correct date on top which is from 2021! it is same for all the messages imported today! the already existing ones are still showing correct date on home page of app tho. so i believe the home page acts like (newly imported) sms was received on the time/date of importing. |
Great!
Yeah, the notification was just an idea. The APK I built only has the debug messages (which should appear on-screen as toasts like the original "Importing..." message).
Yeah, I noticed that when I was testing. I'll split that off into a new issue. Update: #146 |
So we can successfully close this issue when your PR gets merged?
I hope you will start working on that as soon as possible as i already messed up with existing sms :') I have backed up before testing and i hope to restore them properly soon. |
Yes, it's better to create separate issues to track the new bugs, and allow this issue to be closed automatically when #137 is merged (that's the fix for the greyed-out files issue you originally reported).
Well, I'm not a project member (as I said before), so don't pin your hopes on me. I did spot something that might help and I was going to test that, but if it doesn't work out I'll probably leave it to the devs. |
I found a workaround for the timestamp issue (which turns out to be an Android bug) and built a new APK with all the relevant patches, including a patch for a serious deletion issue I discovered along the way. The code is in branch "all-import-fixes" of my fork (all commits, actual workaround). Here is the APK: unofficial-messages-all-import-fixes.apk.zip It should display the same debugging messages as before, but with a new debug message of the form "Fixing dates for N conversations" at the very end. Update: As @Abcd1234-dot pointed out below, my workaround doesn't work for threads that only contain MMS. It turns out that this is due to a bug in Android, see below. (Also, my hack to update existing conversations only applies to SMS, but even if I did the same for MMS it probably wouldn't help due to the Android bug.) |
I will revert back with results after testing it. |
I can confirm that your patches fix almost all the import export issues (including time stamp, selecting .json file stored anywhere on device) for SMS. I think time stamp issue still exist for MMS but I am not 100% sure so someone needs to cross verify it. Another thing thing I want to mention is, app takes time to reflect new changes especially if large number of changes i.e. large backup is restored. To cross verify it, following are steps I followed :
After restoring backup successfully and having all sms since 2021, whenever i launch official Fossify messages app, it takes around 5 seconds to get open and showing threads which was not happening before (not with previous versions of SMT app and fossify app). I want to report something. If I export using the last apk you provided, it is not appending .json to export file (i tried export using native android files & fossify file manager), but still able to select and import the same file successfully without issue. I also support the idea you earlier proposed that a notification showing progress of import/export. |
Great.
You're right. My last APK included a temporary patch that updates the conversation timestamp even if the imported message already exists, but I only did that for SMS (not for MMS). If you are importing a new MMS though, then the conversation timestamp should be updated.
You should probably create a new issue for that. I tried to reproduce this; the first time I open the app it takes several seconds to load the new messages, but the next time I open the app it's near-instant (with 3000 messages across 1000 conversations, based on my example backup from #143). If the app is slow for you each time you open it then that's unacceptable, but if it's only slow during the first launch then it might be inevitable (would still be nice if it could be faster or showed progress, so still worth filing an issue).
Yes, I didn't bother including the .json suffix patch in my last APK, sorry for the confusion. |
In my previous test, (there are some MMS from carrier provider in my old backup, you can check them in previously shared screenshots) When i restored my old backup (along with all other later backups i created), the time stamp was not fixed for MMS, results were same as i reported previously with those screenshot above.
I think it was related with size of MMS, when cleared app data and also set MMS size to no limit in app settings, it got fixed. I guess now we can finally say that whole import export issue along with time stamp can be closed when your PR gets merged successfully? |
I tested this and I can reproduce this issue. It's due to a bug in Android -- the code here only updates the date and snippet for threads that contain at least one SMS, so it skips over threads that only contain MMS. Ideally this should be reported to Google but I'm not planning to do this myself. There is not much that can be done from the app's side. The app could ignore the date & snippet reported by Android and instead recompute them on the app's side, but that seems like too much work to me. |
If it is OS own issue, I guess we can do nothing to fix it. I thank you alot for your time and efforts to fix one of the fundamental functions of app. |
I can confirm that this worked for me too! Edit: Well... turns out it restores 1050 out of 1968 messages. Hmm. Its something, but still. It may have to do with the fact that I have two sim cards. |
Very similar to FossifyOrg/Notes#14 (which has a more detailed commit message). Basically, some file pickers don't automatically append the extension, so we should do it ourselves. It doesn't cause problems. Reported in FossifyOrg#88.
Sometimes Android reports the MIME type as "application/octet-stream" even if the file has a .json extension (specifically, on Android 8 and 9 for backups exported outside the Download folder). Previously, such files could not be selected in the file picker when importing. As a workaround, allow files with that MIME type. Fixes FossifyOrg#88.
Checklist
Affected app version
1.0.1
Affected Android/Custom ROM version
Android 8.1
Affected device model
Moto G5 Plus
How did you install the app?
GitHub releases
Steps to reproduce the bug
Expected behavior
It should be imported without any issues and restore all the SMS and MMS.
Actual behavior
While trying to import same file, it always prompt native android files and the exported file stays greyed out and not able to select it. It also doesn't contain any extensions by default (especially if exported via fossify file manager). But adding extension .json manually doesn't help much either. In my observations, if file is exported via native android files, added .json when android files asks for name and exporting it to "Downloads" directory only works. Even if I export it to any other location other than downloads, it still stays greyed out.
Screenshots/Screen recordings
No response
Additional information
It could be fossify file manager bug as I tried to export back-up from Notally, extension was always missing. Also similar bug is reported as (FossifyOrg/Notes#13). But it doesn't explain why this works in only Download folder and not to any other location.
The text was updated successfully, but these errors were encountered: