Large imports and server DBs

[illgble]

I don't have any experience with Derby but I see that it supports a server mode and it seems you guys are using embedded. How difficult is it to convert to using the server? My main issue is that I'd like my wife and I to both be able to get at the db, but my experience with sharing dbs over file sharing is that it's very unreliable and results in lots of corruptions. Have you guys done a lot of testing with the server config?

Also, I'm trying to import what is probably a very large MSMoney file. My Money says it's on 21,199 of 35,978 and it has slowed to a crawl. I assume the slowness is do to the duplicate detection I saw mentioned elsewhere. The import already crapped out on me once, having run out of RAM, but this was while I was diddling the UI while the import was going. I'm hoping if I leave it alone, it will finish sometime tomorrow. Any reason to think this file is just too large? Anything I can do to speed it up? Because I'll undoubtedly have to do this again if I really decide to go with My Money.

Thanks,
Kevin Donn

[illgble]

Since my last post, about twelve hours ago, the count had crawled up to 23,552 of 35,978 and it's now showing that a "java heap space" error has occurred. I think this is what happened yesterday. It's clearly very unstable and is responding very slowly to user input. Activity Monitor says it's got one cpu pegged, 224MB of real and 3.35GB of virtual memory. It seems pretty clear there's a leak or something terribly inefficient about this import process. Since I've been writing this, the count has gone up to 23,556, but several minutes have elapse as I've been watching it. Every time I get the error dialog cleared away, it comes back up pretty soon.

I guess the bottom line is that I'm not going to be able to import my MSMoney transactions. Bummer.

kd

[mthadm]

Hello,


1) Server vs embedded mode: We had a server mode earlier but it was discontinued since there was no demand for it. My Money opens database in an exclusive mode so you can't corrupt data (but it also means that your wife would need to wait for her turn before she can open the same database) You can replicate the database on the network using one of 3rd party replicators (dropbox, mesh, etc) or simply have it setup on a shared network connection.


2) Large import: We would suggest to split your import into 3 chunks, each with ~10k transactions. When you setup a CSV file in Microsoft Money please filter out some accounts for the first import file on the report setup dialog, then reverse the account selection for another import file. It is a little bit more work but the alternative is to wait overnight which seems to be not practical.


Thanks,

[illgble]

I think my wife and I can work around simultaneous access. One of the problems with MSMoney is that it does support simultaneous access, just not very well. Consequently, occasionally she'll open MSMoney while I'm doing something in it and we'll wind up with poor performance and sometimes a corrupted file. When a second My Money user attempts to open the database, will it actually disallow the second connection?

As for splitting the file up, that's fine so long as it won't result in a different final My Money database. Primarily I'm worried about the detection of transfers between accounts. If I import, say, my credit card account in a single file and then import my checking account in a different file, will transfers between these two accounts still be imported correctly?

Thanks,
Kevin Donn

[mthadm]

Correct, My Money will disallow 2nd connection to the database once database is locked. This locking is done internally on the database level so you can access database on the network from different clients running different operating systems.

Most of the slowdown on large imports is because MyMoney continuously runs duplicate transaction detection; this allows for flexibility at how to separate accounts during imports, it is perfectly safe to split your credit card and checking accounts into two import chucks, the duplicates will be detected and filtered out.

[illgble]

I broke my transactions up into 4 files. The first 3 all had around 10K transactions. The final has 7,682. The first three took a total of maybe half an hour to load. The fourth is creeping. At the current rate it will take over five solid days to load.

The fourth file has only a single account in it and every transaction in it is a transfer to or from my checking account. That seems to be the problem. If I start with an empty db, load this file first, and then try to load the account it transfers with, the first file loads quickly but the second crashes about 1K transactions in with an "Internal Error".

I'm going to submit the bug through the error dialog if you guys want to look it over.

Still looks like I'm out of luck.

kd

[mthadm]

Are there any investment transactions?

[illgble]

There are investment transactions, but I doubt this could account for the crashing behavior or the slow import behavior and I'll tell you why. Remember that I've split my transactions into four files, one of which (I'll call it A) contains nothing but transfers to and from my checking account. I've tried to load these four files two different ways:

1) I loaded all three files except A. These three files include all my investment transactions. File A contains no investment transactions. All the files loaded fine (meaning all my investments loaded fine) until I started importing A.

2) I loaded A first. It loaded fine. Then I tried loading my checking account and it blew up about 1K transactions in. But remember, I've fully loaded my checking account in trial #1 above.

So it would seem that I have been able to load all of my investments in trial #1 and none of these are involved in transfers with A. The problem seems to involve the thousands of transfers I have in file A. But that's just my guess.

kd