MSMoney import drops transactions

General Discussions and Announcements

MSMoney import drops transactions

Postby rkm » Sat Sep 06, 2008 7:24 am

I'm following the recommended way to import from MSMoney by exporting account transactions to a CSV file. Apart from not carrying over any opening balances from accounts set up in MSMoney, which is easily fixed, the main problem I have is that MyMoney ignores duplicate transactions. I've tried re-ordering the transactions to see if it was just adjacent transactions that are ignored but that didn't work either. Changing one of the amounts by a penny, for example, will get the second transaction in but I have too many for this to be a reasonable workaround.

Any ideas what else I can try?
rkm
 
Posts: 10
Joined: Sat Sep 06, 2008 6:55 am

Re: MSMoney import drops transactions

Postby mthadm » Sat Sep 06, 2008 8:50 am

No you can't workaround that. It is done on purpose. If you look at one of the threads on this forum from few days ago you will see some more detailed discussion.

We are adding a new option that disables duplicate transactions logic during imports, this is scheduled for a week from today, in build 33
---
MTH Software

Reminder: MTH Software employees will never ask for the username and password for any of your banking or credit card accounts.
Please do not provide this information when using our contact form.

My Money download page: http://www.mthbuilt.com/download.html
My Money manual: http://wiki.mthbuilt.com/My_Money_2.0_Manual
User avatar
mthadm
Site Admin
 
Posts: 1056
Joined: Mon May 05, 2008 12:58 am

Re: MSMoney import drops transactions

Postby mthadm » Fri Oct 10, 2008 6:05 pm

So you know, version 2.0.35 allows for much more fine grained control over duplicate detection in imports.

You can adjust the behavior or disable duplicate detection through preferences,

Thanks
---
MTH Software

Reminder: MTH Software employees will never ask for the username and password for any of your banking or credit card accounts.
Please do not provide this information when using our contact form.

My Money download page: http://www.mthbuilt.com/download.html
My Money manual: http://wiki.mthbuilt.com/My_Money_2.0_Manual
User avatar
mthadm
Site Admin
 
Posts: 1056
Joined: Mon May 05, 2008 12:58 am

Re: MSMoney import drops transactions

Postby rkm » Sat Oct 11, 2008 11:07 am

I dowloaded 2.0.35 today and tried again. I've set the dupe detection days to 0 but, while It's better, it still drop transactions. I have genuinely duplicate transactions - every single field in the record is the same (in MSMoney they're differentiated by the Memo field text, but that isn't exported) - and these transactions still don't make it.

I don't really want to write something to pre-process the MSMoney export data. Is there anyway to find out from My Money what is being dropped?
rkm
 
Posts: 10
Joined: Sat Sep 06, 2008 6:55 am

Re: MSMoney import drops transactions

Postby rkm » Sat Oct 11, 2008 11:29 am

Oops - talking rubbish about the Memo field. It is exported and it's empty for the txns that aren't feeding in.
rkm
 
Posts: 10
Joined: Sat Sep 06, 2008 6:55 am

Re: MSMoney import drops transactions

Postby mthadm » Sat Oct 11, 2008 12:19 pm

Could you please send us the problem CSV so we understand the issue better, please explain what is the desired outcome, such as transaction #1 should be imported but #2 should be dropped, etc

We don't need the entire file, just the parts that are causing problem.

Also, keep in mind that the reason we detect duplicates is the transfers. If we completely do away with the detection you will end up with duplicate transfers.

To explain this better, imagine you have made a transfer from account 1 to account 2, when we import the file we automatically enter both legs of the transfer the moment we find the first leg, then when we parse account 2 we use duplicate detection to understand that this transaction has already been entered and we skip it. If we completely disable this you will end up with duplicates, but perhaps in your case this isn't a concern because you don't have that many transfers

Thanks
---
MTH Software

Reminder: MTH Software employees will never ask for the username and password for any of your banking or credit card accounts.
Please do not provide this information when using our contact form.

My Money download page: http://www.mthbuilt.com/download.html
My Money manual: http://wiki.mthbuilt.com/My_Money_2.0_Manual
User avatar
mthadm
Site Admin
 
Posts: 1056
Joined: Mon May 05, 2008 12:58 am

Re: MSMoney import drops transactions

Postby rkm » Sat Oct 11, 2008 5:24 pm

The export file has over 9000 data rows. MSMoney exports in order of the "Num" column, so all of these entries are grouped together at the bottom:

Cashpoint,2005-07-28,Cash,R,xxxxxxxxxx,,Property : Improvements - Ford Road,-450.0000
Cashpoint,2005-07-28,Cash,R,xxxxxxxxxx,,Property : Improvements - Ford Road,-450.0000
Cashpoint,2005-08-01,Cash,R,xxxxxxxxxx,,Cash Withdrawal,-50.0000
Cashpoint,2005-08-02,Cash,R,xxxxxxxxxx,,Property : Improvements - Ford Road,-450.0000
Cashpoint,2005-08-02,Cash,R,xxxxxxxxxx,,Property : Improvements - Ford Road,-450.0000

Desired outcome is that all entries are imported, what happens is that the red rows are omitted.

I'm not convinced about the transfers justification for dupe checking - MSMoney account transfers look like this:
,2002-04-05,A Bank Plc,R,xxxxxxxxxx,,Transfer To : yyyyyyyyyy,-84.0000
,2002-04-05,B Bank Plc,R,yyyyyyyyyy,,Transfer From : xxxxxxxxxx,84.0000

I don't know what QIF data looks like and maybe the import logic you have is governed by that (I know of at least two Mac money programs I checked out failed my "evaluation" because they couldn't handle transfers), but the fact that you have a specialization for MSMoney already suggests otherwise. In addition, you put the "Transfer To/From" category text in the Payee field in My Money so I'm puzzled where dupe checking even has to enter the picture.

Anyway, it is how it is. How do we move it forward? (I only have 34 days of my trial period left)
rkm
 
Posts: 10
Joined: Sat Sep 06, 2008 6:55 am

Re: MSMoney import drops transactions

Postby mthadm » Sat Oct 11, 2008 7:59 pm

Let us explain what the problem is based on your data:

when My Money imports this:
,2002-04-05,A Bank Plc,R,xxxxxxxxxx,,Transfer To : yyyyyyyyyy,-84.0000
,2002-04-05,B Bank Plc,R,yyyyyyyyyy,,Transfer From : xxxxxxxxxx,84.0000


it starts with the first line, "Transfer to: yyyyy", and creates 2 transactions, first one is a debit on A Bank Pls (decrease of 84.0000) and the second one is a credit on yyyyyy (an increase of 84.0000). These transactions are linked together, change of amount on A Bank Pls will result in a change of the debit side of it on yyyy. This is what is called a double-entry transaction, and we resolve these entries and create both legs of transfer automatically.

Then, the import encounters the next line, Transfer From: xxxx, it is basically the 2nd leg (or opposite leg) that we already created. Microsoft doesn't provide any information that these transactions are in fact one, so we don't know this at the time and we go ahead and create another double entry transaction, which results in your A Bank Pls being charged twice (-84 times 2) and corresponding increase on yyyyyy of 84*2

What we do to fix this is that we have pretty sophisticated system of rules that weeds out these duplicates after the transfer, this in a turn causes some legitimate entries to be marked as duplicates and lost.

Now with your example, please give us a couple of days, we want to analyze this and see if we can provide a reasonable workaround. We may cut a private build for you to test.

Thanks
---
MTH Software

Reminder: MTH Software employees will never ask for the username and password for any of your banking or credit card accounts.
Please do not provide this information when using our contact form.

My Money download page: http://www.mthbuilt.com/download.html
My Money manual: http://wiki.mthbuilt.com/My_Money_2.0_Manual
User avatar
mthadm
Site Admin
 
Posts: 1056
Joined: Mon May 05, 2008 12:58 am

Re: MSMoney import drops transactions

Postby Zoolook » Thu Oct 16, 2008 2:18 pm

The problem of account transfers and duplicates, plagues every single finance app, and having a 'duplicate' filter, is just taking a hammer to the problem.

A simple solution (at least conceptually) would be to have the program to a 'dead' import, where every transaction is imported and every 'category' is created in the system as a category. In other words, the app does not try and work out the transfers and do double entry.

Afterwards, each account is parsed where you have transactions on the left of the screen and right of the screen showing equal and opposite transactions that are potential transfers. Just tick a box to say yay or nay, and then have the system adjust the category on one side only to be a transfer to the other account and at the same time, delete the item on the target account. Obviously multi-selects here would be useful, and even this process could take some time if you have 5 or 10 years worth of savings and credit card transfers. However, it's a lot less painful than adjusting every category manually on each account separately.

What most apps do is try and filter first, and this rarely works well in my experience, rather than intelligently guiding the user through a workflow that makes sense.

Like I said, conceptually easy, but I have no idea how easy this would be to implement in your framework.
Zoolook
 
Posts: 21
Joined: Sun Jun 01, 2008 12:57 am
Location: Brooklyn, NY

Re: MSMoney import drops transactions

Postby mthadm » Thu Oct 16, 2008 3:23 pm

We have had a prototype for this approach at some point and we decided to drop it.

The reason is that most people convert years worth of data, we have had cases going back to 1992, literally tens of thousands of records, and we found out that poping up dialog asking to identify each transfer transaction manually results is abandoned conversions.

Transfer transactions are surprisingly common in Microsoft Money, for example all mortgage payments contain split-transfers (transfers to principal, interest, escrow, etc), a typical paycheck contains split-transfers etc. so conversion could involve some 30% of transactions being transfers.

If you compare it to the chances of a losing some bona-fide duplicates they are much smaller, do you really have transactions in your register on the same day, drawn on the same amount, with the same check number or a memo ? The only few times that this happens are typically some sort of a cash withdrawal entered manually or cash adjustments also entered manually. It is easier to resolve on a case by case basis, which is why we are troubleshooting the above case.

Having said so, we are constantly looking out for a ways to easy the imports and exports of data so we may revive some kind of a user input into this process.
---
MTH Software

Reminder: MTH Software employees will never ask for the username and password for any of your banking or credit card accounts.
Please do not provide this information when using our contact form.

My Money download page: http://www.mthbuilt.com/download.html
My Money manual: http://wiki.mthbuilt.com/My_Money_2.0_Manual
User avatar
mthadm
Site Admin
 
Posts: 1056
Joined: Mon May 05, 2008 12:58 am

Next

Return to General Discussions and Announcements

Who is online

Users browsing this forum: No registered users and 17 guests

cron