netsgiro: a parser and builder for AvtaleGiro and OCR Giro files
Today I released the first stable release of netsgiro to PyPI. netsgiro is a Python 3.4+ library for parsing and building Nets “OCR” files.
AvtaleGiro is a direct debit solution that is in widespread use in Norway, with more than 15 000 companies offering it to their customers. OCR Giro is used by Nets and Norwegian banks to update payees on recent deposits to their bank accounts. In combination, AvtaleGiro and OCR Giro allows for a high level of automation of invoicing and payment processing.
The “OCR” file format and file format name originates in days when giro payments were delivered on paper to your bank and then processed either manually or using optical character recognition, OCR. I’m not sure how old the format is, but some of the examples in the OCR Giro specification use dates in 1993, and the specification changelog starts in 1999 and ends in 2003. A couple of decades later, the file format is still in daily use by Nets’ AvtaleGiro and OCR Giro services. In other words, I have high hopes that this will be a very stable open source project requiring minimal maintenance efforts.
Here’s an example OCR file, to be used in later examples:
>>> data = '''
... NY000010555555551000081000080800000000000000000000000000000000000000000000000000
... NY210020000000000400008688888888888000000000000000000000000000000000000000000000
... NY2121300000001170604 00000000000000100 008000011688373000000
... NY2121310000001NAVN 00000
... NY212149000000140011 Gjelder Faktura: 168837 Dato: 19/03/0400000000000000000000
... NY212149000000140012 ForfallsDato: 17/06/0400000000000000000000
... NY2121300000002170604 00000000000000100 008000021688389000000
... NY2121310000002NAVN 00000
... NY212149000000240011 Gjelder Faktura: 168838 Dato: 19/03/0400000000000000000000
... NY212149000000240012 ForfallsDato: 17/06/0400000000000000000000
... NY2121300000003170604 00000000000000100 008000031688395000000
... NY2121310000003NAVN 00000
... NY2121300000004170604 00000000000000100 008000041688401000000
... NY2121310000004NAVN 00000
... NY2121300000005170604 00000000000000100 008000051688416000000
... NY2121310000005NAVN 00000
... NY212149000000540011 Gjelder Faktura: 168841 Dato: 19/03/0400000000000000000000
... NY212149000000540012 ForfallsDato: 17/06/0400000000000000000000
... NY2102300000006170604 00000000000000100 008000061688422000000
... NY2102310000006NAVN 00000
... NY210088000000060000002000000000000000600170604170604000000000000000000000000000
... NY000089000000060000002200000000000000600170604000000000000000000000000000000000
... '''.strip()
netsgiro is obviously not a child of recent changes in my spare time interests, but was conceived as a part of our ongoing efforts in automating all aspects of invoicing and payment processing for Otovo’s residential solar and electricity customers. As such, netsgiro is Otovo’s first open source project. Hopefully, it will soon get some company on our public GitHub profile.
As of netsgiro 1.0.0, sloccount(1) counts 1160 lines of code and 948 lines of tests, providing 97% statement coverage. In other words, netsgiro is a small library trying to do one thing well. Meanwhile, enough effort has been poured into it that I’m happy I was immediately allowed to open source the library, hopefully saving others and my future self from doing the same work again.
The library is cleanly split in two layers. The lower level is called the “records API” and is imported from netsgiro.records. The records API parses OCR files line by line and returns one record object for each line it has parsed. This is done with good help from Python’s multiline regexps and enum.IntEnum, and Hynek Schlawack’s excellent attrs. Conversely, you can also create a bunch of record objects and convert them to an OCR file.
>>> import netsgiro.records
>>> records = netsgiro.records.parse(data)
>>> len(records)
22
>>> from pprint import pprint
>>> pprint(records)
[TransmissionStart(service_code=<ServiceCode.NONE: 0>, transmission_number='1000081', data_transmitter='55555555', data_recipient='00008080'),
AssignmentStart(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, assignment_number='4000086', assignment_account='88888888888', agreement_id='000000000'),
TransactionAmountItem1(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, nets_date=datetime.date(2004, 6, 17), amount=100, kid='008000011688373', centre_id=None, day_code=None, partial_settlement_number=None, partial_settlement_serial_number=None, sign=None),
TransactionAmountItem2(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, reference=None, form_number=None, bank_date=None, debit_account=None, _filler=None, payer_name='NAVN'),
TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=1, text=' Gjelder Faktura: 168837 Dato: 19/03/04'),
TransactionSpecification(service_code=<ServiceCode.AVTALEGIRO: 21>, transaction_type=<TransactionType.PURCHASE_WITH_TEXT: 21>, transaction_number=1, line_number=1, column_number=2, text=' ForfallsDato: 17/06/04'),
...
AssignmentEnd(service_code=<ServiceCode.AVTALEGIRO: 21>, assignment_type=<AssignmentType.TRANSACTIONS: 0>, num_transactions=6, num_records=20, total_amount=600, nets_date_1=datetime.date(2004, 6, 17), nets_date_2=datetime.date(2004, 6, 17), nets_date_3=None),
TransmissionEnd(service_code=<ServiceCode.NONE: 0>, num_transactions=6, num_records=22, total_amount=600, nets_date=datetime.date(2004, 6, 17))]
The higher level “objects API” is imported from netsgiro. It combines multiple records into higher level objects. For example, an AvtaleGiro payment request can consist of up to 86 records, which in the higher level API is represented by a single PaymentRequest object.
>>> import netsgiro
>>> transmission = netsgiro.parse(data)
>>> transmission
Transmission(number='1000081', data_transmitter='55555555', data_recipient='00008080', date=datetime.date(2004, 6, 17))
>>> len(transmission.assignments)
1
>>> transmission.assignments[0]
Assignment(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<AssignmentType.TRANSACTIONS: 0>, number='4000086', account='88888888888', agreement_id='000000000', date=None)
>>> len(transmission.assignments[0].transactions)
6
>>> from pprint import pprint
>>> pprint(transmission.assignments[0].transactions)
[PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=1, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000011688373', reference=None, text=' Gjelder Faktura: 168837 Dato: 19/03/04 ForfallsDato: 17/06/04\n', payer_name='NAVN'),
PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=2, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000021688389', reference=None, text=' Gjelder Faktura: 168838 Dato: 19/03/04 ForfallsDato: 17/06/04\n', payer_name='NAVN'),
PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=3, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000031688395', reference=None, text='', payer_name='NAVN'),
PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=4, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000041688401', reference=None, text='', payer_name='NAVN'),
PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.PURCHASE_WITH_TEXT: 21>, number=5, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000051688416', reference=None, text=' Gjelder Faktura: 168841 Dato: 19/03/04 ForfallsDato: 17/06/04\n', payer_name='NAVN'),
PaymentRequest(service_code=<ServiceCode.AVTALEGIRO: 21>, type=<TransactionType.AVTALEGIRO_WITH_PAYEE_NOTIFICATION: 2>, number=6, date=datetime.date(2004, 6, 17), amount=Decimal('1'), kid='008000061688422', reference=None, text='', payer_name='NAVN')]
The higher level API also features some helper methods to quickly build payment requests, the only file variant typically created by anyone else than Nets.
>>> from datetime import date
>>> from decimal import Decimal
>>> import netsgiro
>>> transmission = netsgiro.Transmission(
... number='1703231',
... data_transmitter='01234567',
... data_recipient=netsgiro.NETS_ID)
>>> assignment = transmission.add_assignment(
... service_code=netsgiro.ServiceCode.AVTALEGIRO,
... assignment_type=netsgiro.AssignmentType.TRANSACTIONS,
... number='0323001',
... account='99998877777')
>>> payment_request = assignment.add_payment_request(
... kid='000133700501645',
... due_date=date(2017, 4, 6),
... amount=Decimal('5244.63'),
... reference='ACME invoice #50164',
... payer_name='Wonderland',
... bank_notification=None)
>>> transmission.get_num_transactions()
1
>>> transmission.get_total_amount()
Decimal('5244.63')
>>> data = transmission.to_ocr()
>>> print(data)
NY000010012345671703231000080800000000000000000000000000000000000000000000000000
NY210020000000000032300199998877777000000000000000000000000000000000000000000000
NY2102300000001060417 00000000000524463 000133700501645000000
NY2102310000001Wonderland ACME invoice #50164 00000
NY210088000000010000000400000000000524463060417060417000000000000000000000000000
NY000089000000010000000600000000000524463060417000000000000000000000000000000000
Otovo has been using netsgiro in production for about a month now, and so far so good. We’re surely not the only shop in Norway doing invoicing with Python, so I hope netsgiro will be a useful building block for others as well. If you’re interested in learning more, start with the quickstart guide and then continue with the API reference.
Happy invoicing!