Fundraising/Normalized donation messages

From Wikitech
Jump to: navigation, search

This page documents the in-flight format for donation messages.

Overview

Incoming donations are sent between subsystems as a single packet of data, in a flat dictionary. We use JSON over the wire, and PHP arrays internally. Subscription creations and modifications, refunds, and unsubscribe requests are also sent using similar techniques, and are documented here as well.

We use donation queues to decouple the CRM database from public payments servers, and to not block performance-sensitive tasks.

Legacy format notice: This is the current schema for our messages. Please do not make edits here unless they reflect the existing queue consumer code. Message formats have developed over time and suffer from inconsistency and accumulated cruft, there is a message roadmap with plans to fix this.

Headers

Some data is sent in the message headers in wire format, to make it available for querying.

correlation-id
Certifiably unique identifier sometimes used as the primary key for queuing and dequeuing operations. The default way to construct this ID is GATEWAY-GATEWAY_TXN_ID. If the message does not have a correlation-id, then the native queue server message ID is used instead.
source_name 
freeform name of the application which generated this message
source_type 
category of application.
payments 
came in direct from donatewiki or a banner (payments.wikimedia.org)
listener 
real time messages coming in via a listener service created by the payment provider
audit 
created by the nightly reconciliation job
direct 
message was generated within our CRM system itself, for example a hand-entered donation or refund.
source_host
machine name where initial intake interfacing occurred.
source_run_id
process identifier of the generating code
source_version
revision level of the originating code
source_enqueued_time
unix timestamp encoding when the message was first added to a queue. This header is not updated during requeueing or other operations.

One-time donations

This is the most common type of donation message. They land in the "donations" queue, and are processed by the queue2civicrm consumer.

gateway
Gateway identifier string, e.g. "paypal" or "globalcollect".
gateway_txn_id
Transaction ID string used by the gateway. Sadly, several of our processors cannot guarantee that this is a unique identifier.
date
Time donation was received, in unix timestamp seconds since epoch.
currency
(required) Original currency of transaction. Do not use the deprecated original_currency or original_gross fields, these are too confusing. We'll introduce "settled_currency", etc., when it becomes necessary to track FOREX across processor accounts.
gross
(required) Total amount of transaction, in original currency.
fee
(optional) Fees charged by the payment processor, in original currency. If unspecified, this will be assumed zero, or calculated from gross - net if available.
net
(optional) Amount after subtracting fees.
settled_date
(NOT IMPLEMENTED YET) Unix timestamp at which the transaction was settled.
settled_currency
(NOT IMPLEMENTED YET) Currency code in which settlement was made.
settled_gross
(NOT IMPLEMENTED YET) Total amount settled.
settled_fee
(NOT IMPLEMENTED YET) Processor fee, in settlement currency.
email
Donor's email. If unspecified, we may substitute with "nobody@wikimedia.org" for validation purposes. This default is stripped out again before storing to the database.
first_name
middle_name
last_name
organization_name
(optional) If given in place of first/last_name, an Organization contact will be created rather than an Individual.
street_address
supplemental_address_1
supplemental_address_2
city
state_province
postal_code
country
Billing or mailing address country—not necessarily the same as the contribution_tracking country of web origin.
payment_method
Primary payment method, e.g. "cc". TODO: enumerate.
payment_submethod
Payment method details, e.g. "visa". TODO
gateway_status
Raw gateway status code, if available.
notes
Text blob will be stored as a CiviCRM note, associated with the contact.
contact_source 
TODO WAT?
gateway_account
Our name for the gateway account, e.g. "default", "Euro". We support multiple accounts per processor.
letter_code 
TODO
gift_source 
TODO
restrictions 
TODO
import_batch_number
Conceptually broken, this should be renamed. It's a number internal to AZ Lockbox.
check_number 
contact_type
(optional) "Organization" if you want to create an Org record in Civi. This is normally implicit, according to which name fields were filled out.
anonymous
(unused)
optout
(unused)
language
Our best guess at the donor's preferred contact language.
recurring
Always "0" TODO: EH?
utm_campaign
Mapped into the direct_mail_appeal custom field
direct_mail_appeal 
contribution_tracking_id 
completion_message_id
If a message does not contain enough information to import a donation into Civi, this property gives the ID of a message in the pending queue with the rest of the data. Currently used for AstroPay and Amazon IPN messages.
log_id
Like completion_message_id, but for looking up missing information in payment cluster logs. Used for AstroPay and Amazon audit file messages.

Recurring PayPal donations

gateway 
txn_type 
PayPal-style recurring subtype. One of (subscr_signup, subscr_modify, subscr_payment, subscr_cancel, subscr_eot, subscr_failed)
subscr_id 
Gateway subscription ID
subscr_date 
payment_date 
mc_currency 
mc_amount3 
"gross" for all subtypes except subscr_payment
amount3 
another name for mc_amount3
mc_gross 
"gross" field for subscr_payment
mc_fee 
period3 
Must be "1 M", meaning we charge every month.
recur_times 
Number of installments. "0" means forever (default)
payer_email 
first_name 
last_name 
address_street 
address_city 
address_zip 
address_state 
address_country_code 
address_name 
recurring 
Always "1"
cancel 
Cancellation timestamp
option_selection1 
Premium size
option_selection2 
Premium language

Recurring other gateway donations

txn_type 
subscr_id 
subscr_date 
frequency_unit 
one of (day, week, month, year), or empty
frequency_interval 
installments 
recurring 
must be 1-ish
cancel 

Other fields are the same as for one-time donation messages

Refunds

gateway 
type 
Refund subtype, one of (chargeback, refund)
date 
gross_currency 
gross 
gateway_parent_id 
Gateway transaction ID of the original transaction
gateway_refund_id 
Gateway transaction ID of the refund

Unsubscribe

email 
contribution-id 

Damaged

When invalid messages are kicked out of a queue, they are copied into the "ORIG_QUEUE-damaged" queue using the following format:

(header) error 
WmfException error code, like "INVALID_MESSAGE"
(header) correlation-id 
Same as the original message's correlation ID
error 
error message
original 
json-encoded original message