Fundraising.wikimedia.org

From Wikitech

Fundraising.wikimedia.org holds some miscellaneous fundraising tools and handles some of the payment processing for donations to the Foundation and runs on Grosley.

PayPal IPN Listener

The PayPal IPN Listener is a stand alone script that allows us to validate PayPal transactions. When a user makes a transaction for the Foundation through PayPal, PayPal POST's the transaction to our IPN Listener. The listener pushes the message to a 'pending' queue on ActiveMQ, then sends the message back to PayPal for verification of its authenticity. If the message is considered valid, it is then pushed to a 'verified' queue (likely the 'donations' queue) on ActiveMQ. If it is not, then the message is left on the pending queue for handling elsewhere.

Files

SVN

http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/

On Grosley

The PayPal IPN Listener lives in /opt/fundraising_misc/gateway_handling/paypal/IPN

  • IPNListener_Standalone.php: This file provides some configuration information and executes the business portion of PaypalIPNListener.php
  • PaypalIPNListener.php: This contains a class with methods to communicate with ActiveMQ and Paypal, as well as to run through the logic of verifying payments from Paypal.

Logging

The listener logs output for the regular IPN listener to: 

/var/log/fundraising/paypal_ipn/out.log

The recurring IPN listener logs to: 

/var/log/fundraising/paypal_ipn/recurring.log

These logs are rotated daily and will stick around for 7 days. When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes. They are stored on storage3 in: 

/archive/fundraising/logs/paypal_ipn/out.<rotate_date>.gz

Wwhere <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).

Access

The listener is accessible to PayPal via HTTP at http://fundraising.wikimedia.org/IPNListener_Standalone.php

Testing

If you sign up for a PayPal sandbox account, you can send test IPN messages to the listener:

  1. Go to https://developer.paypal.com/
  2. Log in
  3. Under 'Test Tools' click on 'Instant Payment Notification (IPN) simulator'
  4. Enter the listner url (http://fundraising.wikimedia.org/IPNListener_Standalone.php)
  5. Select the transaction type (should test completed and declined) and fire it off
  6. See what happens in the queue and check the logs

Further documentation

CiviCRM Global Collect Recurring Payment Processor

This recurring payment processor relies on the StandaloneGlobalCollectAdapter to initiate transactions with Global Collect.

See the code

This module is included with Drupal CiviCRM:

http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-civicrm/sites/all/modules/civicrm_recurring_globalcollect

Configuration

StandaloneGlobalCollectAdapter

You will need to copy the adapter to the library.

Go to the root of your Drupal CiviCRM installation: 

cd sites/all/modules/civicrm_recurring_globalcollect/library

mkdir StandaloneGlobalCollectAdapter

cd StandaloneGlobalCollectAdapter

svn co http://svn.wikimedia.org/svnroot/wikimedia/trunk/fundraising-misc/standalone/StandaloneGlobalCollectAdapter/ .

cd ..

ln -sn StandaloneGlobalCollectAdapter/StandaloneGlobalCollectAdapter.php

CiviCRM Recurring GlobalCollect Processor

Installation

Install the module in the Drupal admin interface.

Permissions

Home > Administer > User management > Permissions

Look for the section civicrm_recurring_globalcollect module

And enable for administrators and any other groups you deem necessary.

Remember, this is potentially dealing with donors personal financial accounts.

Configuration

Set the urls for using Global Collect or for testing.

Home > Administer > Site configuration > CiviCRM Recurring GlobalCollect Processor

If you leave the batch size at 0, the module will not process anything.

Drush

Drush has a bug that needs to be patched:

http://drupal.org/files/issues/704848-drush-run-twice.patch

before you run this command:

Get help from this Drupal module with Drush: 

drush help civicrm-recurring-globalcollect

or 

drush help crg

Your options: 

Process recurring payments to GlobalCollect

Examples:
 drush civicrm-recurring-globalcollect     # Process recurring payments                                                   
 drush crg --url true                      # TESTING: Always return a successful response from Global Collect             
 drush crg --url false                     # TESTING: Always return a failed response from Global Collect (404)           
 drush crg --url 404                       # TESTING: Always return a 404 response from Global Collect                    
 drush crg --url 504                       # TESTING: Always return a 504 response from Global Collect                    
 drush crg --url 404.php                   # TESTING: Send the request to a url relative to to tests/resources/responses/ 
 drush crg --url "https://example.org/"    # TESTING: Send the request to a full url.                                     

Options:
 --url                                     Allowed parameters: true | false | 404 | 504 | some-relative-url | https://example.org 

Aliases: crg

Payments

You can customize this processor: 

Go to: Home > Administer > Site Configuration > CiviCRM Recurring GlobalCollect Processor

You have ability to set:

  • Global Collect Url
  • Default Testing Url
  • StandaloneGlobalCollectAdapter Path
  • Email Notification Address
  • Global Collect Merchant ID
  • Failure retry rate
  • Catch up missed days

Successful Payments

Successful payments will be rescheduled for the following month.

Failed Payments

Failed payments will be cancelled on the third failure.

Each failure reschedules the payment for the following day, by default. This can be customized in CiviCRM: Failure retry rate

StandaloneGlobalCollectAdapter

Recurring Payments IPN Listener

The Recurring Payments IPN Listener is very similar to the PayPal IPN listener. In fact, it exists as a child class of the PayPal listener.

At the moment, all it does differently from the regular PayPal listener is that it ensures that the transaction type is something relating to subscriptions (recurring payments), verifies the message against PayPal, and then queues the entire POST object from PayPal in a special recurring payments queue on ActiveMQ. We store the messages in the queue for now because we currently have no sane way to bring the transactional information into CiviCRM.

Files

SVN

http://svn.wikimedia.org/viewvc/wikimedia/trunk/fundraising-misc/queue_handling/paypal/IPN/

On Grosley

The Recurring Payments IPN Listener lives in /opt/fundraising_misc/gateway_handling/paypal/IPN

  • IPNListener_Recurring.php: This file provides some configuration information and executes the business portion of PaypalRecurringIPNListener.php
  • PaypalRecurringIPNListener.php: This contains a class with methods to communicate with ActiveMQ and Paypal, as well as to run through the logic of verifying payments from Paypal.

Logging

The recurring IPN listener logs to: 

/var/log/fundraising/paypal_ipn/recurring.log

These logs are rotated daily and will stick around for 7 days. When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes. They are stored on storage3 in: 

/archive/fundraising/logs/paypal_ipn/recurring_out.<rotate_date>.gz

Where <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).

Access

The listener is accessible to PayPal via HTTP at http://fundraising.wikimedia.org/IPNListener_Recurring.php

PayflowPro Pending Transaction Verification

The PayflowPro pending transaction verification script pulls transactions off of the ActiveMQ 'pending' queue for PayflowPro, and then attempts to verify those transactions with PayflowPro. If a transaction is verified, it is then pushed onto the confirmed 'donations' queue. If it fails with an error code of 126 ("Fraud Protection Services Filter — Flagged for review by filters") or 26 ("Invalid vendor account"), the message is left alone on the pending queue until the issue is resolved. If it fails with any other error code, the message is discarded from the vendor queue.

For more information about the PayflowPro side of things, check out the PFP documentation: https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_gateway_payflowpro In particular, I recommend the 'Developer Guide' PDF.

Files

The files that power this are located in /opt/fundraising_misc/gateway_handling/payflowpro

  • stompPFPPendingProcessorSA.php - Contains the class StompPFPPendingProcessorSA which has the methods necessary to communicate with PayflowPro as well as ActiveMQ, and the logic to route transactions correctly.
  • executeStompPFPPendingProcessorSA.php - configuration and execution for the processor. Configuration should be fairly self-explanatory. Of note, you can specify log location, batch size, queues to use, etc.

Logging

Output from the script is logged to: 

/var/log/fundraising/pfp_pending_processing/out.log

These logs are rotated daily and will stick around for 7 days. When the log files are first rotated, a compressed copy is sent to storage3 for archival purposes. They are stored on storage3 in: 

/archive/fundraising/logs/pfp_pending_processor/out.<rotate_date>.gz

Where <rotate date> is the date the logs were rotated in the format YYYYMMDD_HHiiss (in UTC).

Access

There is no web access to this script. It currently gets periodically executed by a Hudson job.

Testing

There is a donation set up that mimics the credit card contribution page on payments.wikimedia.org, but passes everything through as tests (see #Credit_Card_donation_Test_Interface for more info). In order to trigger a donation that should be sent to the 'pending' queue (rather than going directly to the 'donations' queue), set a donation value for over $9999.

  1. Go to http://fundraising.wikimedia.org/test/mediawiki/index.php/Donate (only works inside of the WMF office a la payments.wikimedia.org)
  2. Fill out the form, be sure to set payment method to 'Credit Card'. Submit the form.
  3. Use some fake credit card information - be sure to get the appropriate credit card #s to use from the PayflowPro developer documentation (http://tinyurl.com/352axwu) - look on page 49.
  4. The transaction should then be logged in the 'test_pending_pfp' queue (see ActiveMQ to find out how to check).
  5. You can either manually execute the pending processor or wait for the Hudson job to run - to execute manually, just run php /path/to/executeStompPFPPendingProcessorSA.php
  6. You should then see the transaction pushed over to the 'test_donations' queue if it was a verifiable transaction.

Jenkins

We are using Jenkins as a cron-replacement for handling some of the automation of certain fundraising tasks. (Note that we were previously using Hudson, so some older documentation and code may refer to Hudson instead, although they are basically the same software.)

Why

Jenkins is like cron on 'roids - it offers many improvements over cron. The folks at FourKitchens turned us onto it when we were preparing for the fundraising infrastructure improvements in July 2010. They made a blog post extolling its virtues, so rather than repeat them, here's a link: http://fourkitchens.com/blog/2010/05/09/drop-cron-use-hudson-instead

The biggest advantages for our purposes are:

  • Jenkins sends us an email when a 'build' (a singular job execution) fails, and then again when there's a successful build following a failure.
  • Jenkins logs console output of the command it executes, and it can be reviewed through the Jenkins interface
  • Jenkins keeps track of how long it takes for builds to be executed

What we use it for

Jenkins is currently being used to replace all the Drupal/CiviCRM functionality that we would normally use cron for. The following is a list of 'jobs' that Jenkins manages:

  • Civicrm.wikimedia.org cron run - This executes the Drupal/CiviCRM cron.php script, which performs some automated tasks and periodic site maintenance (clearing caches, etc)
  • CiviMail Receive
  • CiviMail Send
  • Civicrm-dev.wikimedia.org cron run
  • dev CiviMail receive
  • dev CiviMail send
  • Donations queue consume - executes the Queue2CiviCRM drush command which triggers queue consumption
  • PayflowPro Queue Processing - executes the PayflowPro pending transaction verification script which verifies transactions in the pfp_pending queue.

Access

The Jenkins instance is available on port 8080 of fundraising.wikimedia.org. For an account, contact Arthur.

Issues

See Grosley#Potential_issues

Drush

For our Drupal/CiviCRM setup, we take advantage of Drush - a shell utility for Drupal. It facilitates performing certain Drupal tasks at the command line, which you would otherwise need to do through the Drupal web interface. It is possible to extend Drush by writing your own Drush script, which can give command-line functionality to your custom Drupal modules.

Files

We are currently using an antiquated version of Drush (v 3.x).

Drush is currently installed in /opt/drush, and the actual Drush utility is sym-linked to /usr/local/bin/drush. In order to use it, you can either run: 

$ drush -r <path to Drupal installation> --help

or 

$ cd /path/to/Drupal
$ drush --help

The '--help' option will print out further usage instructions.

CiviCRM

We are using a development version of the CiviCRM Drush extension (details [hhttp://civicrm.org/blogs/deepak/civicrm-and-drush-integration here]). This gives us some command line access to CiviCRM. While not actually necessary for any of the automated tasks we use Drush for, it makes performing some CiviCRM maintenance (eg clearing caches) and database actions possible to do from the command line.

Queue2CiviCRM

The Queue2CiviCRM module takes advantage of Drush. A Drush extension is packaged with the module (in queue2civicrm/queue_consume.drush.inc). This extension add a Drush command which will execute queue2civicrm - causing queue2civicrm to consume donations from the queue in batches, as configured in the Drupal interface. To execute queue2civicrm using Drush from the command line, run the following from within your Drupal directory: 

$ drush -v --user=1 queue-consume

'queue-consume' can also be replaced with its command alias, 'qc'. 'The '-v' adds a level of verbosity to the output of the command (which is useful - we use this to capture output in Hudson during queue consumption to help us identify where something went wrong in the event that things explode). '--user=1' causes Drupal to be bootstrapped as the admin user, which (based on the user permissions scheme in Drupal) is necessary for inserting contributions and contacts into CiviCRM.

Retrieved from "https://labtestwikitech.wikimedia.org/w/index.php?title=Fundraising.wikimedia.org&oldid=2224"