NYCPHP Meetup

[Justin Dearing]

On Mon, Feb 15, 2010 at 6:23 AM, Leam Hall <leam at reuel.net> wrote:

>
> What I don't understand is why you need to have a cron job to deal with the
> user data. Why not have your processing script called when the user submits?
> This keeps your script from having to go through the entire database to find
> uncommitted changes. If you're going to have a gazillion row database aren't
> you going to be spending a lot of time on queries just to find those not
> committed?
>
>
Disconnecting the submission and the processing is helpful in a lot of
situations. Sometimes you have a small amount of data that takes a lot of
processing. For example an ETL (
http://en.wikipedia.org/wiki/Extract,_transform,_load)  job that has a lot
of validation. If I am an expediting warehouse and I receive pick and ship
orders from online storefronts via XML files, I will want to make sure my
inventory database says I can fill the pick and ship orders. I might receive
these XML file via a web service that does basic schema validation and
returns a receipt of some kind.Those XML files can go into a queue (be it a
loading table, directory in a file system, MSMQ/ActiveMQ/TibcoQ or other
technology). At some point the files will be processed, pick and ship orders
put in another queue, a status message prepared for the storefront that
submitted the order.

This disconnect provides you several advantages. For one, "threading" gets
easier when you have a disconnect between submission and receiving. If
submission order is not that important, you can scale your submission and
processing ends separately. Second, your submitters don't have to wait for
you to process their data. They can wait for the report and deal with the
problems later. Third, you can schedule database writes to not occur during
peak read times.

Regards,

Justin Dearing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.nyphp.org/pipermail/talk/attachments/20100215/8f21d253/attachment.html>