WIP: Avoid unnecessary operations on the database.

[sunilpawar]

Overview

When your activity table has millions of rows, and mailing is scheduled with a large number of recipients. Every time an email is sent, a batch of 10 records gets recorded with bulk email activity. Every batch call, we fetch the activity ID.Although it is one call per batch, it can be avoided like the existing $activityTypeID variable.

The goal is to avoid select queries on big tables when mailing is sending...

Before

On every batch select query used to activity ID and activity target record type.

Avoid save/update operation on activity record.

After

Now only one call per job run.

[civibot]

🤖 Thank you for contributing to CiviCRM! ❤️ We will need to test and review this PR. 👷

Introduction for new contributors...

• If this is your first PR, an admin will greenlight automated testing with the command ok to test or add to whitelist.
• A series of tests will automatically run. You can see the results at the bottom of this page (if there are any problems, it will include a link to see what went wrong).
• A demo site will be built where anyone can try out a version of CiviCRM that includes your changes.
• If this process needs to be repeated, an admin will issue the command test this please to rerun tests and build a new demo site.
• Before this PR can be merged, it needs to be reviewed. Please keep in mind that reviewers are volunteers, and their response time can vary from a few hours to a few weeks depending on their availability and their knowledge of this particular part of CiviCRM.
• A great way to speed up this process is to "trade reviews" with someone - find an open PR that you feel able to review, and leave a comment like "I'm reviewing this now, could you please review mine?" (include a link to yours). You don't have to wait for a response to get started (and you don't have to stop at one!) the more you review, the faster this process goes for everyone 😄
• To ensure that you are credited properly in the final release notes, please add yourself to contributor-key.yml
• For more information about contributing, see CONTRIBUTING.md.

Quick links for reviewers...

• ➡️ PR Demo site (can be rebuilt with the command test this please).
• 📖 Review standards
• 🗒️ Review template (brief or verbose)

➡️ Online demo of this PR 🔗

[ufundo]

The idea makes sense to me - but is the scope of those statics restricted enough?

I fear if multiple different mailings were sent during the same cron job run, all of the contacts would get linked to the same activity.

I'm not sure of the calling code, but maybe a protected instance var on the MailingJob class could work ( something like this $this->recipientActivityId )?

[ufundo]

Also just noting IIRC that there's a CiviMail to disable the activity creation entirely, which I think is often a good idea at scale

[sunilpawar]

The idea makes sense to me - but is the scope of those statics restricted enough?

I fear if multiple different mailings were sent during the same cron job run, all of the contacts would get linked to the same activity.

I'm not sure of the calling code, but maybe a protected instance var on the MailingJob class could work ( something like this $this->recipientActivityId )?

@ufundo you are correct, let me think about it.

[seamuslee001]

Picking up @ufundo's feedback my thinking would be to do something like here

if (!isset(Civi::$statics[CLASS]['activity_ids'])) {
Civi::$statics[CLASS]['activity_ids'] = [];
}
if (!isset(Civi::$statics[CLASS]['activity_ids'][$this->mailing_id])) {
// run the query
and set Civi::$statics[CLASS]['activity_ids'][$this->mailing_id] = found activity id and then just use Civi::$statics[CLASS]['activity_ids'][$this->mailing_id] as the reference to the activity id going forward or something