Answers.Body: <p>@Andrei - thanks a lot for pointing me to the solution!</p>
  
  <p>Based on <a href="https://civicrm.stackexchange.com/questions/14566/how-do-i-create-a-token-that-automatically-gets-urls-convert-to-trackable-links">your answer</a>, I wrote this function that did the trick for me for finding and converting links in arbitrary <code>$html</code>: </p>
  
  <pre><code>function trackUrls($html, $contactId, $jobId){
  
      // Find all URLs
      preg_match_all('#\bhttps?://[^\s()&lt;&gt;]+(?:\([\w\d]+\)|([^[:punct:]\s]|/))#', $html, $matches);
  
      // We need to find the mailingId and the eventQueueId
      $mailingJob = civicrm_api3('MailingJob', 'getsingle', ['id' =&gt; $jobId]);
      $mailingEventQueue = civicrm_api3('MailingEventQueue', 'getsingle', [ 'job_id' =&gt; $jobId, 'contact_id' =&gt; $contactId]);
  
      foreach($matches[0] as $url){
          $replacements[$url] = CRM_Mailing_BAO_TrackableURL::getTrackerURL($url, $mailingJob['mailing_id'], $mailingEventQueue['id']);
      }
  
      $html = str_replace(array_keys($replacements), $replacements, $html);
  
      return $html;
  }
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-20T18:10:56.017Z
Answers.Text: Very nice! Glad it helped
Answers.UserDisplayName: Andrei
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-07T21:41:57.853Z
Answers.Text: We just need to be careful not to turn authenticated (ie. checksums) or links with other personal data in into CiviCRM's cheap and rather too simple tracked URLs - you could easily harvest all that info with a simple loop.
Answers.UserDisplayName: artfulrobot
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-07-20T17:38:53.107Z
Answers.LastActivityDate: 2017-07-20T17:38:53.107Z
Answers.UserDisplayName: michaelmcandrew
Body: <p>I have created a token using hook_civicrm_tokenValues that inserts content into a CiviMail. The content includes URLs that we want to track.</p>
  
  <p>As I understand it (and I don't understand it very well) in CiviMail, token substitution happens after links have been converted to trackable urls, which means that any links that are inserted into tokens will not / cannot be tracked.</p>
  
  <p>I'm wondering what the smartest way to get around this is?</p>
  
  <p>Options I am considering are:</p>
  
  <p>1) <strong>run the relevant CRM_Mailing_BAO_TrackableURL functions again once the tokens have been subsituted</strong> and observing if there are any side effects (maybe some stuff coming from other tokens should not be tracked). FWIW, it looks like CRM_Mailing_BAO_TrackableURL::getTrackerURL() already takes some steps to not track links in the format <code>civicrm/extern/</code> which would cover links like <code>civicrm/extern/url.php?u=129&amp;qid=77</code>.</p>
  
  <pre><code>// hack for basic CRM-1014 and CRM-1151 and CRM-3492 compliance:
  // let's not replace possible image URLs and CiviMail ones
  if (preg_match('/\.(png|jpg|jpeg|gif|css)[\'"]?$/i', $url)
    or substr_count($url, 'civicrm/extern/')
    or substr_count($url, 'civicrm/mailing/')
  ) {
    // let's not cache these, so they don't get &amp;qid= appended to them
    return $url;
  }
  </code></pre>
  
  <p>2) <strong>switch the order of execution so that token substitution happens before links have been converted to trackable urls</strong> (but there may important reasons why it happens in that particular order).</p>
  
  <p>3) <strong>create trackable URLs as part of the token generation</strong> though I wonder if I would be able to access the data I need for the appropriate CRM_Mailing_BAO_TrackableURL function at this point and if the function is expecting to be called in a certain context.</p>
  
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-18T09:06:01.117Z
Comments.Text: do you mean links that have been inserted into tokens using `hook_civicrm_tokenValues` ? I'm not sure if I follow your question
Comments.UserDisplayName: Andrei
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-18T10:24:05.943Z
Comments.Text: Yes - I've updated the question to reflect that.
Comments.UserDisplayName: michaelmcandrew
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-18T12:49:01.193Z
Comments.Text: I don't quite have an answer, I use a mix of 1 and 3, logic for particular URLs and then apply `CRM_Mailing_BAO__TrackableURL::getTrackerURL()` within `hook_civicrm_tokenValues`, if you are in the Mailing context the job id is available and you can get the rest of required params to build the trackable URLs, not for standalone email though, and I don't know about SMS
Comments.UserDisplayName: Andrei
Comments.ContentLicense: CC BY-SA 3.0
Comments.CreationDate: 2017-07-18T12:49:48.853Z
Comments.Text: See https://civicrm.stackexchange.com/q/14566/872
Comments.UserDisplayName: Andrei
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-12T16:10:03.367Z
Comments.Text: You can try my fork of flexmailer https://github.com/artfulrobot/org.civicrm.flexmailer/tree/issue/30
Comments.UserDisplayName: artfulrobot
Tags: <civimail><tokens><tracked-links>
Title: Tracking links that have been added via tokens
ThreadText: # Tracking links that have been added via tokens
  
  (Asked by: username_0 on 2017-07-18)
  
  I have created a token using hook\_civicrm\_tokenValues that inserts content into a CiviMail. The content includes URLs that we want to track.
  
  As I understand it (and I don't understand it very well) in CiviMail, token substitution happens after links have been converted to trackable urls, which means that any links that are inserted into tokens will not / cannot be tracked.
  
  I'm wondering what the smartest way to get around this is?
  
  Options I am considering are:
  
  1) **run the relevant CRM\_Mailing\_BAO\_TrackableURL functions again once the tokens have been subsituted** and observing if there are any side effects (maybe some stuff coming from other tokens should not be tracked). FWIW, it looks like CRM\_Mailing\_BAO\_TrackableURL::getTrackerURL() already takes some steps to not track links in the format `civicrm/extern/` which would cover links like `civicrm/extern/url.php?u=129&qid=77`.
  
  ```
  // hack for basic CRM-1014 and CRM-1151 and CRM-3492 compliance:
  // let's not replace possible image URLs and CiviMail ones
  if (preg_match('/\.(png|jpg|jpeg|gif|css)[\'"]?$/i', $url)
    or substr_count($url, 'civicrm/extern/')
    or substr_count($url, 'civicrm/mailing/')
  ) {
    // let's not cache these, so they don't get &qid= appended to them
    return $url;
  }
  
  ```
  
  2) **switch the order of execution so that token substitution happens before links have been converted to trackable urls** (but there may important reasons why it happens in that particular order).
  
  3) **create trackable URLs as part of the token generation** though I wonder if I would be able to access the data I need for the appropriate CRM\_Mailing\_BAO\_TrackableURL function at this point and if the function is expecting to be called in a certain context.
  
  **Tags:** civimail, tokens, tracked-links
  
  > **Comments:**
  > **username_1**: do you mean links that have been inserted into tokens using `hook_civicrm_tokenValues` ? I'm not sure if I follow your question  
  > **username_0**: Yes - I've updated the question to reflect that.  
  > **username_1**: I don't quite have an answer, I use a mix of 1 and 3, logic for particular URLs and then apply `CRM_Mailing_BAO__TrackableURL::getTrackerURL()` within `hook_civicrm_tokenValues`, if you are in the Mailing context the job id is available and you can get the rest of required params to build the trackable URLs, not for standalone email though, and I don't know about SMS  
  > **username_1**: See https://civicrm.stackexchange.com/q/14566/872  
  > **username_2**: You can try my fork of flexmailer https://github.com/username_2/org.civicrm.flexmailer/tree/issue/30  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2017-07-20. Score: 1 | Accepted answer
  
  @username_1 - thanks a lot for pointing me to the solution!
  
  Based on [your answer](https://civicrm.stackexchange.com/questions/14566/how-do-i-create-a-token-that-automatically-gets-urls-convert-to-trackable-links), I wrote this function that did the trick for me for finding and converting links in arbitrary `$html`:
  
  ```
  function trackUrls($html, $contactId, $jobId){
  
      // Find all URLs
      preg_match_all('#\bhttps?://[^\s()<>]+(?:\([\w\d]+\)|([^[:punct:]\s]|/))#', $html, $matches);
  
      // We need to find the mailingId and the eventQueueId
      $mailingJob = civicrm_api3('MailingJob', 'getsingle', ['id' => $jobId]);
      $mailingEventQueue = civicrm_api3('MailingEventQueue', 'getsingle', [ 'job_id' => $jobId, 'contact_id' => $contactId]);
  
      foreach($matches[0] as $url){
          $replacements[$url] = CRM_Mailing_BAO_TrackableURL::getTrackerURL($url, $mailingJob['mailing_id'], $mailingEventQueue['id']);
      }
  
      $html = str_replace(array_keys($replacements), $replacements, $html);
  
      return $html;
  }
  
  ```
  
  > **Comments:**
  > **username_1**: Very nice! Glad it helped  
  > **username_2**: We just need to be careful not to turn authenticated (ie. checksums) or links with other personal data in into CiviCRM's cheap and rather too simple tracked URLs - you could easily harvest all that info with a simple loop.  
  
  ---
  
  