DEEP DIVE: Opioids ~ Unsupervised Methods for Kick-back Detection

Alternative Title: How to kill 6 days during Spring Break.

The unmasking of the opioids epidemic over the last several years has brought new scrutiny to marketing practices associated with high-risk pharmaceuticals products; Exposé journalistic efforts have been especially effective in shedding light on how misinformation and miscalculations of associated risk factors catalysed a torrent of addiction and in documenting individual stories of plight. In turn, there has been a slow but steady increase in investigative and prosecutory actions against pharmaceutical manufacturers that facilitated quid-por-quo kick-back schemes that financially rewarded physicians that artificially increase their prescription rates for addictive products.

Watching this unfold, I became interested in how we can use anecdotal evidence to develop new data models for identifying high-risk marketing practices. The application of such models could have transformative effects within the legal, journalistic and pharmaceutical sectors. Imagine taking a case-study of one back actor’s behaviour and using it as a framework for identifying similar activity patterns across the broader market.

This data-centric strategy would enhance investigative targeting and facilitate better management of available resources for legal and journalistic efforts. From a pharmacetical manufacturer perspective, it would allow companies to validate their marketing activities against their peers using a litmus test to see if their sales departments exhibit any hallmarks of high-risk practices. Furthermore, such an approach could be modified and re-purposed to target look-back project and assist with various program remediation efforts.

Specifically, I set out to develop a methodology for identifying pharmaceutical companies and products associated with high-risk marketing practices using only free and public data. For this build, I integrated various FDA & CMS data-sources and used Insys Therapeutics, Inc. as my bad actor subject.

Since available data sources don’t provide response class (e.g. prescription volumes), I experimented with various unsupervised learning methods before building the final model using hierarchical clustering. Output clusters were validated using a spot-check of negative news and sentiment analysis of relevant news headlines and article summaries pulled using the Google News API with the “company+product+marketing” query.

The subsequent sections will detail model design and demonstrate its effectiveness in using historic payment activity to isolate products associated with aggressive high-risk payment behaviours and current kick-back litigation.

The Data:

As far as free and public pharmaceutical data goes – you don’t get any better than the CMS Open Payments database, which is a ledger of pharmaceutical marketing transactions. I used an enhanced version of this data set, where I paired it with beneficial ownership information (i.e. “who owns that subsidiary?”), the FDA National Drug Catalog and the CMS Physician Information data set. Here’s a summary of enhancement advantages.

  • Beneficial Ownership: Discussed in a previous blog post, this was trace-back of all companies to their corresponding parents. The trace was based on SEC 10k scrubs and effective for subsidiary-filed transactions where the parent was a publicly traded entity on the US stock market. Given the dynamics of the local pharmaceutical market, this covers most major players with notable exceptions like Takeda Pharmaceuticals.
  • FDA National Drug Catalog (‘NDC’): This is a database of all pharmaceutical products approved for the US market and includes key product demographic information such as the marketing start date, biochemical method of action [MoA], primary active substance, and pharmacologic class [EPC].
    With regards to integration, there’s a false sense that CMS and NDC data can be easily joined over the common Product ndc-code field – this isn’t the case since the CMS ndc-code fields are not standardized and there are several ndc-code formats that are distinguishable by dash-separated sub-codes. Needless to say, almost all of the CMS transactions are listed as unparsed integer string, so I developed a fuzzy-matching logic that attempted various parsing structures and cross-validates potential match candidates using other quasi-common CMS, FDA and beneficial ownership fields. This amounted to about 12-hours of design, testing and validation. In the end, I was able to join ~99.5% non-validated codes with their standardized NDC counterparts.
    Not to be salty, but I would like to thank Janssen Pharmaceuticals (J&J) – and the like – for making my life especially difficult. Case in point, OLYSIO (for Hep. C) transactions were filed under 4459 erroneous NDC codes and INVOKANA (for Diabetes) transactions were filed under 801 codes. Please fix any outstanding data-quality issues.
  • CMS Physician Information: This is a companion directory file distributed by CMS that contains demographic information for every physician listed as receiving payments from pharmaceutical companies. It is effectively a subset of the NPPES registry and contains information like each physician’s primary commercial address, the states they are licensed to practice in and their associated practice specialty classes (e.g. “Allopathic & Osteopathic Physicians|Internal Medicine|Interventional Cardiology”).

The Case-Study:

The case of Insys Therapeutics was natural jumping off point because of the tremendous amount of media coverage regarding its kick-back program (here, here and here), with the indictment document for medical fraud by the USAO of Massachusetts proving to be the most robust source of anecdotal evidence of an illegal kick-back program.

So, what made the product so dangerous and how did everything go so wrong?

In 2011 Insys Therapeutics received FDA clearance for Subsys, a fentanyl based painkiller, for the singular use of addressing break-through cancer pain (BTCP). BTCP is defined as severe acute pain that overwhelms around-the-clock pain management medications used to control chronic pain associated with late-stage cancer. These bouts of crippling pain can last up to an hour and are estimated to affect up to 75% of individuals with late-stage cancer. As such, Subsys was designed to be a high-strength opioid product to be used intermittently on top of per-existing pain management medication by a narrow subset of population.

This was the flagship product and for a small pharmaceutical company, resigned to a small and crowded sector of the pharmaceutical space. As such, simple market dynamics and the lack of sufficient oversight can be identified as the contributing factors for bad actor behaviour.

What made the product especially dangerous was it’s sublingual delivery method. Typical pill-based opioids take time to enter the blood stream because they must travel to the gastrointestinal tract to be absorbed by the small intestines and processed by the liver. Subsys was designed to by-pass this natural delay mechanism by being applied underneath the tongue, where near-immediate adsorption can take place due to the profusion of capillaries – granting direct access to the bloodstream. Combined with it’s high-strength formulation, this design significantly increased the probability of abuse and dependence if prescribed outside of narrow approved target population.

However, competitive pressures of the opioid market meant that the product couldn’t be (as?) profitable if it was restricted to late-stage cancer patients. There were already four (4) other sublingual fentanyl products competing for a patient population of one to two million individuals, administered to by only a few hundred physicians nationwide, when the Subsys hit the market in March 2012. Failing to commercialize their flagship product posed significant financial risk for the company at large.

Consequently, the company devised and executed a kick-back scheme between 2012 and 2015 that targeted top prescribers  and financially rewarded them for prescribing Subsys for various non-BTCP (“off-label”) conditions. Insys Therapeutics procured third-party pharmacy data and used it to identify the top 10% of prescribers for competing sublingual fentanyl products, and then encouraged it’s sales representatives to aggressively pursue these physicians.

Insys Therapeutics cast a wide net – encouraging physicians to prescribe Subsys for numerous off-label uses. If the physician acquiesced, they were financially compensated using the company’s marketing budget. These physicians were treated to regular lavish dinners and nights out wheres Insys Therapeutics picked up the tab, with some receiving direct cash payments.

These marketing expenses were then reported to the CMS as compensation for education speaking engagements.

As noted in the indictment:

“Speaker Program events were often just social gatherings at high-priced restaurants that involved no education and no  presentation … [including] the same repeat attendees … friends of the co-conspirator practitioner [and] support staff employed by the speaker. Many speaker events had no attendees at all.”

These benefits amounted to anywhere from several to tens of thousands of dollars per physician, annually. In return for these benefits, physicians began prescribing Subsys to patients that did not have cancer and for conditions that should have been treated using traditional pain medications. When prescription rate expectation were not met by individual physicians, upper management actively used sales representatives to apply pressure on under-performing prescribers.

Per communications sent by Alec Burlakoff (VP of Sales) to a regional sales director and sales representative, respectively:

“[w]here is … [Practitioner #9], we cannot go a single day with out [sic] a prescription from … [Practitioner #9]. I do not want to hear excuses, we pay good money here (we need 1 a day from …[Practitioner #9]).”


“Where is … [Practitioner #9]? Not even close to meeting anyone’s expectations thus far, perhaps – We had failed in setting our expectations? We were looking to go from 40 percent market share to 90 percent? …I have to sit in the corporate office and answer these questions face to
face. It is not fun, and the recent move we made on an ABL (Area Business Liaison) appears as if it is potentially not worth it?”

New prescriptions were then fraudulently reimbursed using Medicare, Medicaid and private insurance programs with Insys Therapeutics’ Reimbursement unit masking real diagnoses.  Reimbursement unit personnel were instructed to give the following response if questions were raised by the insurance provider:

“The physician is aware that the medication is intended for the management of breakthrough pain in cancer patients. The physician is treating the patient for their pain (or breakthrough pain, whichever is applicable).”

Notably, this response, or variations thereof, sidesteps question all together while loosely implying that the patient was receiving the medication for a BTCP condition.

The Methodology:

~ Data Prep ~

As noted above, the kick-back scheme was executed through EOY 2015 and kick-back payments were filed as “Speaker Fees” with CMS. As such, I decided to limit my analysis to only payment data from the 2015 (PY2015), the peak of the kick-back scheme, due to high computation resource requirements. PY2015 data consisted of 7.6M transactions totaling $654M after excluding the following from the original data-set:

  • Any international, US territory, US military transactions because these were considered out-of-scope.
  • Any payments made in support of teaching hospitals because they do not yield benefits to any individual physicians.
  • Any transactions associated with products not covered by Medicare or Medicaid because pharmaceutical companies are not required to report product-level information for such transactions. Consequently, the information regard what product was actually promoted is absent for most payments in this category.
  • Products associated low payment activity; This exclusion was made after data aggregation and before anecdotal predictors were calculated, using a minimum value of $10K and 100 transactions.

I noted in a previous post that the native CMS data-structure is horrendous, so it was further processed to enable efficient querying.
See the post for technical details.

Under the CMS reporting schema, payments are reported under 15 different transaction types with three (3) speaker fee variants:

    1. Compensation for services other than consulting, including serving as faculty or as a speaker at an event other than a continuing education program.
      Example: “A physician who frequently prescribes a particular drug is invited by the company that makes that drug to talk about the medicine to other physicians at a local restaurant. The physician is paid for preparation time as well as the time spent giving the talk.”
    2. Compensation for serving as faculty or as a speaker for an unaccredited and non-certified continuing education program.
      Example: “Drug company Y gives money to a teaching hospital to help pay for the hospital’s annual course for its physicians. The course is an update on the latest treatments for diseases.”
    3. Compensation for serving as faculty or as a speaker for an accredited or certified continuing education program.
      Example: “Drug Company X gives money to a society to organize a continuing education program on a particular condition that Drug Company X produces a drug to treat. The company recommends Dr. C be chosen as one of the lecturers. Because the company specifically recommended the use of Dr. C, they must report the speaking fee under Dr. C’s name

Each of these three (3) transaction-types have unique distributions, as visualized and examined in a previous post. A cursory review of Subsys transactions found that payments were filed almost exclusive under the first of the three (hereafter referred to as “speaker payments” or “fees”).

Using indictment information, the 15 transaction-types were assigned Low, Medium and High Risk designations based on how easily they could be leveraged to facilitate a kick-back payments, giving consideration to average payment size and volume. Transaction types associated with the Subsys kick-back program were designated as high risk.

  • High Risk Transaction Types
    • Speaker at a venue other than a continuing education program
    • Entertainment
    • Travel and Lodging
  • Medium Risk Transaction Types
    • Consulting Fee
    • Food and Beverage
  • Low Risk Transaction Types
    • Charitable Contribution
    • Speaker for a certified continuing education program
    • Speaker for a non-certified continuing education program
    • Current or prospective ownership or investment interest
    • Education
    • Gift
    • Grant
    • Honoraria
    • Royalty or License
    • Space rental or facility fees (teaching hospital only)

~ Data Exploration ~

The first point of interest was that Subsys high-risk payment distributions did not deviate significantly from the broad opioid market or the pharmaceutical market, at large.

Constructed using a sample of 500 doctors receiving high-risk payments.
Aggregated over all Subsys 2015 high-risk payments, 228 physicians.


Here are some more distribution plots for select activity types to drive my point home.
Please excuse the sub-par visualizations, but these were meant my personal reference while exploring the data.

…Coding: (‘SUB‘ – Subsys, ‘OP‘ – Other Opiods’ and ‘BM‘ – Broad Pharma Market)

After a brief disheartening moment, I decided to step back ad take a broader of the the aggregated data – and there it was.

I noted that expenditure allocations (%) were differentiating factor, proving that sometimes things are simpler than they seem. Specifically, speaker payment accounted for approximately for 20% of total marketing expenditure when sampling across the broad market (excluding opioids), but 60% for opioid products overall and 80% for Subsys, in particular. This meant that for every $100 of Subsys promotional activity, $80 went to speaker fees associated with the kick-back program. Similar allocation deviations were observed for other medium and high-risk payment types.

~ Data Aggregation & Anecdotal Measures ~

Data aggregated using a parent-subsidiary-product stratification. This offered a number of advantages over simply aggregating on a product basis because, for example, a single product can be licensed out it’s original manufacturer to various third-parties. Furthermore, for companies with multiple subsidiaries marketing the same product, the marketing behaviour can vary significantly between subsidiaries depending on how decentralized the compliance controls are.

Various predictors were constructed using anecdotal evidence from the Insys indictment.

  1. Transaction Activity: This is just one of those things that makes sense, intuitively. Payment total value ($) and volume (N) were aggregated, log-values were used in downstream calculations. [Field: “Marketing_Expenditure_Total”]
  2. Payment Risk Allocation: Payment expenditure was aggregated across the three (3) risk categories and expressed at a percentage of total payment volume. [“High_Risk”, “Medium_Risk”]
  3. Market Breadth: Subsys was restricted to a small patient class, so I created a proxy measure for this attribute. Using an approach similar to the Payment Risk Allocation aggregation, I calculated marketing allocations across the six (6) physician-types and 199 physician specialties native to NPPES registry. Then, I measured skewness across these domains. [“tp_spread”, “HRtp_Spread”, “sp_spread”,”HRsp_spread”]
  4. Product Life Cycle: Based on the Subsys case-study, it became evident that newer products are more likely to be associated high-risk marketing activities because of the need to rapidly garner physician attention during early commercialization. I created a life-cycle proxy measure by calculating the difference between each product’s minimum marketing start date (FDA NDC) and December 31, 2015. The minimum date was used because the NDC database lists all approved versions of a product, with different dates for different approved dosages. Since there are data-quality issues with CMS ndc-code data, my matching algorithm was able to link erroneous codes to the correct NDC product listing, with ambiguity regarding the appropriate product version. Notably, the NDC database doesn’t have marketing start date information for around ~5% of records (primarily biologicals) so these were assigned population mean (5.7 years). [“YOM”]
  5. Offering Concentration Risk: Insys Therapeutics faced significant financial risk because of their single product portfolio, so I created a proxy measure using a count of unique products at the parent level. [“Offering”]
  6. Market Competition: Subsys was in competition with 4 other similar products when it hit the market in 2013, so I developed a proxy measure using pharmalogical class data sourced from the NDC database. This proved to be a decent, but not perfect measure. A better approach would have been to examine the number of products on the market with overlapping specific treatment approvals. However, this information is not available in any standardized FDA data-set. I found that I could source this information by building a web-crawl CenterWatch website. Unfortunately, this would probably take 30 hours for design, testing and integration, so I put it on the back-burner for now.
    However, I don’t want to give the impression that this measure is meaningless. Subsys, for example, is classified as a “Full Opioid Agonist” with 14 peer products. [“Peers”]
  7. Abuse Product Risk: Any product can prescribed inappropriately, however the risk is relative to product’s capacity for fostering abuse. Consequently, I leveraged the NDC DEA Drug Schedule field, scoring products not on the schedule as {0} and Schedule V, IV, III and II products as {1.0, .90, .70, .50}, respectively, in an effort to quantify relative risk. [“DEA_Schedule”]

The predictors that did not naturally fit on [0,1]  domain were then normalized so no single variable could dominate the clustering process. I refrained from using the traditional normalization technique of forcing each field to conform to a mean of zero (0) and standard deviation of one (1, “sd1”) because predictor distributions we significantly non-normal, even after log-transformations. Consequently, this method tended to introduce heavy, biasing, outliers when forcing the distribution.

Instead, I used cumulative density estimates of either gamma or exponential distributions, depending on fit-quality, after making log-transformations where necessary. This procedure had the benefits of total standardizing all variables onto the [0,1] domain and creating natural stratification between extreme highs and lows. For example, for the Total Expenditure predictor, the [0.30 – 0.35] domain encoded a much narrower range of values than the [.80 -.85] domain. Specifically, $118,777 to $149,879 and $3.1M to $6.3M, respectively.


~  Clustering & Sentiment Analysis~

Hierarchical clustering was used because it allows for easy visualization of the grouping procedure and doesn’t depend on any cluster shape or size uniformity. Clusters were grown using the average linkage method which evaluates the mean euclidean distance between the encoded predictors of adjacent elements and sub-groups. Single (best case) and complete (worst case) linkage methods were observed to create lob-sided trees with tenuous relationships between intra-cluster elements.

Initial clustering results.

The biggest challenge associated with this unsupervised technique is deciding where to stop the linkage process. Conventional wisdom says to stop the clustering process when growth rates have stabilized and only a few natural clusters are left, but there is no clear consensus when it comes to methodology and it becomes a judgement call when the number of natural groups is not known with any certainty.

Sure, eye-balling it works well enough, but it’s a pain when you’re constantly tweaking the code. I created a quick-and-dirty method for subdividing the tree into fast, medium and slow growth-rate sections akin to the elbow-method used for k-means modeling. The technique involved modeling tree growth by calculating linkage-distance increase relative to tree height at each iteration, and modeling the running-total against tree-height.

This yields a stable logarithmic relationship and can be accurately approximated using simple regression. I initially defined the thresholds between the low, medium and high growth-rate sections by where the slope equals 1 and .5 on the log approximation, but this tended to create per-mature cut-offs for long trees. After some experimentation, I found that the intersection points between log and linear regression curves produced consistent results across numerous test-models.

The clustering process was then halted at the threshold between the medium and slow growth-rate sections, generating 32 cluster partitions of varying size.

The visualizations below show each phase of the automated cutting process.

I used news sentiment to validate the cluster partitioning process. My expectation was to observe negative sentiment concentrated in the “Subsys” cluster and largely neutral coverage for the remaining clusters since individual pharmaceutical products don’t normally receive heavy coverage from major outlets. I used the News API, which is connected to all the major publications (e.g. NYT, WSJ, WAPO, etc.) and several aggregators,  to pull article headlines and summaries regarding marketing practices associated with each product (query: ‘company’ + ‘product’ + ‘marketing’).

Sentiment was grade using the sentimentr package, with the product sentiment taken as the net sentiment of news coverage for greater separation between lightly and heavily covered products.

The Findings:

To recap, indictment documents were reviewed to establish a list of quantifiable factors. Various data sources were then integrated and used to develop predictors associated with high-risk marketing and kick-back behaviour. A hierarchical clustering method was used scan the broad market and identify products with payment behaviours similar to Subsys.

The model did a good job of clustering high-risk & aggressively marketed products, with strong separation between DEA regulated & non-regulated products and strong grouping of products sold the same parent company.

Here are heat-maps for clusters with five (5) or more members, sorted by parent company name. If you open the full-size image and examine the row names, you’ll notice how the many-member clusters are typically dominated by a handful of large pharmaceutical companies, which can be attribute to high internal marketing controls resulting in consistent marketing behaviour and similar product features across their portfolios.

A overall market segmentation can be observed by plotting clusters relative to their distance from the “Subsys cluster” (ID = 2, a.k.a K02) and their mean allocation to high-risk payment types.
Points were sized based on the amount of members in each cluster and coloured by the mean news sentiment associated with cluster members.

The are several note-worthy observations:

  1. K02 is the only cluster to have a negative average sentiment score. This suggesting that the model successfully grouped all of the aggressively marketed opioid products that are currently receiving a high amount of negative news coverage based on historic 2015 payment data.
  2. Marketing for opioids and other DEA controlled substances is not uniform. There are numerous micro-clusters (largely restricted to a distance < 5) where the marketing allocation to high-risk payment types is less than 20%. More over, K17 is a cluster of opioid products with an allocation greater than high-risk Subsys cluster, but the model partitioned the two groups because significant difference in how the products are marketed. K17 products are characterized by small peer-product groups based on pharmacological class, parents with much larger product portfolios, and more mature product life-cycles. Interestingly, K17 is further differentiated from K02 based on member pharmalogical classes. K17 is composed of non-opioid products, whereas K02 contains almost exclusive opioid agonists.
  3. Large marketing allocations to high-risk transaction types is not unique to opioids. There are six (6) clusters predominately composed of products not under DEA control that have average high-risk payment allocations greater than or equal to K02. K06 & K21 stand out in particular because if you use point-to-point distance from K02 as proxy for risk associated with marketing behaviours, then their risk is particularly high – especially after you discount for how far the DEA Schedule predictor should pushing these clusters away.

This data can then formatted to visualize portfolio distributions by companies. Here I have plotted (%) of company products in each cluster, with the companies sorted top-to-bottom based on their total marketing expenditure and the clusters left-to-right based on their distance from the Subsys cluster. Things get messy when you label so many rows, but you can read company names if you click on the image and zoom-in.

Portfolio concentration ~ products. White is HIGH, Red is LOW.

Here is the same visualization, but showing the percentage of total marketing spend for products in each cluster.

Portfolio concentration ~ spend. White is HIGH, Red is LOW.

So what products are in high-risk K02, Subsys, cluster?

  • Androgel – “Ongoing battle against thousands of lawsuits that claim AndroGel, a once-blockbuster drug marketed to treat low testosterone in men, causes heart attacks, strokes and other injuries.”
  • Abstral – “Galena Biopharma has agreed to pay more than $7 million to settle allegations made in a whistleblower suit that the company gave kickbacks to doctors to boost prescriptions for the company’s sublingual fentanyl drug Abstral.”
  • Subsys – You know the story already.
  • Xartemis – “The drugmaker disclosed in August that it had received a subpoena from the U.S. Justice Department related to its promotional practices and sales involving opioid products including Exalgo and Xartemis XR.”
  • Ultiva – No negative news.
  • Hysinla -Purdue Pharma has pledged to stop marketing this and other opioids.
  • Oxycontin – Purdue Pharma has pledged to stop actively marketing this and other opioids.
  • Butrans – Purdue Pharma has pledged to stop actively marketing this and other opioids.
  • Vyvanse – No negative news; Central Nervous System Stimulant.
  • Vimpat – No negative news; Anti-epileptic.
  • Zohydro – No negative news, although the FDA advisory committee did recommended that Zohydro not be granted market access in 2013.

Again, the objective of this model and expanded forms is to guide due-diligence – queuing members of the “bad actor” cluster and those of nearby clusters, with similar risk factors expressions, for additional review to determine if payment activity is benign or indicative of additional illegal activity within the market. I hope I have clearly demonstrated how anecdotal evidence of illegal activities be used to easily develop unsupervised machine learning models that identify similar behaviours in the broad market.

Feel free to explore the aggregated data. The file has product information, as well as encoded predictor fields & cluster designations.

Download File: ModelOutputSummary

Other interesting things to do in the future:

  • Obviously, it would be interesting to apply this model over over 2016 and soon to be released 2017 data to see how market dynamics have changed in the market.
  • Conduct a deep-dive review of products in k02 neighbors and non-DEA regulated, high-risk payment clusters.
  • Scrub the CenterWatch website for better peer-analysis.
  • Not to go full FBI mode – but I have geo-coded all of the physician address information. It would be interesting to build a similar model at the physician level using case information from physician prosecutions, apply that over geospatial data and compare the results with information about opioid epidemic hot-spots.
  • Not to go full Deep State – but I have also paired the geo-data with Census Bureau demographic data, so it would be interesting to take any geospatial results and explore how  it maps on to different demographic-clusters.
  • Graduate from uni and live a calm and peaceful life.

And now, the new album by Galym Moldanazar which literally dropped TODAY; Explains the sudden US tour this month.
Saw this guy in on New Years two years back – absolutely brilliant.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s