Open Payments Data #4: Things to come.

Sorry – no visualizations; Just a quick preview of things I’m hoping to roll out in the next month.

I am going to try to do something a little unorthodox: use Rstudio’s Shiny framework to launch data exploration app (ahem!).

…Sure, people use Shiny to build dashboards all the time, but the I don’t see many people trying to see how far they can. I’ve seen Shiny used  to deploy small scale applications, but I have something much larger in mind. In the absence of time to get more acquainted with Python and its Django web development framework, I’ll see how far I can push the tools I have at hand.

I already did something similar when I built an enterprise data management utility using it, last year. Nothing too grandiose – just something to automate manual workflows, analyse & manipulate unstandardized input files, and generate standardized outputs – but it was definitely something outside the traditional use-case scenario.

Now, it’s time to push the use-case scenario again.
The basic idea is to create a public application for exploring Open Payments data and showcasing various analyses I complete as they are ready to be published.

So far, I’ve built out the foundation ~ basic GUI and flexible directory query logic.


Considering that I’m sitting on 100GB of prepped data files right now, getting back-end development right on this thing is going to be crucial.

As it stands, the idea is to push my data out to Amazon Redshift and try to minimize resource strain on the application, itself.

Early takeaways:

  • The ‘DT’ and ‘data.table’ packages are your best friends since they are much more memory efficient when manipulating and presenting large datatables than native dataframes.
  • I am going to have to build a quite a few reference and data summary tables as I begin pushing visualizations.
    Trying to render visualizations off raw transaction table subsets is going to be too memory intensive. I’ll have to precalculate distribution parameters & such for all doctors and their corresponding sub-populations, and then use those values to render any visualizations.
  • CSS is alot of fun – I never used it too much before.
    It brings me back to the pre-MySpace glory days, when you had to build your own personal website using bare bones HTML.
  • Thank god for postgreSQL (if you’re bored).

The first iteration of the app will be patient centric, but I am hoping to quickly things up with secondary applications targeted towards manufactures and hospitals.
I’m going to try to make this thing as interactive as possible, while keeping up user-friendliness standards.

The ultimate goal is to have something that even your grandma would intuitively understand how to use.

On an unrelated note, Dan Carlin’s WWI podcast has been coming up a lot in conversation, recently. If you have 20 hours of free time and want to learn about the most brutal military conflict in human history, I highly recommend it.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s