Structure.rest

A joined data pipeline made by chaining multiple queries together.
Alpha Logo design

Inspiration

Several months ago, I described a problem to Michael that seemed extremely large in scope to me. I love to build features, and show them off. Whenever I release a new feature, I post about it on Facebook, Hacker News, and my own Wordpress blog. I usually have a video that generates some engagement and a demo environment that I want people to try out. All of these sources were generating analytics data, but I had no real way of using it to guide my business decisions. I could release an article and get 300 views on Hacker News, but I didn’t know if that led to use of a specific feature or just general interest. I needed help from a data engineer.

Michael had some suggestions at the time, but connecting all of those APIs up was just not possible with the tools we had. We tabled the discussion, and I solved my problem by changing its scope - I just kept a close eye on analytics that seemed the most important: signups, page views, and in-app clicks that indicated users were getting value out of a newly released feature.

A few months later he told me about data warehousing and why it’s useful. He had been working on an idea where you could graph out SQL query models, and watch as they execute in real-time on a platform called Snowflake. This solved a real problem that I was already having. I wanted to analyze whether my marketing materials had a significant correlation with use of the features I was releasing. I dropped what I was working on, and started working with Michael on Structure.

What it does

A UI for writing and chaining SQL queries together.

How I built it

React-Redux-Typescript-Jest and blood-sweat-tears

Challenges I ran into

Configuring multiple tools in the space to work with each other is hard work.

Accomplishments that I'm proud of

The Login flow (using OAuth), the Github Rest APIv3 integration for persistence, the cool data pipelines that I have built with the product.

What I learned

Large project management involves building as flat of a file structure as possible. Typescript interfaces help a lot with providing extra information for how to use reusable functions.