Panorama Engineering Goes Civic Hacking!

At Panorama, one of our core company values is to contribute to the communities we are a part of and in the spirit of this value, one of the goals of the engineering team this quarter is to further this culture of giving back to the community. As part of this goal, we are undertaking initiatives to get involved with the thriving local developer community in Boston.

We identified CodeAcross Boston 2015, a civic tech hackathon organized by Code for Boston, as a cool local developer event to attend; David, Geoffrey and I represented Panorama at the weekend-long hackathon. Our goals for this event were twofold: first, get acquainted with other developers around Boston, and second, hack on a project that we believed could benefit the people of the city.

The Birth of

At the end of a day and a half of hacking, we came up with, a simple web app that lets subway commuters learn about any potential issues with the subway line using crowd-sourced updates from riders already on their way. The problem the application tries to solve is the lack of information in the regular MBTA alerts (most alerts only say “minor delays”) by adding feedback from other commuters as a measure of the real magnitude of a given problem. Geoffrey wrote a fantastic personal blog post outlining our experience building the app during the hackathon and describing some great takeaways from the event.

The app rests on top of two building blocks, both of which are open source.

  1. Meteor: Meteor, a JavaScript web framework built on top of Node.js and backed by a MongoDB database, forms the core of the platform. Meteor is emerging as one of the most prominent tools for rapid prototyping, and it turned out to be a perfect match for us: after only about 30 minutes of installing, running a meteor create mbta-ninja (creates an app with a set of default packages), and doing the introductory tutorial, we could build a simple app that exposes an interface to create alerts, persists them to the database and in real-time syncs updates with clients using simple HTML and isomorphic Javascript. The framework handles everything seamlessly.
  2. MaterializeCSSThe frontend rests firmly on Google’s material design guidelines and we leveraged MaterializeCSS’s reusable components and animations extensively to accomplish a frontend grounded in material design principles.

Thanks to these two awesome open source tools, at the end of the weekend, we had a fully functional web application hosted on Heroku for users to report, confirm and view crowd-sourced alerts on the MBTA’s red line on an interface optimized for mobile.


Ship Early, Ship Often

Any emerging startup these days is advised to release early and then iterate quickly. Although we don’t plan on becoming a startup (we love working at Panorama!), this advice proved to be really valuable for us. Within the first five days of launching the app, we had about 14,000 people visit the site over 21,000 times. We got featured in several news sources including BetaBoston, BDCWire, BostInno, and even got a short interview on CBS Boston. In the first two days we only had one subway line in the app, and realized we’d have to iterate on a few things fast.


From a product perspective, the biggest thing we had to iterate on was the alert expiration algorithm.

When we built the app, we really had no algorithm that decided if an alert should be displayed or not; all alerts expired after 30 minutes of their last confirmation. We realized quickly that this was a problem since an event that was reported as cleared by many people still persisted for 30 minutes from the time someone last confirmed it. For the past few weeks, the following simple alert expiration algorithm has served us well:

  • Points(Alert) = 20 when the alert is created,
  • Points(Alert) = Max(20, Points(Alert) + 10) when a user confirms the alert, 
  • Points(Alert) =Points(Alert) – 5 when a user marks the alert as cleared,
  • Points(Alert) = Points(Alert) -1 in a Cron job that runs every minute, and
  • An alert expires when Points(Alert) <= 0.

Let’s visualize how these rules pan out over the lifecycle of a simple alert.

As the figure depicts, with these rules, events that have been confirmed recently stay visible as the confirmations increase their points, while those that have been recently marked as cleared get removed fairly quickly.

On the side of taking the app from a prototype to a real product, we had to make some changes to how we were using Meteor.

During rush hours, when a lot of commuters were looking at our app, even 2X dynos on Heroku would throw memory errors. Several articles we found on running Meteor apps successfully in production pointed at one simple thing: refactor the code to remove the autopublish package that mirrors the entire database to the client (the client holds this database in memory using minimongo)  and the insecure package that grants clients write access to the database. Removing autopublish meant the server could publish to the clients only the alerts that have not expired, and not the entire database of all alerts ever created, reducing memory consumption greatly.

We also received requests to add more lines and transit options. The app now includes all subway lines, all commuter rail lines and the bus service to the airport.

So What’s Next for the Ninja?

With the Boston subway getting back on its feet, users don’t feel the need to check the site before starting their commute every day; the user base has converged to a few hundred daily active users. Sites that rely on user generated content (UGC) thrive on a user’s need to give back to a platform that creates value for them. If we have to keep users incentivized to report alerts on the platform, the platform needs to continue generating value for them. Most of our planned next steps are focused on continued value for our users.

  1. Native apps: Sending push notifications using native apps every day at the time of users’ commutes with alerts on the line they care about would create daily value. In our preliminary investigations, Meteor seems to plug in well with Phonegap and without much additional configuration can create native packages. Apps enable features related to location detection and geofencing.
  2. Greater transit specificity: With the silver line bus service, we added support for transit-specific alerts (for example “Bus stop inaccessible” alerts for snowed-in bus stops). With transit specificity, the commuter rail, which suffers from delays even during calm weather, has surfaced as a unique use case, which we hope to iterate upon. Although we treat the commuter rail the same as the subway on the app, it is inherently different due to its reduced frequency resulting in higher stakes around trains being missed.
  3. Integrating with MBTA alerts: Since crowdsourced alerts can add the missing user feedback to MBTA alerts, it would be useful to see both official and crowd-sourced alerts in a unified interface. Stations and routes are currently hardcoded in the app and as part of this, we hope to move to the standard GTFS format.
  4. Exposing an API: Flipping the previous point, the MBTA has expressed interest in using crowd-sourced alerts as a data source for official alerts, which currently are generated from unstructured data sources such as Twitter and video streams. Exposing our structured data in an API that the MBTA consumes will also tremendously aid in increasing engagement since users will be able to see additional value that their generated content creates.

In summary, building the Ninja has been a tremendously fun and rewarding experience for us, and we hope to keep increasing the project’s usefulness and impact. Check out the Github page to dig more into the code or to contribute.

Sound fun? We’re hiring!

Related Posts
A Homemade Distributed Computing Project
Ruby Gem – ExternalFields