Migrate to a Relation database

[qwo]

@bschoenfeld any objections about migrating it from mongo to a relational db? I feel like it could open up easier ways to query the data.

[bschoenfeld]

No objections.

On Wednesday, November 25, 2015, Stanley Zheng [email protected]
wrote:

@bschoenfeld https://github.com/bschoenfeld any objections about
migrating it from mongo to a relational db? I feel like it could open up
easier ways to query the data.

—
Reply to this email directly or view it on GitHub
#10.

[ttavenner]

What is the rough size/quantity of data were are currently working with?

[bschoenfeld]

There's a few ways to look at that. If we are just talking circuit court
criminal cases, we've got 110,000 records with detailed data, all from
2014, and those records are about 2.5KB each in Mongo. We've got 7,000,000
records with no details (just name, offense, and case number) which are 0.3
KB each. If we got all details for all 7M cases, we could be looking at
15-20 GB of data.

On Wed, Dec 9, 2015 at 6:44 PM, Tommy Tavenner [email protected]
wrote:

What is the rough size/quantity of data were are currently working with?

—
Reply to this email directly or view it on GitHub
#10 (comment)
.

[ttavenner]

Last Wednesday I started building out a server on Digital Ocean that could host the data and the API, if that is amenable to you. We should have plenty of space to get started and we can resize if need be. The data would be stored in Postgres and the API would be Nginx and Hapi.js. We can probably expose the DB externally so your scraper could write directly to it.