Data & techie stuff for BAFS 2017

[Rod Smith]

@cvcalhoun 152766 wrote:

Yes. Go to freezingsaddles.com/authorize and authorize it to read your private rides.

Thanks!

[BobCochran]

Is there any way to get regular data updates in JSON, TSV, or CSV format? JSON preferred. I’m interested in plugging it into MongoDB just for my own NoSQL training.

If it is not possible, this is just fine.

Thanks a ton

Bob

[jrenaut]

@BobCochran 152875 wrote:

Is there any way to get regular data updates in JSON, TSV, or CSV format? JSON preferred. I’m interested in plugging it into MongoDB just for my own NoSQL training.

If it is not possible, this is just fine.

Thanks a ton

Bob

It’s not impossible, it’s just a fair bit of work. It’s possible we could do a one-time dump of some of the data, but we’d have to discuss privacy issues and what we would and wouldn’t provide.

[hozn]

@BobCochran 152875 wrote:

Is there any way to get regular data updates in JSON, TSV, or CSV format? JSON preferred. I’m interested in plugging it into MongoDB just for my own NoSQL training.

If it is not possible, this is just fine.

Thanks a ton

Bob

Are you looking for leaderboard data (there may already be an API for this) or individual ride data?

If the latter, I would suggest that you can simply submit a pull request (https://github.com/hozn/freezingsaddles) for a new API that addresses privacy concerns — i.e. one that respects rides marked private (doesn’t export them) and maybe anonymizes user names/IDs and ride titles / IDs.

[BobCochran]

@hozn 152888 wrote:

Are you looking for leaderboard data (there may already be an API for this) or individual ride data?

If the latter, I would suggest that you can simply submit a pull request (https://github.com/hozn/freezingsaddles) for a new API that addresses privacy concerns — i.e. one that respects rides marked private (doesn’t export them) and maybe anonymizes user names/IDs and ride titles / IDs.

Hi Hozn,

Thanks for getting back to me. This is not at all an urgent issue and I don’t need the data. There is no reason at all to make you do extra work, either. The reason for my inquiry was idle curiosity and I understand about the issues you are bringing up. So don’t sweat this!

With that said I’ll look over the GitHub page you mention. I know a bit of Node and ES6 (although my use of Promises needs a lot more improvement) and I’ve taken some of the free MongoDB classes. MongoDB, the company, does provide free training databases and I have most of them on my server. Maybe I can be an assist to you in some other ways.

Thanks a ton

Bob

[hozn]

I have been considering (once I find more time) making an ES6 (Aurelia framework) interactive version of the leaderboards. Lately I am doing a lot of ES6 development, though I am not a UI developer.

Yeah, probably there won’t be any development on the public API unless you want to volunteer to do the work. We have way more ideas than people.

[BobCochran]

@hozn 152896 wrote:

I have been considering (once I find more time) making an ES6 (Aurelia framework) interactive version of the leaderboards. Lately I am doing a lot of ES6 development, though I am not a UI developer.

Yeah, probably there won’t be any development on the public API unless you want to volunteer to do the work. We have way more ideas than people.

I’m just looking over your repository now. I’ll probably fork it in a few minutes and play with it over the coming weeks. I used to work with MySQL. These days, NoSQL has my attention. I’ll look things over…no promises on my part…but perhaps in 2017 I can be a genuine help to you in some way.

Bob

[BobCochran]

I’m still looking at the repo, and calculating in terms of small contributions to the current implementation plus a different implementation using MongoDB as the database and Node.js as the language. The aggregation pipeline processing offered by MongoDB is great to use. This would be a pretty different project to work on through 2017. That means “months of programming.”

As mentioned, I’ll also make small pull requests to help provide documentation on the existing code.

Thanks a ton

Bob

[jrenaut]

Hozn may disagree, but I don’t think aggregation is our bottleneck – MySQL is sufficient for that (even though hozn hates it). Our bottlenecks are Strava API rate limits, which we can’t do anything about, and developer bandwidth, which is theoretically fixable but hasn’t happened yet.

[BobCochran]

Thanks, Jon! Maybe some riders could be instructed to use RideWithGPS … doesn’t Google own them? I extract lots of data from YouTube using the Google API for that. So maybe RideWithGPS has an accessible API, I don’t know. And perhaps they too have rate limits.

I’ll feel free to do my own experimentation with MongoDB. I’ve taken their free online classes and I really like that aggregation pipeline. Add a little Node here and some Express there and a nice security certificate and it can get interesting.

If I come up with anything useful at the end of 2017, and I might not come up with a thing, it will be there on GitHub for you and Hans and anyone else to grab. We will see.

Thanks a ton

Bob

[BobCochran]

And I agree with Hans that MySQL is not a great database. It’s just so 1990s.

Bob

[hozn]

Yeah, I like to complain about MySQL, but I don’t hate relational databases. Postgres is an enterprise-grade database and is fantastic. But MySQL is sufficient as a storage engine for this project and honestly this data is quite well suited to relational model. The advantage to mongo would be ability to store entire activity documents and query them, but I would be interested to see the performance of ad-hoc aggregations, etc. To be fair I have only used mongo for basic storage and retrieval.

But if you like JS enough to consider Node, I am sure we could use your help on the UI.

[jrenaut]

I hadn’t thought of that – being able to store and search an entire TCX file or equivalent would be interesting. Likely a ton of work, though.

[BobCochran]

I’ll study what you folks have…perhaps I can be a help to you over 2017.

I was thinking over dinner that it would be interesting for an app to be just like Git itself: the app needs to be distributed in nature. Like if the app could be packed into a container of some sort like a Docker container. Small. Lean. mean. Instead of having one copy of the app, let there be any number of copies, which can all speak to each other. So if you have n riders out there, let them be n workers each with a copy of the full app in a container. Some sort of a container.

A container needs an engine it can harness. It has to be an engine that just about everyone has. A cell phone could be that engine. I would say that very few riders would go anywhere on their bicycles without at least a cell phone. But there are other engines too. Garmin devices, for example, although discussing Garmins is wading into quicksand. Well…what else? A single board computer of some sort.

So you have n rides out there and each one has a cell phone and the phone could be running a nice virtual machine in the form of a Docker container. And no I don’t know Docker well. I haven’t studied it closely and most of all played with it the way I should. But anyhow these riders are each running a virtual machine and inside that machine, a Freezing Saddles app is humming away. It’s recording the miles the rider is doing. It’s storing that data away. Its being assisted by Strava or Ride With GPS or whatever, but that individual rider isn’t being rate limited. Then the ride finishes and what happens? All these containers start talking to each other, just like git repositories talk to each other. They push and pull data. Suddenly each person has a completely up-to-date copy of everyone else’s ride statistics. Everyone has the same leaderboards, and they are like no more than a few minutes out of sync with all the other rider’s leaderboards. And that includes you folks, the Freezing Saddles app people. So suddenly no one loses the full data and leaderboard standings. If their phones get flushed down the toilet, its no sweat. They can install the container into their new phone, start it running, and be up date in a jiffy because all those other riders are totally up to date and their containers are talking to each other and the “data-less” container on the replacement phone. Data-less? Not for long!

These container apps ignore trainer rides because trainers are stationery and their grid coordinates are not changing.

So I’m not saying this is an answer to the limitations you folks are facing. I’m just throwing out an idea. I bet someone else has done this before me. So I’m not even offering an original idea.

Meanwhile, Node, or some other ES6/ES7 scripting engine, Express, and MongoDB all containerize well.

Even SQLite containerizes well.

Anyhow I’m just throwing ideas out. I’m not telling you guys how you ought to do things. You have the app running, and it is running perfectly fine for now. I have a lot to learn about how you implement Freezing Saddles as an app. Once I understand that a lot better, perhaps I’ll be useful to you. (Grin.)

Bob

[jrenaut]

If you can get that distributed model to work at scale, I can find you a half a billion dollars in venture capital.

[Author]

Posts