Lukas Murdock
Subscribe for updates and more.

Miami University Explorer

Planted 02022-05-06

As part of the CSE 252 — Web Applications Programming class, I wanted to explore the open APIs Miami offers for courses and buildings. Courses, because the current online course list is incredibly aggravating to use and I wanted to building my own course explorer. And buildings because Miami would often give only the building code, e.g., HUG, yet have no directory to determine what building HUG is—you would have to search Google for some article or post that references both the building and code.

I wanted to use this project to explore Deno (TIL pronounced Deeno not Dehno). A few differences between Deno vs. Node:

  • Deno requires explicit permissions for file, network, and environment access
  • Deno does not use package.json for module resolution
  • Deno does not use npm; instead, Deno uses modules referenced as URLs or file paths
  • Deno uses “ES Modules” and does not support require()
  • All async actions in Deno return a promise. Thus Deno provides different APIs than Node
  • Deno always dies on uncaught errors.

My original plan was the following stack:

  • 🦕 Deno — a single binary executable
  • 🪶 SQLite — a single file database
  • 🏗️ Hugo — a single binary executable
  • 🧁 Tufte.css — a single file css framework
  • 🔎 Autocomplete.js – a single file autocomplete

Deno and SQLite would index the courses and buildings. Hugo would quickly build every page I wanted upfront. Tufte.css would handle all styling through semantic HTML. And Autocomplete.js would enable searching for courses with a text input.

I did end up exploring the APIs and creating all the API functions in TypeScript using Deno. This exploration gave a great deal of learnings as Deno is webby—it implements standard web APIs, so, when you get better at Deno, you get better at the web. And I learned about:

  • Transport Layer Security
  • Stream API
  • User Timing API
  • Transfer-Encoding HTTP Headers

Transport Layer Security

Trying to fetch the academic terms endpoint in Deno threw an HandshakeFailure error.

//
// * Connected to ws.miamioh.edu (134.53.247.53) port 443 (#0)
// * ALPN, offering h2
// * ALPN, offering http/1.1
//

Deno does not support weak ciphers. And this server is using an old insecure TLS version that Deno refuses to accept.

I don’t know anything about TLS handshakes or ciphers, so running curl -v https://ws.miamioh.edu/api/academic/banner/v2/academicTerms/current was interesting, and I noticed this message


// SSL connection using TLSv1.2 / AES256-SHA256
// ALPN, server did not agree to a protocol

Then I came across SSL Labs and ran a test for the server and it returned:

  • This server accepts RC4 cipher, but only with older protocols.
  • This server does not support Forward Secrecy with the reference browsers.
  • This server does not support Authenticated encryption (AEAD) cipher suites.
  • This server supports TLS 1.0 and TLS 1.1.

So, yeah. Deno does not support weak ciphers. And this server is using an old insecure TLS version that Deno refuses to accept.

Stream API

Streaming seems like an optimal solution to load large data in chunks as it comes in. But, I don’t know how to stream JSON, XML (other than e.g., ndjson). I started digging into the Fetch whatwg spec and crawling for solutions before coming across these streams experiments by Dean Hume that helped tremendously in implementing a streaming solution that I ultimately did not use.

Other resources:

User Timing API

Performance measures.

Transfer-Encoding HTTP Headers

I learned about JSON streaming through newline-delimiters. When the Miami API would get over ~5Kb, it would stop sending Content-Length headers and instead send Transfer-Encoding: chunked alongside newline-delimited JSON.

The Transfer-Encoding headers are:

Transfer-Encoding: chunked
Transfer-Encoding: compress
Transfer-Encoding: deflate
Transfer-Encoding: gzip

Chunked directive

Data is sent in a series of chunks. The Content-Length header is omitted in this case and at the beginning of each chunk you need to add the length of the current chunk in hexadecimal format, followed by '\r\n' and then the chunk itself, followed by another '\r\n'. The terminating chunk is a regular chunk, with the exception that its length is zero. It is followed by the trailer, which consists of a (possibly empty) sequence of header fields. (MDN)

This article on HTTP Chunked Encoding is helpful.

I built an MVP of this project in Deno with an SQLite module and starting writing my own ORM before it became more complex and less interoperable than I wanted it to be. I enjoy the flexibility and experience of Prisma, but Deno support was not there. I had imagined the steps being the following:

  1. Query data (from API)
  2. Map data to schema
  3. Write data to database
  4. Write JSON files from database
  5. Write HTML files from JSON files

But, turns out, Hugo cannot build pages from data source and Prisma had yet to support Deno.

I spun out a few of the Types and functions I had written into a small Deno script, a Miami University Enrollment Checker.

As time began crunching, I wanted to try out a non-serverless app and chose to start with the Remix Indie Stack:

The Remix Indie Stack also comes with a few things I didn’t use:

After moving to Remix, I originally planned on building out:

But, I did not have time to implement these.

I ported all the API functions and TypeScript declarations to this new repo and began building out the routes (iterated a few times on the route structure):

  • /buildings/${buildingCode}
  • /courses/${subjectCode}
  • /courses/${subjectCode}/${courseCode}
  • /courses/${subjectCode}/${section}

For example:

  • /buildings/hug
  • /courses/cse — CSE Classes at Miami University
  • /courses/cse/252 — CSE 271 Web Application Programming classes at Miami University
  • /courses/cse/a — CSE 271 Web Application Programming section A classes at Miami University

Using Remix, I routed all subjects, e.g., /courses/cse, to a $subjectCode route. To handle the varying subject subfolders I created a catch-all route and parsed out subject, code, and section to determine how to query the database.

Using Fly, I’m able to deploy the Remix and SQLite full-stack app easily. However, as this is a hobby project, I’m not looking to spend money on the servers. Sticking with the lowest memory option means the server is unable to handle some of the larger subject requests.

The Remix Indie Stack app code is available on Github. The code is branched as I would eventually like to remake this app in a more accessible way for beginner developers (Docker feels like getting beat over the head compared to serverless options). Using this project idea might be a good first real project with Rails or Django, to test out other “fullstack frameworks”. But, the current hosting options of serverless seem to provide greater levels of free tiers.

In the end, the routes with Remix ended up being:

  • /buildings Show all buildings
  • /buildings/load Trigger a server-side fetch of all buildings and load into database
  • /buildings/$buildingCode View a building (e.g., HUG)
  • /courses Show all course subject codes
  • /courses/$subjectCode Show all courses in a subject (e.g., CSE)
  • /courses/$. Catch-all route to search for specific course codes or sections
  • /courses/load Trigger a server-side fetch of all courses in a given term
  • /courses/random Redirect to a random course

Acknowledgements

Thanks to these resources: