Miami University Explorer
As part of the CSE 252 — Web Applications Programming class, I wanted to explore the open APIs Miami offers for courses and buildings. Courses, because the current online course list is incredibly aggravating to use and I wanted to building my own course explorer. And buildings because Miami would often give only the building code, e.g., HUG, yet have no directory to determine what building HUG is—you would have to search Google for some article or post that references both the building and code.
- Deno requires explicit permissions for file, network, and environment access
- Deno does not use package.json for module resolution
- Deno does not use npm; instead, Deno uses modules referenced as URLs or file paths
- Deno uses “ES Modules” and does not support require()
- All async actions in Deno return a promise. Thus Deno provides different APIs than Node
- Deno always dies on uncaught errors.
My original plan was the following stack:
- 🦕 Deno — a single binary executable
- 🪶 SQLite — a single file database
- 🏗️ Hugo — a single binary executable
- 🧁 Tufte.css — a single file css framework
- 🔎 Autocomplete.js – a single file autocomplete
Deno and SQLite would index the courses and buildings. Hugo would quickly build every page I wanted upfront. Tufte.css would handle all styling through semantic HTML. And Autocomplete.js would enable searching for courses with a text input.
I did end up exploring the APIs and creating all the API functions in TypeScript using Deno. This exploration gave a great deal of learnings as Deno is webby—it implements standard web APIs, so, when you get better at Deno, you get better at the web. And I learned about:
- Transport Layer Security
- Stream API
- User Timing API
- Transfer-Encoding HTTP Headers
Transport Layer Security
Trying to fetch the academic terms endpoint in Deno threw an
// // * Connected to ws.miamioh.edu (126.96.36.199) port 443 (#0) // * ALPN, offering h2 // * ALPN, offering http/1.1 //
Deno does not support weak ciphers. And this server is using an old insecure TLS version that Deno refuses to accept.
I don’t know anything about TLS handshakes or ciphers, so running
curl -v https://ws.miamioh.edu/api/academic/banner/v2/academicTerms/current was interesting, and I noticed this message
// SSL connection using TLSv1.2 / AES256-SHA256 // ALPN, server did not agree to a protocol
Then I came across SSL Labs and ran a test for the server and it returned:
- This server accepts RC4 cipher, but only with older protocols.
- This server does not support Forward Secrecy with the reference browsers.
- This server does not support Authenticated encryption (AEAD) cipher suites.
- This server supports TLS 1.0 and TLS 1.1.
So, yeah. Deno does not support weak ciphers. And this server is using an old insecure TLS version that Deno refuses to accept.
Streaming seems like an optimal solution to load large data in chunks as it comes in. But, I don’t know how to stream JSON, XML (other than e.g., ndjson). I started digging into the Fetch whatwg spec and crawling for solutions before coming across these streams experiments by Dean Hume that helped tremendously in implementing a streaming solution that I ultimately did not use.
User Timing API
Transfer-Encoding HTTP Headers
I learned about JSON streaming through newline-delimiters. When the Miami API would get over ~5Kb, it would stop sending Content-Length headers and instead send
Transfer-Encoding: chunked alongside newline-delimited JSON.
The Transfer-Encoding headers are:
Transfer-Encoding: chunked Transfer-Encoding: compress Transfer-Encoding: deflate Transfer-Encoding: gzip
Data is sent in a series of chunks. The Content-Length header is omitted in this case and at the beginning of each chunk you need to add the length of the current chunk in hexadecimal format, followed by
'\r\n'and then the chunk itself, followed by another
'\r\n'. The terminating chunk is a regular chunk, with the exception that its length is zero. It is followed by the trailer, which consists of a (possibly empty) sequence of header fields. (MDN)
This article on HTTP Chunked Encoding is helpful.
I built an MVP of this project in Deno with an SQLite module and starting writing my own ORM before it became more complex and less interoperable than I wanted it to be. I enjoy the flexibility and experience of Prisma, but Deno support was not there. I had imagined the steps being the following:
- Query data (from API)
- Map data to schema
- Write data to database
- Write JSON files from database
- Write HTML files from JSON files
But, turns out, Hugo cannot build pages from data source and Prisma had yet to support Deno.
I spun out a few of the Types and functions I had written into a small Deno script, a Miami University Enrollment Checker.
As time began crunching, I wanted to try out a non-serverless app and chose to start with the Remix Indie Stack:
- Static Types with TypeScript
- Code formatting with Prettier
- Styling with Tailwind
- Production-ready SQLite Database
- Database ORM with Prisma
- GitHub Actions for deploy on merge to production and staging environments
- Linting with ESLint
- Fly app deployment with Docker
- Healthcheck endpoint for Fly backups region fallbacks
The Remix Indie Stack also comes with a few things I didn’t use:
- Email/Password Authentication with cookie-based sessions
- End-to-end testing with Cypress
- Local third party request mocking with MSW
- Unit testing with Vitest and Testing Library
After moving to Remix, I originally planned on building out:
- JSON schema metadata to all course and building pages
- Online server-side search with Prisma/SQLite
- Offline client-side search with Cache API or IndexedDB API
But, I did not have time to implement these.
I ported all the API functions and TypeScript declarations to this new repo and began building out the routes (iterated a few times on the route structure):
- /courses/cse — CSE Classes at Miami University
- /courses/cse/252 — CSE 271 Web Application Programming classes at Miami University
- /courses/cse/a — CSE 271 Web Application Programming section A classes at Miami University
Using Remix, I routed all subjects, e.g., /courses/cse, to a $subjectCode route. To handle the varying subject subfolders I created a catch-all route and parsed out subject, code, and section to determine how to query the database.
Using Fly, I’m able to deploy the Remix and SQLite full-stack app easily. However, as this is a hobby project, I’m not looking to spend money on the servers. Sticking with the lowest memory option means the server is unable to handle some of the larger subject requests.
The Remix Indie Stack app code is available on Github. The code is branched as I would eventually like to remake this app in a more accessible way for beginner developers (Docker feels like getting beat over the head compared to serverless options). Using this project idea might be a good first real project with Rails or Django, to test out other “fullstack frameworks”. But, the current hosting options of serverless seem to provide greater levels of free tiers.
In the end, the routes with Remix ended up being:
/buildingsShow all buildings
/buildings/loadTrigger a server-side fetch of all buildings and load into database
/buildings/$buildingCodeView a building (e.g., HUG)
/coursesShow all course subject codes
/courses/$subjectCodeShow all courses in a subject (e.g., CSE)
/courses/$.Catch-all route to search for specific course codes or sections
/courses/loadTrigger a server-side fetch of all courses in a given term
/courses/randomRedirect to a random course
Thanks to these resources: