Jack Scott

WSU Learning Guide Parser

Live Link

I have uploaded this project into this server, it can be found here: WSU Learning Guide Parser

Project Background & Methodology

This project eventuated over the course of a weekend, started it on Saturday afternoon, and finished Sunday midday. My inspiration for completing the project was so I could upload a learning guide and have a quick overview of the general important information I needed quickly. Is it a faster process than just opening the learning guide and searching for the information? probably not, but I find it pretty cool regardless.

I wasn't sure where to start this project, so I figured I'd learn a bit of NodeJS in the process and decided to start with that. For this, I used the PDF.js library, with a bit of research I was able to quickly load the text from a single page, and search through that using the built in String functions in Javascript. From there, I made it combine every single page into one text block, and then I began writing functions to extract certain information from different parts of the document

Within a couple of hours, I had a mostly working example that was able to fetch key information such as Subject Coordinators, tutors, assessments etc from the document. From there, I moved the code out of NodeJS, and into a regular Javascript based webpage. The only challenge with this was processing the uploaded file into a datastream that PDF.js could then read, luckily this problem was solved online in the past.

Whilst its still somewhat unpolished, for all the learning guides I tested, it was able to extract most of the key information (namely the subject coordinator, and the assessment summary). At some point in the future, I wish to keep working on it, and improving how it parses the PDF.

Images

Before Parsing

Learning Guide Parser Interface Before Running

After Running

Bluetooth RGB Controller

Soure Code & Background

I have made this project opensource on Github, you can find it here.

This was a multi-faceted project that I completed some years ago. There are two main components to this project, the Android Application, and the Arduino code.

Android Application

At the time, the most challenging part of this project was the Android Application, firstly because I was inexpected with Java at the time (~2-3 years ago), but secondly because it involved sending data over a bluetooth connection, which seemed very complex despite trying to achieve a simple task. In the end I managed to get the application to send a HEX colour code from a colour picker over the bluetooth connection, which was then received by the Arduino, and decoded into the appropriate RGB values.

Circuit Board Design

My experience is in Programming, not circuit board design, so an electrical engineer might look at my PCB and laugh, however, the circuit board that I created achieved the job that I needed it to. I began this process by using EasyEDA, which allowed me to both draw my circuit, as well as draw the PCB Design. I was able to export the design files, and send them to a manufacturer to have them produced professionally. The circuit design is below:

Arduino Code

After completing the above two steps, I tackled the last problem of this project: the arduino code. This was arguably the most simple problem in the whole project, all the program does is read from the bluetooth input line, converts it into RGB values, and changes a PWM output based on the value, with the exception of the RGB value conversation, it was all stuff that I had done previously.

Fuel Price Map

Background

This started in a similar fashion to the train tracking project I noted above, in that I found an online API, and decided to figure out how to make something out of it. In this case, the NSW Government publishes several APIs on their platform they call "API.NSW". Included in this is a Fuel Price API, which is used to power the NSW Fuel Check app that the state government publishes.

I decided to use this API to create a basic project that allows you to inspect the fuel prices anywhere in NSW, and even restrict down to your location, and find the prices of service stations around your location (leveraging the Javascript geolocation API). Like most of the projects I complete, there are two parts of this, the first part is the backend data processor, this runs entirely on PHP, the script will pull the latest fuel prices and store the JSON it gets into memory, and then process the information it got. The database has a static list of stations, as they rarely change, and then the prices table in the database is updated using the new prices.

The frontend consists of two parts, a purely HTML/JS frontpage that uses AJAX to make requests to a PHP handler I wrote, and then the PHP handler which dynamically constructs the SQL query that is required. Dynamically strucuring the WHERE conditions in the SQL statements was an interesting problem, but one that several users on Stackoverflow had solved in the past, so I relied on that.

Images

Before Running

After Running

Public Transport Tracker

I began this project some time ago after I had discovered that Transport for NSW has an API that provides real time positions of all public transport vehicles in the network, this include trains, busses, ferries, light rail etc, it has allowed for some really cool tracking services online. For this project, I have only setup an injestion script for the trains API, however, I could expand this out to track busses, ferries, and other modes of transport. Expansion would just require adding a new table to the database, and then adjusting the API to provide the new dataset in GeoJSON format.

I did this project because I wanted to learn more about GeoJSON, and the GTFS data that is provided from the API

Backend

There are two parts of the backend, the python script that fetches the data from the API, decodes it, and stores it into the database, and the PHP "API" which powers the front end. It is possible to write both parts in PHP, which I intend to do sometime in the future, but when I began this project I didn't plan for the web interface, hence I didn't write the data ingestion in PHP.

The PHP API is relatively easy, all it is doing is fetching records from the past 12 hours, and then forming a GeoJSON feature collection which can be used by the frontend to display the train positions on a map. The most complex part of this is forming the GeoJSON, which took several attempts to get the correct formatting that the frontend would process correctly.

Frontend

The frontend for this project is very simple, its a full screen Leaflet.js map that has some Javascript running the background that makes an AJAX request every 30 seconds to the PHP API, and then processes that and displays it on the map. The Leaflet realtime plugin is able to dynamically move the position of the map markers as the train positions change in the database.

Images

Reddit Bots

Background

On Reddit, there is a subreddit known as RequestABot, where users can go, and request for others to create them Reddit bots, usually for free, but sometimes those users do offer payment. For a period of time, I was an active contributor to the subreddit, and I ended up completing several requests, most of which I have published on my Github, so anyone who might require the same bot can access the source code for it.

Account Cleaner

This bot is designed to overwrite all the text of posts and comments on an account, and then delete those posts and comments, essentially scrubbing the users account. This means that reddit history services such as Pushshift cannot see what the true content of those things are

Gfycat to Redgifs Converter

Gfycat and Redgifs (NSFW) are both GIF hosting services that are owned by the same company, they used to be one platform, but at some stage split into two separate platforms. This bot, after being called by a user takes the URL from the post, and converts it from the old Gfycat format, which doesn't include sound into the new Redgifs format which includes sound, and then responds to the user with that new link.

Gilding Checker

This was an request that I abandoned before I finished it, however, the main functionality is working. It goes through the top posts in a specified subreddit for that week, and then prints out how many awards each post has.

MC Video Downloader

This requester wanted a script that could take the top 50 posts from a certain subreddit, and download all videos that appeared in those top 50 posts. This script searches the top 50, collects an array of the videos that are posted, and then uses the python package urllib to download those videos to the users computer.

MadeMeSmile Downloader

This is very similar to the bot above, where a user wanted to download videos from a certain subreddit over a specified time period, however, the difference for this one is that the downloading has a lot more error checking, meaning that downloads are more likely to succeed instead of failing.

Mod Queue Bulk Approve

This bot/script gets the whole mod queue from a users account, and then approves every post in that queue. This bot was requested because the poster had an abandoned subreddit which had hundreds of posts in the modqueue that would've taken a long time to manually approve each one.

Record Flairs

This bot periodically scans the whole subreddit for users, and the flairs that they have given themselves, the script then keeps a record of which user has a flair, and records their names in the Subreddit wiki page. This process is only designed to run once as it stores a state of the processed users.

Remove Automod Comment

This subreddit moderator that requested this bot had a system setup for verifying users of a specific subreddit, if a user wasn't verified, then Reddits AutoModerator would put a comment on that users post informing everyone that the user posting wasn't verified. The bot requester wanted a bot that could go through posts, and check if a user had been verified the automod comment, and if so, remove the automod comment.

Respond to Modmail Keyword

The name of this bot is fairly self explanatory, it searches for a predefined list of trigger words in the subreddit Modmail, and if it finds an any instance of those words, then it responds with a predefined response. This is useful if you are getting a lot of the same complaints etc.

Reddit Discord Message Posting

This bot continually watches the comment stream of a subreddit, and if it seems its own name, then it will use Discord Webhooks to send a message to a specified discord server/channel.

My Portfolio

Github

Before Parsing

After Running