Hackathon scraper

• 

  Github commits

• 

  Github project interface

• 

  Codebase

• 

  Generated .json file

Inspiration

Having to search for hackathons to participate in can be a bit stressful, so having an application that gathers current running hackathons from different websites can relieve the stress to an extent.

What it does

For now, it scrapes devpost, a website that is specialized in posting current running hackathons.

How we built it

It was build on the python programming language, with libraries like selenium and beautiful soup, to scrape the website and to parse html contents into readable files respectively.

Challenges we ran into

It was a bit challenging having to scrape websites with dynamic contents, as I was much more familiar with the python requests library, but that did not work for this project.

Accomplishments that we're proud of

Seeing the hackathons beautifully converted into a .json file was something I'm proud of.

What we learned

I learnt:

1. How to scrape websites with dynamic contents.
2. How to parse html generated response. 3, How to lint files with Ruff linter and github actions.

What's next for Hackathon scraper

Hackathon Scraper can be improved on to fetch data from more websites, this can be converted later on into an API, which will then be published online.

Built With

• beautiful-soup
• python
• selenium