If Data Have You, Consume It They Will

PokéAPI, the REST API for Pokémon, is a project I started in October 2013 with the intention of demonstrating good RESTful practices.

In the space of 48 hours I built a really hacky application and stuck it online on a tiny little Digital Ocean droplet.

PokeAPI

Because of the data it was providing, it gained popularity very quickly. There are hundreds of websites about Pokémon but none of the data they had was consumable through an API.

The weekend after I launched the project, I spent all my time learning about caching, distribution, and how to handle heavy traffic. I did not expect the application to get so popular. It was a pretty hectic time. It was fun.

I think one of the reasons it ended up processing so many requests was because of something my friend told me recently:

I think Chad has a really good point: No matter the data, even if it's only usable by 2 or 3 people someone will consume it. It's an idea I've thought about for a long time and it is nice to see friends who I hold in high regard thinking of the same thing.

PokeAPI Version 2

Since October 2013 PokéAPI has served over 13.5 million API requests, averaging 35 thousand API calls a day. It has been the source of a number of cool applications and has even been used in school lessons to help teach APIs.

Here's a chart of the requests for each day, with a yellow growth trend line:

chart_one

I really wish this wasn't a joke: some days, people on the internet made nearly 300,000 requests for data about Pokémon.

Pokémon.

Now, I have a confession: I never planned to build something that would be used so much. Because of this I've negelected the project and left out quite a lot of data. For example, generation 6 Pokémon (there have been that many) don't have any attributes at all. I never planned to set any time aside for adding new content and I've had hundreds of personal emails asking me to update it. I feel awful for neglecting what is obviously a beloved and fun API.

I'm sorry folks.

So, I think it's time I did something about it. Today I am announcing that PokeAPI will become an open source project and big updates will be made over the next few months which will include:

Deprecation?

Deprecating the V1 API will mean that no more data or changes will be made to it as of January 1st 2015. It will be put into a stable state and after that no more content or data will be added to it. Instead, all new development and new data will be added to the V2 API.

From January 2015 an extra HTTP Header will be in the responses from the API recommending people upgrade.

Contributing

The project can be forked on GitHub right now. If you want to contribute, look at the extensive list of issues that I have addressed there are a tonne that I need help on.

I will be dedicating the next 6 weeks of my time towards building this so if you want to contribute or support me, feel free to donate through PayPal:

This will help me keep the server up and running which so far I've been paying for out of my own pocket. I am also leaving my current role at Twilio to pursue more technical roles, so I can dedicate every day to this if I can afford to.

A long time ago...

I've always said there should be APIs for everything. As I said earlier:

"Any large data set, when easily accessible through an API, will be consumed by at least one developer."

So I'm starting a new project that I will be working on alongside the V2 updates for PokeAPI. The title of this post probably gave it away already...

The Star Wars API

The Star Wars API will provide a RESTful API for Planets, People, Starships, Vehicles, Species and Films from the six major Star Wars films. Why just these films? There is much dispute about canon and I believe George Lucas recently said that the major films would be considered cannon. To avoid data clashes and weird hacks to make things match up properly, I'll just keep it to the major films. For now.

In the same style as PokéAPI SWAPI will be a completely open API with generous rate limiting and a REST-ish design. What's more SWAPI will have some interesting Resource relations such as "Pilots" on Vehicles:

HTTP GET http://swapi.co/api/vehicle/1

HTTP/1.1 200 OK  
...
{
    "vehicle_id": "1",
    "name": "T-65 X-wing",
    "common_name": "X-wing",
    "pilots" : [
          { 
              "name": "Luke Sykwalker",
              "people_id": "1",
              "url": "/people/1",
          },
          { 
              "name": "Wedge Antilles",
              "people_id": "34",
              "url": "/people/34",
          }
    ],
    ...
}

and "Films" on planets:

HTTP GET http://swapi.co/api/planets/7

HTTP/1.1 200 OK  
...
{
    "planet_id": "7",
    "name": Mustafar,
    ...
    "films": [
        {
            "film_id": "3",
            "name" "Revenge of the Sith",
            "url": "/films/3"
        }
    ],
    ...
}

It should be a pretty fun API to build as no easily formatted data for this data exists yet. I might end up scraping Wookiepedia or just watching the films over and over to get the right content.

Want to help out? You can sign up to get early access to the API which will be coming in the next few months. I'd love some support so email me if you want to gather data for me or help me build the API.

( Also, depending on how people feel, I might erase episode 1 from the database completely. :D )

And yes: SWAPI will include data from Star Wars episode VII: The Force Awakens once I or my other data gatherers can get valid information on it.

If you've got any questions about all this contact me via email or tweet me

Here's to the next six weeks!



More Posts

Prev  A Simple API in Go

Next  Building A Star Wars API