Case Study

Amazon Wishlist Scrapper – How to get data from big companies if API does not allow it

Building a product from scratch is always a challenge but it is our everyday work so we are used to it. We are not afraid to create and test new solutions and that is what we did in that case. If there is no API from Amazon we will get data in our own way. And this is the story about that.

Not the first choice

When the customer came to us with his idea, we were pretty excited. It was the project that we knew will bring us some challenges, and that is what we love to do – find a solution to the problems. As always, we have proposed to organise the Product Design Workshop.

Unfortunately, we couldn’t go through with our plan as our customer decided to choose another software development company. Not for long – we got back in touch in December and started cooperation.

The idea

Our customer came to us with a great vision, but our job was to make it work. During the Product Design Workshop, we went together through topics like a business model, competition analysis, and technical requirements but the crucial part was to put all the great ideas into a consistent process.

The idea was to make gifting more accessible, faster, and less awkward. We wanted to give users an app that will let them connect with friends and easily buy a present from their wishlists. We wanted to use wishlist from big players like Amazon, Walmart, or Target and upload them to our app. Sounds easy, right?

First challenge

After a quick research, we found out that there is no Amazon API that would allow us to grab the user’s wishlist and display it in our app. Of course, we could contact Amazon and ask them to share with us this kind of data, but we would need to show them the value of this solution, and at the moment we were just a group of random people with the idea that meant nothing to the big company.

So we figured out that we can build a scraper that would only take the information from Amazon’s website. The wishlist would need to be public, the user would paste the link, and our scraper would grab the picture, the price, the description, and display it in our app. Of course, the shopping process would still be conducted through Amazon’s website. 

Is it legal?

We wanted to check first if creating a scraper is even legal. We have read online about cases that even went to court (https://www.imperva.com/blog/is-web-scraping-illegal/) and most of them were won by scraper creators. We did not found any law forbidding the usage of scrapers.

When you find yourself in a great area it is always good to think if your actions would harm anyone. After analysing the process we wanted to create, we concluded that there is no way that our app would have a bad influence on the business of scraped parties. If anything, we would be helping them by increasing the purchase of products shown in our app. So we decided to give it a green light.

Proof of concept

Deciding to go with the scraper was not resolving our problem. We still had to check if the scraper would actually work. When you choose to use data owned by other platforms, you need to be aware that there is a lot of factors that can stop you from doing that.

For example, some platforms have protection against attacks that detects bot visiting website and may block them. And our scraper was nothing else than just a bot. Also when you “teach” your scraper where on the website it will find the photo or the description of the product you need to remember that the platform may change its appearance in the future and your scraper will be lost.

With that in mind, we decided to start with a POC (Proof of Concept) stage that would assure us that our solution is working. We have engaged two developers into 2 weeks sprint to check it. We have validated our idea by creating scraper for Amazon as well as the process of inviting friends. We also got lucky and found a great library so we could actually give it to the customer for testing. And it is needless to say that it was a success.

Let’s build the MVP

As always when you are excited about your project and ideas, you may think that all the features you have in mind are necessary for the first version of the app, but it is important not to lose your mind. We have planned the next Product Design Workshop and created mockups for our MVP version of the app. Not only the customer had his ideas, but the whole team took part in the brainstorming session. We decided that we don’t want to limit ourselves to platforms that give you the opportunity to create a wishlist.

What if you find your ideal gift on some random website? You should have a chance to add it to the wishlist. And what if you want a game that you see at your friend’s house, but you don’t want to look it up on the Internet? You should just snap a photo and add it to your wishlist.

With that and a few other ideas, we have decided to create an in-app browser that would be able to grab data from any website just by clicking on a “add to wishlist” button. There were also a few questions like what if someone would buy the gift that another person already bought. Or how people will know what address they should send a gift? And we found a solution – you can check it out by downloading an app 😉 (https://beri.gifts/)

Team and organisation

For this project as a Product Owner, we had our customer who was not a technical person but was sure what are his expectations when it comes to the product. We booked two Fullstack Developers – one preferred working on a backend and the other on the frontend. This way, they could focus on their own parts, but they could still support each other in case they had any blockages.

We have also added a Customer Success Manager who was making sure that everything goes well, that developers understand the customer’s vision, and there is no miscommunication between the customer and the dev team. And last but not least – our Project Supervisor who was taking care of all the organisation issues and topics like budgets and estimations.

Basically, the issues that everyone hates, but without taking care of them, the cooperation would not go well. We gave ourselves 5 sprints that lasted a week each. We have conduct weekly review/retro/planning meetings + daily meetings. Our customer as a Product Owner was present on each of them what made our work easier.


Designs

As we like to give our customers a “full package” lack of designs or branding was not an issue here. The customer provided us with a name and a logo and left the rest in our hands. Our graphic designer started from 3 different designs based on inspirations that the customer sent us. When we have decided on a style, we wanted we began to work on each view.

Our designer was focused on UX/UI as well, so we had to change our vision a little bit. We simplified some of the views. Added modals that no one thought about before and everyone was happy with the final product. We also helped our customers with a simple landing page so it would match the design of the app.

You shall not pass!

So we managed to finish the app on time. The timing was great as it was the middle of the pandemic so our solution would resolve the problem of gifting while social distancing. The customer did some tests with his friends, and after a few fixes, we were ready to release the app. And our work usually ends here. The app on Google Store was released, and Apple was not so happy with our solution. Not only our customer was confused, so were we! In our decade long history of creating apps, we never had so many problems with releasing the app.

It wasn’t something that we could fix as Apple had problems mostly with the app logic and… the fact that we use data from Amazon and other third parties. There was a moment where all of us were afraid of what will happen next, but got it together and convinced the customer that there is no need to change the app. We just had to convince Apple that we were right. 🙂 After many messages and explanations, we did it! You can find the app in both stores and gift your friends with gifts of their dreams

Do you like this article? Share it with your friends:
Author:

Patrycja Jach

Write to me: p.jach@gmihub.com

More case studies

Build your product with GMI

Send us an inquiry and get the answer at the same day!

Hire Us