Amazon Wishlist Scrapper – How to get data from big companies if API doesn’t allow it

Building a product from scratch is always a challenge, but it’s also our everyday job, so we’ve gotten used to it. We are not afraid to create and test new solutions and that is what we did in this case. If there is no API to retrieve the data we will do it in our own way. And that’s what this story is about.

Not the first choice

When our client came to us with their idea we were excited. We knew this project would bring us many challenges, and that’s what we love to do – find solutions to those challenges. As always, we offered to host a Product Design Workshop.

Unfortunately, they didn’t come to fruition because the client decided to work with another software house. But not for long. A few months later he contacted us again and we were able to start working together.

The idea

Our client came to us with a great idea, but our job was to make it work. During the Product Design Workshop we covered topics like business model, competitive analysis, technical requirements, but still the most important part was to put these great ideas into a cohesive process.

The goal of the app was to make the gift-giving process simpler, faster, and less awkward. We wanted to give users the ability to connect with friends and buy a gift from their wish list. We wanted to take existing lists on well-known sites like Amazon, Walmart and Target and bring them into our app. Sounds simple, right?

First Challenge

After a quick research, we found that there is no API that allows us to download the wishlist from Amazon and make it available in our app. Of course, we could have contacted Amazon and asked them to make this information available, but we would have had to show them the value of such a solution. And at this point, we were just a group of strangers who meant nothing to such a large company.

So we decided that we would build a scraper that would pull this information from their platform for us. The wishlist would have to be public, the user would paste a link to it, and our scraper would copy the image, price, description, and link to the product and display it in our app. Of course, the purchasing process would still take place through Amazon’s platform.

Is this legal?

First of all, we wanted to check if using a scraper is legal. We read a bit about litigation taking place on this topic and most of it was won by scraper creators. We found no law prohibiting their use.

When you’re in the so-called “gray area,” it’s always a good idea to ask yourself, are my actions harming anyone? After analyzing our process, we came to the conclusion that our app will in no way harm the business of other platforms. At most, it can help increase the sales of the products displayed in our app. So we gave this project a green light.

Proof of concept

Deciding to create a scraper still didn’t solve our problem. We still had to see if it would work for the portals we had chosen. If you decide to use data from others, you need to remember that there are many factors that can prevent you from doing so.

For example, some platforms have protection against attacks and detect bots visiting their sites, so they can block them. And our scraper was just such a bot. Additionally, when you “teach” such a scraper where on the page are photos and product descriptions you have to take into account that the look of the page may change over time and the scraper will get lost.

For this reason, we decided to start our work with a POC (Proof of Concept), during which we would make sure that everything would work as we intended. We engaged two developers to work for a 2-week sprint to test it. During this time we created a scraper for Amazon and a process for inviting friends. Additionally, we got lucky and found a great library that we used to create the front-end of our app, which allowed us to get it ready for client testing. The POC was successful.

Let’s build an MVP

As always, when you get excited about your project, you may think that all the functionality you came up with is necessary from the beginning of the application. It’s important not to get lost in it. We scheduled another Product Design Workshop, created and initial mockups for the MVP version.

Not only did the client leave with ready-made ideas, but our entire team got involved in the brainstorming process. We decided not to limit ourselves to only platforms that already have ready-made wish lists.

What if a user finds his dream gift on another site? He should be able to add it to his wishlist. What if he sees a game at his friend’s house, but he doesn’t want to look for it on the Internet? He should be able to add the product to his list as a simple picture.

Based on these and some other ideas, we decided to build an internal browser in the app that would be able to pull data from all possible websites using the “add to wishlist” button. There were also a few questions like what if someone has already bought a given gift from the list or how to know what address to send the package to. For all these questions we created solutions, which you can check by downloading our application 😉 (https://beri.gifts/)

Team and organization

In this project our client became our Product Owner. He was not a technical person, but he had clear expectations about how his application should look like. We booked two Fullstack Developers – one of them preferred to work on the back-end, the other on the front-end. This allowed each of them to focus on their part of the job, but still be able to help each other with any roadblocks.

We also added a Customer Success Manager to make sure everything was running as planned, the developers understood the client’s business goals, and there were no communication issues between the development team and the client.

And finally, our Project Supervisor, who dealt extensively with organization, as well as budgets and estimates. In fact, everything that we are not fond of in projects, but without which the cooperation would not run smoothly.

We planned the work into 5 weekly sprints. Each week we had meetings to review the work we had done and plan the work for the following week (Review/Retro/Planning), and daily status meetings (Daily meetings). Our client appeared at each of these meetings, which made our work much easier.

Designs

Because we always try to provide our clients with the “full package” the lack of corporate identity and views for the application was not an issue here. The client provided the company name and logo and left the rest in our hands. Our graphic designer started by preparing 3 views inspired by the look of the pages the client sent us. Once we decided on the look of the application we moved on to designing each of the views.

Our graphic designer also focused on UX/UI, so we had to make adjustments to our process at times. Some of the views were simplified, and we added modals that we hadn’t thought of before. Everyone was happy with the final look. We also helped the client create a simple landing page to match the look of the application.

You shall not pass!

We were able to deliver the application on time. The timing was great because it was happening during a pandemic and the first lockdown. As a result, our app solved the problem of gift giving when everyone had to be in their homes.

First, our client tested the app with his friends. After a few tweaks, we were ready to officially release the product. And this is usually where our work ends. In the Google store the application appeared immediately, but Apple was not satisfied with our solution. Not only was our client confused, so were we! During our 10 years of working on apps, we never had so many problems releasing our work.

This wasn’t something we could fix with a line of code, because Apple had a problem with the very logic of our app and… the fact that we use data pulled from Amazon and other platforms.

For a moment, we were frankly terrified of what was coming next. Our client wanted to rewrite half the application to meet Apple’s requirements, but we were able to stop him. We just had to convince Apple that we were the ones who were right. 🙂 After a long exchange of words and a million explanations we succeeded! You can find our app in both stores and send your friends the gifts they’ve always dreamed of!