yahoo-auctions-watchdog

It automatically and regularly scrapes predefined yahoo auctions listing pages, and sends notifications to a configured Telegram bot when new listings are found, with basic data.
Whenever a new listing appears on a search page, you will be notified, on mobile and on desktop.

Note for users located in EU/UK

Yahoo auctions is not available anymore to users located in europe or in the UK. You need to run this program on a server which is located in another continent (America, Asia, etc), or use a VPN/proxy.

Quick start

• Make sure Golang is installed on your machine.
• Clone the repository
• Set up the yahoo auctions search urls in the config.toml file (more details below)
• Set up your Telegram credentials in the .env file (more details below)
• Run make build in order to build the executable.
• Run make run to launch the program.

Config.toml

Search URLs

To add a new url to scrape, add the following lines into the config.toml file:

[[searches]]
url = "copied URL"

Add as much as you want:

[[searches]]
url = "https://auctions.yahoo.co.jp/search/search?auccat=&tab_ex=commerce&ei=utf-8&aq=-1&oq=&sc_i=&exflg=1&p=%E3%83%9D%E3%82%B1%E3%83%A2%E3%83%B3%E3%82%AB%E3%83%BC%E3%83%89&x=0&y=0"

[[searches]]
url = "https://auctions.yahoo.co.jp/search/search?auccat=&tab_ex=commerce&ei=utf-8&aq=-1&oq=&sc_i=&exflg=1&p=%E3%83%94%E3%82%AB%E3%83%81%E3%83%A5%E3%82%A6&x=0&y=0"

Other parameters:

• delay: period, in seconds, between two scraping loops. Keep it reasonably high.
• changerate: rate that will be applied to the original prices in yen, so they will be displayed in a currency that you are familiar. If you are fine with the yen currency, you can set the changerate at zero and only display the prices in yen in the Telegram messages template in config.toml.
• downloadphotos:
  • if set to true, then the photo of each listing will be downloaded, and the messages to Telegram will be sent as "photo with caption" instead of raw text. This is useful if Telegram doesn't display the URLs previews (for example if you live in EU/UK).
  • if set to false, then the messages will be sent to Telegram as raw text.

Important notes:

You need to use search urls with the 新着順 (New) sort filter enabled, e.g. urls which contain s1=new.
That way, new listings will be detected at the top of the results list.

Telegram and .env

• First, you need to create a Telegram account.
• Then, for the following steps, you need to download and use the desktop version.

You can create a new Telegram bot via this link.

• Send the /newbot command to BotFather, and follow the steps to create a new bot. Once the bot is created, you will receive a token.
• Set the TELEGRAM_TOKEN variable in the .env file with your token.

TELEGRAM_TOKEN="1222533313:AAFwNd_HsPtpxBy35vEaZoFzUUB74v5mBpW"

Then, you need to find your chat ID.

• Paste the following link in your browser. Replace <Telegram-token> with the Telegram token.

https://api.telegram.org/bot<Telegram-token>/getUpdates?offset=0

• Send a message to your bot in the Telegram application. The message text can be anything. Your chat history must include at least one message to get your chat ID.
• Refresh your browser.
• Identify the numerical chat ID by finding the id inside the chat JSON object. In the example below, the chat ID is 123456789.

{  
   "ok":true,
   "result":[  
      {  
         "update_id":987654321,
         "message":{  
            "message_id":2,
            "from":{  
               "id":123456789,
               "first_name":"Mushroom",
               "last_name":"Kap"
            },
            "chat":{  
               "id":123456789,
               "first_name":"Mushroom",
               "last_name":"Kap",
               "type":"private"
            },
            "date":1487183963,
            "text":"hi"
         }
      }
   ]
}

• Set the TELEGRAM_CHAT_ID variable in the .env file with this value.

TELEGRAM_CHAT_ID=123456789