📣 Upcoming Change: Moving Repositories & Repository Cleanup

[funilrys]

Hello everyone,

We hope this message finds you well. We wanted to share some important updates about changes happening to our project that will take place on Wednesday, December 18th, 2024.

We will be moving the following repositories to the new @Phishing-Database GitHub organization:

• mitchellkrogza/Phishing.Database
• mitchellkrogza/phishing

---

Why are we making this change?

We’re moving to the @Phishing-Database organization to give the project more focus and direction, and to easily invite external trusted contributors to participate. This move will allow us to:

• Better organize our development efforts and prioritize important areas of the project.
• Streamline collaboration by providing a more structured and community-friendly environment.
• Open the doors for more external contributors and trusted collaborators to join and contribute, enhancing our collective work.

---

Addressing GitHub's Feedback

We've also received feedback from the GitHub Support Team about inefficiencies caused by accumulated data in our repository and any of its forks. They highlighted that our repository’s current setup causes inefficiencies on GitHub's infrastructure and version control operations, particularly:

• Frequent overwriting of non-textual data leading to significant storage growth.
• Non-text files putting a strain on GitHub's version control operations.

Once we complete our repository cleanup and restructuring, the GitHub Support Team will:

• Inform fork owners about an incoming fork detachment that will be carried out by GitHub to reduce the strain on their infrastructure.

We’re committed to addressing these issues by aligning our setup with GitHub's infrastructure best practices, with support from the GitHub's Support Team if needed.

---

What changes will be made?

1. Repository Migration to @Phishing-Database
   We’ll consolidate everything under @Phishing-Database to better organize collaboration, reduce maintenance overhead, and facilitate more contributions from trusted external participants.

2. Removing non-textual files
   To optimize performance on GitHub’s infrastructure, we’ll remove the following files in the coming days or hours:

   Large Datasets

   • ALL-phishing-domains.tar.gz
   • ALL-phishing-links.tar.gz
   • input-source/ALL-phishing-domains.zip
   • input-source/ALL-phishing-links.zip
   • input-source/ALL-feeds.zip
   • input-source/ALL-feeds-URLS.tar.gz

   Assets Folder

   • assets/Tines-Sponsorship-Badge-Purple.png
   • assets/blue.jpg
   • assets/green.jpg
   • assets/kofi5.png
   • assets/limacharlie-logo-sponsorship-logo.png
   • assets/phishing-logo.jpg
   • assets/red.jpg
   • assets/spacer.jpg
   • assets/twitter-35.png
   • assets/twitter-512.png

   These files were contributing to the unnecessary growth and load on GitHub’s infrastructure and will be hosted elsewhere by us or moved to dedicated repositories when possible.

3. Repository Cleanup

   We’ll review, reorganize, and clean up the repository to remove any unnecessary files, folders, or outdated content. Our goal is to reduce or completely minimize all files in the repository.

---

🔗 Download the Full List of Data

Soon, the full phishing datasets will no longer be hosted in the repository as a single large binary file. If you need to access the complete phishing lists and datasets, please use the following links.

File Name	Official Source
ALL-phishing-domains.lst	Download
ALL-phishing-links.lst	Download
ALL-phishing-domains.tar.gz	Download
ALL-phishing-links.tar.gz	Download
phishing-domains-ACTIVE.txt	Download
phishing-domains-INACTIVE.txt	Download
phishing-domains-INVALID.txt	Download
phishing-IPs-ACTIVE.txt	Download
phishing-IPs-INACTIVE.txt	Download
phishing-IPs-INVALID.txt	Download
phishing-links-ACTIVE.txt	Download
phishing-links-INACTIVE.txt	Download
phishing-links-INVALID.txt	Download

---

How does this affect you?

• 🧑‍💻 For Active Contributors:

  You’ll be invited to join the @Phishing-Database organization. This will provide a more structured environment for collaboration and enable smoother contributions.

• 👥 For Non-Active Contributors & General Users:

  You’ll still have access to the repositories and can contribute via Pull Requests as before. The project remains open and community-driven.

• 🔄 Regarding HTTP Redirects:

  If you’re fetching files without following redirects, you’ll need to update your URLs after the repository move. We’ll try to automate many of these changes through Pull Requests, but any help would be appreciated.

• 📂 Repository Structure Update:

  You may need to update your scripts or tools to accommodate the new repository structure.

---

Contributions Are Always Welcome!

Got an idea, bug fix, or improvement? Send us your Pull Requests – we’re excited to grow and build the project together with the community!

---

Thank you for your continued support and collaboration. Let’s keep building this project together! 🚀

— @mitchellkrogza & @funilrys

[spirillen]

What will happen to https://raw.githubusercontent.com/mitchellkrogza/Phishing.Database/master/phishing-IPs-ACTIVE.txt and equivalent IP based lists? I can only see the ALL-phishing-domains.lst & ALL-phishing-links.lst are addressed in this msg?

[spirillen]

Hey @mitchellkrogza and @funilrys are you up as late as I am, or are you up early 😄

Thanks for the update, The lists was not for me, but for other that might come to this question. Hoping it will help others.

[funilrys]

Bothways 😄

[T145]

@mitchellkrogza Please consider security researchers. Invalid or inactive domains may not be useful to users who just want a blocklist, but they are relevant to security research. If they're generated as a byproduct with PyFunceble anyway, why not include them? As a side note, I've personally not seen much of a difference between invalid or inactive PyFunceble output (though invalid has been empty every time I check it) so invalid generation could be ignored entirely.

[spirillen]

Good point @T145 had some of the same thought, but not sure how mane might be using the list of INACTIVE IP addresses, but they would for sure help those who wants to delete outdated records vs constantly building new lists on same data.

[funilrys]

@T145, that's a good point - thanks for raising it.

Let me rephrase: We will always consider the needs of security researchers and the community and will continue to publish these files as we see no reason to stop including them.

That said, we may have to coordinate with the GitHub Support Team regarding the size of the these files. If they are deemed "too big", we will split them into multiple smaller files (see @Ultimate-Hosts-Blacklist) and publish the "merged" version at phish.co.za.

Our goal is to find the right balance between reducing strain on GitHub's infrastructure and maintaining usability for both the community & security researchers.

cc: @mitchellkrogza

[benolot]

Thanks for organising this so far both @funilrys and @mitchellkrogza! I have a couple of questions;

• Is there any plan to publish checksums or some other kind of mark of authenticity for the external download page on the GitHub?
• Could you not use GitHub releases instead? Without then having to rely on a "third party" domain
• With new contributors coming on board, what are you expecting of those new contributors or encouraging them to do?

Otherwise this seems like a great move for the project!

[spirillen]

Is there any plan to publish checksums

Nice thought, now they are leaving GH, a sha256 or md5 checksum files would be good to verify integrity of files.

[funilrys]

Hey @benolot,

Sorry for the waiting time.

Thank you for taking the time! Your suggestion about checksums directly influenced the decision to publish them. They are now available here: https://github.com/phishing-Database/checksums.

To address your other questions - that future contributors may have:

1. Why not use GitHub Releases?
   While GitHub Releases is a solid choice for many projects, our datasets are large and updated regularly. I'm not sure if this setup could still strain GitHub’s infrastructure - as highlighted by the GitHub Support Team. That said, we’ll coordinate with the GitHub Support Team before attempting to publish GitHub Releases - if possible.

2. Contributions are made in Free Time – Let’s respect that!!
   We all contribute to this project in our free time, and everyone’s commitment is deeply appreciated. Whether you’re an active contributor, someone who pops in occasionally, or just starting out, your time and effort matter.

3. Help us improve the flow of information
   We encourage all contributors to help us maintain a smooth flow of incoming information, such as:

• Reporting issues and potential bugs
• Submitting phishing attempts that you’ve discovered or were targeted with
• Sharing insights and improvements to help strengthen the project

4. Help us refining detection or tackling false positives
   We also face challenges with false positives, particularly from top-level or clean domains. While many of these are handled by https://github.com/Ultimate-Hosts-Blacklist/whitelist, which forms the base of our whitelisting efforts, there’s still much work to be done.

Thank you again for taking the time - and contribution 😉 ! 🚀

P.S.: The "third-party" domain actually exists since we started this project back in 2018. In the past, it was primarily used to publish the last-hour and today files. We are now just extending its usage.

[benolot]

Thanks for the reply, and no worries about the time. As you said, it's a free time thing 😊 That addresses all my concerns - looking forward to the future of the project 🎉