Enhancements: Further Payload download capabilities / VT Api Integration / munin

[Gi7w0rm]

Hey,
Just a quick question:
Would it be possible to enable the download of other files, maybe based on extensions ?
I am currently looking at an OpenDir full of malicious .vbs and .txt files (which themselves contain powershell/vbs code)
Would be awesome to have a possibility to download them too, but as I see it changing the payload processing file is not that easy as it is based on File Magic...
So maybe there is room for enhancement here :)

Also, I was thinking that it would be really helpfull to have some kind of VirusTotal Integration to automatically check files against VT. Similar to the ClamAV module. Guess it would be important though to make it a standalone module as an API Key is requiered.
On the other hand, one could also imagine a module based on "Munin" by Florian Roth.
https://github.com/Neo23x0/munin
Its also based in python and would enable further capabilites, such as automated checking against:
Virustotal
HybridAnalysis
Any.Run
URLhaus
MISP
CAPE
Malshare
Valhalla
Hashlookup

Which probably would be a great addition for all folks dealing with malware.

Just some thoughts I had while working with subcrawler :)
Greetings
Gi7w0rm

[jstrosch]

Hi @Gi7w0rm - the saving of content has become a little fragmented. As you point out, there is a way to save content based on libmagic via the payloads module, but this is dependent upon having libmagic values to match on. With scripts, this can be a bit tricky since they often don't have a clear signature. We could also try to match on file extension, but those are usually quite inconsistent with how they are deployed. I just pushed an Elastic output module, which provides the ability to save all of the HTTP response content, so that is one way to get the content. However, I think there should be a way that is not dependent on a module, so going to create an issue to move some of this functionality into the core of the framework :)

Yes, definitely have considered modules for VT, URLhaus, etc - these would make great processing modules. There are limits of course, based on the need for an API key and API limits (for example, last I checked a free VT account limited the number of API queries). Personally, I'd like to create a module to check and then upload URLs to the URLHaus, it's on my list of things to do anyway :) Happy to help if you want to tackle creating one, there is a lot of room for more functionality!

[Gi7w0rm]

Hey @jstrosch,
Thank you for opening this discussion.
I gave the idea of a VT module a second thought after your comment and you are right about the API Quota limitations. A VT module wouldn't work for the Service Mode of subcrawl, as it is limited to 500 Queries a day if you use the Free VT API. Which subcrawl service mode would swallow in seconds. However, I still think it is a valuable addition to active use mode and I might tackle it, if I find time.

As you might have noticed, I am still struggling in getting the service mode to run.
However, if you could explain me where I would need to add a new Input module so that it gets loaded on start of the service mode, I might be able to test out my Threatfox import module now.

I am also adding the vanilla and undebugged source code here, just in case you find time to have a look at it and already see any kind of problem or can make use of it:

threatfox.py.txt

You will probably have to rename it before you try to run it.
I have to admit I have never worked with Kafka before and the code is blatantly put together from your urlhaus module.
But it should probably do the job. Maybe... Eventually... Hopefully ? XD
A description of the threatfox api can be found here:
https://threatfox.abuse.ch/api/
As far as I know, for looking up data, there is no API key needed. So I basically just pull the data for botnet_cc and payload_delivery which is the only place where URL's, Domains or IP: Port combinations can appear. I then go on adding these to the kafka queue.
I hope subcrawl knows how to deal with IP:Port combinations ?

As I stated above, code is undebugged, so no garantie it is working in the current state. But it's taking 2 weeks already to debug my weird issue with the service mode and I kinda just want to get this out and appreciate your excelent help with my subcrawl issues by doing so.
Feel free to give feedback :)

Cheers
Chris (Gi7w0rm)