base on E-mails, subdomains and names Harvester - OSINT    [](https://inventory.raw.pm/) What is this? ------------- theHarvester is a simple to use, yet powerful tool designed to be used during the reconnaissance stage of a red<br> team assessment or penetration test. It performs open source intelligence (OSINT) gathering to help determine<br> a domain's external threat landscape. The tool gathers names, emails, IPs, subdomains, and URLs by using<br> multiple public resources that include:<br> Passive modules: ---------------- * anubis: Anubis-DB - https://github.com/jonluca/anubis * bevigil: CloudSEK BeVigil scans mobile application for OSINT assets (Requires an API key, see below.) - https://bevigil.com/osint-api * baidu: Baidu search engine - www.baidu.com * binaryedge: List of known subdomains (Requires an API key, see below.) - https://www.binaryedge.io * bing: Microsoft search engine - https://www.bing.com * bingapi: Microsoft search engine, through the API (Requires an API key, see below.) * brave: Brave search engine - https://search.brave.com/ * bufferoverun: (Requires an API key, see below.) https://tls.bufferover.run * censys: [Censys search engine](https://search.censys.io/) will use certificates searches to enumerate subdomains and gather emails<br> (Requires an API key, see below.) https://censys.io * certspotter: Cert Spotter monitors Certificate Transparency logs - https://sslmate.com/certspotter/ * criminalip: Specialized Cyber Threat Intelligence (CTI) search engine (Requires an API key, see below.) - https://www.criminalip.io * crtsh: Comodo Certificate search - https://crt.sh * dnsdumpster: DNSdumpster search engine - https://dnsdumpster.com * duckduckgo: DuckDuckGo search engine - https://duckduckgo.com * fullhunt: Next-generation attack surface security platform (Requires an API key, see below.) - https://fullhunt.io * github-code: GitHub code search engine (Requires a GitHub Personal Access Token, see below.) - www.github.com * hackertarget: Online vulnerability scanners and network intelligence to help organizations - https://hackertarget.com * hunter: Hunter search engine (Requires an API key, see below.) - https://hunter.io * hunterhow: Internet search engines for security researchers (Requires an API key, see below.) - https://hunter.how * intelx: Intelx search engine (Requires an API key, see below.) - http://intelx.io * netlas: A Shodan or Censys competitor (Requires an API key, see below.) - https://app.netlas.io * onyphe: Cyber defense search engine (Requires an API key, see below.) - https://www.onyphe.io/ * otx: AlienVault open threat exchange - https://otx.alienvault.com * pentestTools: Cloud-based toolkit for offensive security testing, focused on web applications and network penetration<br> testing (Requires an API key, see below.) - https://pentest-tools.com/ * projecDiscovery: We actively collect and maintain internet-wide assets data, to enhance research and analyse changes around<br> DNS for better insights (Requires an API key, see below.) - https://chaos.projectdiscovery.io * rapiddns: DNS query tool which make querying subdomains or sites of a same IP easy! https://rapiddns.io * rocketreach: Access real-time verified personal/professional emails, phone numbers, and social media links (Requires an API key,<br> see below.) - https://rocketreach.co * securityTrails: Security Trails search engine, the world's largest repository of historical DNS data (Requires an API key, see<br> below.) - https://securitytrails.com * -s, --shodan: Shodan search engine will search for ports and banners from discovered hosts (Requires an API key, see below.)<br> https://shodan.io * sitedossier: Find available information on a site - http://www.sitedossier.com * subdomaincenter: A subdomain finder tool used to find subdomains of a given domain - https://www.subdomain.center/ * subdomainfinderc99: A subdomain finder is a tool used to find the subdomains of a given domain - https://subdomainfinder.c99.nl * threatminer: Data mining for threat intelligence - https://www.threatminer.org/ * tomba: Tomba search engine (Requires an API key, see below.) - https://tomba.io * urlscan: A sandbox for the web that is a URL and website scanner - https://urlscan.io * vhost: Bing virtual hosts search * virustotal: Domain search (Requires an API key, see below.) - https://www.virustotal.com * yahoo: Yahoo search engine * zoomeye: China's version of Shodan (Requires an API key, see below.) - https://www.zoomeye.org Active modules: --------------- * DNS brute force: dictionary brute force enumeration * Screenshots: Take screenshots of subdomains that were found Modules that require an API key: -------------------------------- Documentation to setup API keys can be found at - https://github.com/laramies/theHarvester/wiki/Installation#api-keys * bevigil - Free upto 50 queries. Pricing can be found here: https://bevigil.com/pricing/osint * binaryedge - $10/month * bing * bufferoverun - uses the free API * censys - API keys are required and can be retrieved from your [Censys account](https://search.censys.io/account/api). * criminalip * fullhunt * github * hunter - limited to 10 on the free plan, so you will need to do -l 10 switch * hunterhow * intelx * netlas - $ * onyphe -$ * pentestTools - $ * projecDiscovery - invite only for now * rocketreach - $ * securityTrails * shodan - $ * tomba - Free up to 50 search. * zoomeye Install and dependencies: ------------------------- * Python 3.11+ * https://github.com/laramies/theHarvester/wiki/Installation Comments, bugs, and requests: ----------------------------- * [](https://twitter.com/laramies) Christian Martorella @laramies [email protected] * [](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1 * [](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1 Main contributors: ------------------ * [](https://twitter.com/NotoriousRebel1) Matthew Brown @NotoriousRebel1 * [](https://twitter.com/jay_townsend1) Jay "L1ghtn1ng" Townsend @jay_townsend1 * [](https://twitter.com/discoverscripts) Lee Baird @discoverscripts Thanks: ------- * John Matherly - Shodan project * Ahmed Aboul Ela - subdomain names dictionaries (big and small) ", Assign "at most 3 tags" to the expected json: {"id":"11900","tags":[]} "only from the tags list I provide: [{"id":77,"name":"3d"},{"id":89,"name":"agent"},{"id":17,"name":"ai"},{"id":54,"name":"algorithm"},{"id":24,"name":"api"},{"id":44,"name":"authentication"},{"id":3,"name":"aws"},{"id":27,"name":"backend"},{"id":60,"name":"benchmark"},{"id":72,"name":"best-practices"},{"id":39,"name":"bitcoin"},{"id":37,"name":"blockchain"},{"id":1,"name":"blog"},{"id":45,"name":"bundler"},{"id":58,"name":"cache"},{"id":21,"name":"chat"},{"id":49,"name":"cicd"},{"id":4,"name":"cli"},{"id":64,"name":"cloud-native"},{"id":48,"name":"cms"},{"id":61,"name":"compiler"},{"id":68,"name":"containerization"},{"id":92,"name":"crm"},{"id":34,"name":"data"},{"id":47,"name":"database"},{"id":8,"name":"declarative-gui "},{"id":9,"name":"deploy-tool"},{"id":53,"name":"desktop-app"},{"id":6,"name":"dev-exp-lib"},{"id":59,"name":"dev-tool"},{"id":13,"name":"ecommerce"},{"id":26,"name":"editor"},{"id":66,"name":"emulator"},{"id":62,"name":"filesystem"},{"id":80,"name":"finance"},{"id":15,"name":"firmware"},{"id":73,"name":"for-fun"},{"id":2,"name":"framework"},{"id":11,"name":"frontend"},{"id":22,"name":"game"},{"id":81,"name":"game-engine "},{"id":23,"name":"graphql"},{"id":84,"name":"gui"},{"id":91,"name":"http"},{"id":5,"name":"http-client"},{"id":51,"name":"iac"},{"id":30,"name":"ide"},{"id":78,"name":"iot"},{"id":40,"name":"json"},{"id":83,"name":"julian"},{"id":38,"name":"k8s"},{"id":31,"name":"language"},{"id":10,"name":"learning-resource"},{"id":33,"name":"lib"},{"id":41,"name":"linter"},{"id":28,"name":"lms"},{"id":16,"name":"logging"},{"id":76,"name":"low-code"},{"id":90,"name":"message-queue"},{"id":42,"name":"mobile-app"},{"id":18,"name":"monitoring"},{"id":36,"name":"networking"},{"id":7,"name":"node-version"},{"id":55,"name":"nosql"},{"id":57,"name":"observability"},{"id":46,"name":"orm"},{"id":52,"name":"os"},{"id":14,"name":"parser"},{"id":74,"name":"react"},{"id":82,"name":"real-time"},{"id":56,"name":"robot"},{"id":65,"name":"runtime"},{"id":32,"name":"sdk"},{"id":71,"name":"search"},{"id":63,"name":"secrets"},{"id":25,"name":"security"},{"id":85,"name":"server"},{"id":86,"name":"serverless"},{"id":70,"name":"storage"},{"id":75,"name":"system-design"},{"id":79,"name":"terminal"},{"id":29,"name":"testing"},{"id":12,"name":"ui"},{"id":50,"name":"ux"},{"id":88,"name":"video"},{"id":20,"name":"web-app"},{"id":35,"name":"web-server"},{"id":43,"name":"webassembly"},{"id":69,"name":"workflow"},{"id":87,"name":"yaml"}]" returns me the "expected json"