What is Content Discovery?

Its not the obvious things available in a web application, it's the process of gathering information on things that are not for public access.

Eg: Staff usage portal, older version/development version of the website, backup files, etc. Three ways of discovering content of a website,

• Manually
• Automated
• OSINT (Open-Source Intelligence)

Manual Discovery

• robots.txt - it shows the search engines which pages are allowed, and which aren’t allowed to show.

• Favicon - sometimes if the favicon is not referenced properly while deploying we could tell on what framework was used to build based on the default favicon.

If we use curl and get the hash of the default favicon used, we can compare it to this database and find the framework.

 curl <https://static-labs.tryhackme.cloud/sites/favicon/images/favicon.ico> | md5sum

PS C:\\> curl <https://static-labs.tryhackme.cloud/sites/favicon/images/favicon.ico> -UseBasicParsing -o favicon.ico

PS C:\\> Get-FileHash .\\favicon.ico -Algorithm MD5

• Sitemap.xml - It gives a list of all files the website owner wishes to show the public, sometimes the older endpoints that aren’t used in the current web application might be available.