Scribd Logo
Загрузить

Добро пожаловать в Scribd!

  • Robots

    Загружено:

    Parvez Zaidi
    6 просмотров3 страницы
    Robolts

    Авторское право

    © © All Rights Reserved

    Доступные форматы

    TXT, PDF, TXT или читайте онлайн в Scribd

    Этот документ был вам полезен?

    Это неприемлемый материал?

    Пожаловаться на этот документ
    Robolts

    Авторское право:

    © All Rights Reserved
    Скачайте в формате TXT, PDF, TXT или читайте онлайн в Scribd
    Отметить как неприемлемый контент
    6 просмотров3 страницы

    Robots

    Загружено:

    Parvez Zaidi
    Robolts

    Авторское право:

    © All Rights Reserved
    Скачайте в формате TXT, PDF, TXT или читайте онлайн в Scribd
    Отметить как неприемлемый контент
    из 3

    # Begin robots.

    txt file
    #/-----------------------------------------------\
    #| In single portal/domain situations, uncomment the sitmap line and enter domai
    n name
    #\-----------------------------------------------/
    #Sitemap: http://www.DomainNamehere.com/sitemap.aspx
    User-agent: *
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /DesktopModules/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /images/
    Disallow: /Install/
    Disallow: /js/
    Disallow: /Portals/
    Disallow: /Providers/
    Disallow: /Resources/ContentRotator/
    Disallow: /Resources/ControlPanel/
    Disallow: /Resources/Dashboard/
    Disallow: /Resources/FeedBrowser/
    Disallow: /Resources/OpenForceAd/
    Disallow: /Resources/Search/
    Disallow: /Resources/Shared/
    Disallow: /Resources/SkinWidgets/
    Disallow: /Resources/TabStrip/
    Disallow: /Resources/Widgets/
    Disallow: /Activity-Feed/userId/ # Do not index user profiles
    User-agent: msnbot
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /DesktopModules/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /images/
    Disallow: /Install/
    Disallow: /js/
    Disallow: /Portals/
    Disallow: /Providers/
    Disallow: /Resources/ContentRotator/
    Disallow: /Resources/ControlPanel/
    Disallow: /Resources/Dashboard/
    Disallow: /Resources/FeedBrowser/
    Disallow: /Resources/OpenForceAd/
    Disallow: /Resources/Search/
    Disallow: /Resources/Shared/
    Disallow: /Resources/SkinWidgets/
    Disallow: /Resources/TabStrip/
    Disallow: /Resources/Widgets/
    Disallow: /Activity-Feed/userId/ # Do not index user profiles
    Crawl-delay: 5
    User-agent: Slurp
    Disallow: /*/ctl/ # Slurp permits *
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /DesktopModules/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /images/
    Disallow: /Install/
    Disallow: /js/
    Disallow: /Portals/
    Disallow: /Providers/
    Disallow: /Resources/ContentRotator/
    Disallow: /Resources/ControlPanel/
    Disallow: /Resources/Dashboard/
    Disallow: /Resources/FeedBrowser/
    Disallow: /Resources/OpenForceAd/
    Disallow: /Resources/Search/
    Disallow: /Resources/Shared/
    Disallow: /Resources/SkinWidgets/
    Disallow: /Resources/TabStrip/
    Disallow: /Resources/Widgets/
    Disallow: /Activity-Feed/userId/ # Do not index user profiles
    Crawl-delay: 5
    User-agent: Googlebot
    Disallow: /*/ctl/ # Googlebot permits *
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /DesktopModules/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /images/
    Disallow: /Install/
    Disallow: /js/
    Disallow: /Portals/
    Disallow: /Providers/
    Disallow: /Resources/ContentRotator/
    Disallow: /Resources/ControlPanel/
    Disallow: /Resources/Dashboard/
    Disallow: /Resources/FeedBrowser/
    Disallow: /Resources/OpenForceAd/
    Disallow: /Resources/Search/
    Disallow: /Resources/Shared/
    Disallow: /Resources/SkinWidgets/
    Disallow: /Resources/TabStrip/
    Disallow: /Resources/Widgets/
    Disallow: /Activity-Feed/userId/ # Do not index user profiles
    User-agent: Yahoo Pipes 1.0
    Disallow: /admin/
    Disallow: /App_Browsers/
    Disallow: /App_Code/
    Disallow: /App_Data/
    Disallow: /App_GlobalResources/
    Disallow: /bin/
    Disallow: /Components/
    Disallow: /Config/
    Disallow: /contest/
    Disallow: /controls/
    Disallow: /DesktopModules/
    Disallow: /Documentation/
    Disallow: /HttpModules/
    Disallow: /images/
    Disallow: /Install/
    Disallow: /js/
    Disallow: /Portals/
    Disallow: /Providers/
    Disallow: /Resources/ContentRotator/
    Disallow: /Resources/ControlPanel/
    Disallow: /Resources/Dashboard/
    Disallow: /Resources/FeedBrowser/
    Disallow: /Resources/OpenForceAd/
    Disallow: /Resources/Search/
    Disallow: /Resources/Shared/
    Disallow: /Resources/SkinWidgets/
    Disallow: /Resources/TabStrip/
    Disallow: /Resources/Widgets/
    Disallow: /Activity-Feed/userId/ # Do not index user profiles
    # End of robots.txt file

    Вам также может понравиться