usteki siteadresi.com daki bölume kendi sitemizi yazıyor..
ve robot.txt oalrak kayıtediyoruz..
son oalrakta ftp mizde host sayfasına yani puplic html yada http docsun olduğu bölume atıyoruz...böğlece google sitenizi daha kolay indexlemiş oluyo..
ve robot.txt oalrak kayıtediyoruz..
son oalrakta ftp mizde host sayfasına yani puplic html yada http docsun olduğu bölume atıyoruz...böğlece google sitenizi daha kolay indexlemiş oluyo..
PHP:
## robots.txt generated by http://www.siteadresi.com robot generator# Please, we do NOT allow nonauthorized robots any longer.#User-agent: asteriasDisallow:User-agent: BackDoorBot/1.0Disallow:User-agent: Black HoleDisallow:User-agent: BlowFish/1.0Disallow:User-agent: BotALotDisallow:User-agent: BuiltBotToughDisallow:User-agent: Bullseye/1.0Disallow:User-agent: BunnySlippersDisallow:User-agent: CegbfeiehDisallow:User-agent: CheeseBotDisallow:User-agent: CherryPickerDisallow:User-agent: CherryPickerElite/1.0Disallow:User-agent: CherryPickerSE/1.0Disallow:User-agent: CopyRightCheckDisallow:User-agent: cosmosDisallow:User-agent: CrescentDisallow:User-agent: Crescent Internet ToolPak HTTP OLE Control v.1.0Disallow:User-agent: DittoSpyderDisallow:User-agent: EmailCollectorDisallow:User-agent: EmailSiphonDisallow:User-agent: EmailWolfDisallow:User-agent: EroCrawlerDisallow:User-agent: ExtractorProDisallow: User-agent: FoobotDisallow: User-agent: Googlebot Disallow: User-agent: Gezginbot Disallow: User-agent: Harvest/1.5Disallow: User-agent: hloaderDisallow: User-agent: httplibDisallow:User-agent: humanlinksDisallow: User-agent: InfoNaviRobotDisallow: User-agent: JennyBotDisallow: User-agent: Kenjin SpiderDisallow: User-agent: Keyword Density/0.9Disallow: User-agent: LexiBotDisallow: User-agent: libWeb/clsHTTPDisallow: User-agent: LinkextractorProDisallow: User-agent: LinkScan/8.1a UnixDisallow: User-agent: LinkWalkerDisallow: User-agent: LNSpiderguyDisallow: User-agent: lwp-trivialDisallow: User-agent: lwp-trivial/1.34Disallow: User-agent: Mata HariDisallow: User-agent: Microsoft URL Control - 5.01.4511Disallow: User-agent: Microsoft URL Control - 6.00.8169Disallow: User-agent: MIIxpcDisallow: User-agent: MIIxpc/4.2Disallow: User-agent: Mister PiXDisallow: User-agent: mogetDisallow: User-agent: moget/2.1Disallow: User-agent: mozilla/4Disallow: User-agent: Mozilla/4.0 (compatible; BullsEye; Windows 95)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 95)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 98)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows NT)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows XP)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows 2000)Disallow: User-agent: Mozilla/4.0 (compatible; MSIE 4.0; Windows ME)Disallow: User-agent: mozilla/5Disallow: User-agent: NetAntsDisallow: User-agent: NICErsPRODisallow: User-agent: Offline ExplorerDisallow: User-agent: OpenfindDisallow: User-agent: Openfind data gathereDisallow: User-agent: ProPowerBot/2.14Disallow: User-agent: ProWebWalkerDisallow: User-agent: QueryN ****searchDisallow: User-agent: RepoMonkeyDisallow: User-agent: RepoMonkey Bait & Tackle/v1.01Disallow: User-agent: RMADisallow: User-agent: SiteSnaggerDisallow: User-agent: SpankBotDisallow: User-agent: spannerDisallow: User-agent: suzuranDisallow: User-agent: Szukacz/1.4Disallow: User-agent: TeleportDisallow: User-agent: TeleportProDisallow: User-agent: TelesoftDisallow: User-agent: The IntraformantDisallow: User-agent: TheNomadDisallow: User-agent: TightTwatBotDisallow: User-agent: TitanDisallow: User-agent: toCrawl/UrlDispatcherDisallow: User-agent: True_RobotDisallow: User-agent: True_Robot/1.0Disallow: User-agent: turingosDisallow: User-agent: URLy WarningDisallow: User-agent: VCIDisallow: User-agent: VCI WebViewer VCI WebViewer Win32Disallow: User-agent: Web Image CollectorDisallow: User-agent: WebAutoDisallow: User-agent: WebBanditDisallow: User-agent: WebBandit/3.50Disallow: User-agent: WebCopierDisallow: User-agent: WebEnhancerDisallow: User-agent: WebmasterWorldForumBotDisallow: User-agent: WebSaugerDisallow: User-agent: Website QuesterDisallow: User-agent: Webster ProDisallow: User-agent: WebStripperDisallow: User-agent: WebZipDisallow: User-agent: WebZip/4.0Disallow: User-agent: WgetDisallow: User-agent: Wget/1.5.3Disallow: User-agent: Wget/1.6Disallow: User-agent: WWW-Collector-EDisallow: User-agent: Xenu'sDisallow: User-agent: Xenu's Link Sleuth 1.1cDisallow: User-agent: ZeusDisallow: User-agent: Zeus 32297 Webster Pro V2.9 Win32Disallow: