Knah-Tsaeb/MyShaarli
Knah-Tsaeb/MyShaarli
1
0
Fork 0
Code Issues Pull requests Releases Wiki Activity

doc: server config: basic usage of robots.txt/HTML robots meta-tag/crawler control mechanisms

Browse source
This commit is contained in:
nodiscc 2018-08-30 20:03:00 +02:00
parent 2b4f391559
commit 6c44d604a1
1 changed files with 7 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

7
doc/md/Server-configuration.md
View file

@ -397,6 +397,7 @@ http {
``` ```
## Proxies ## Proxies
If Shaarli is served behind a proxy (i.e. there is a proxy server between clients and the web server hosting Shaarli), please refer to the proxy server documentation for proper configuration. In particular, you have to ensure that the following server variables are properly set: If Shaarli is served behind a proxy (i.e. there is a proxy server between clients and the web server hosting Shaarli), please refer to the proxy server documentation for proper configuration. In particular, you have to ensure that the following server variables are properly set:
- `X-Forwarded-Proto` - `X-Forwarded-Proto`
@ -405,6 +406,12 @@ If Shaarli is served behind a proxy (i.e. there is a proxy server between client
See also [proxy-related](https://github.com/shaarli/Shaarli/issues?utf8=%E2%9C%93&q=label%3Aproxy+) issues. See also [proxy-related](https://github.com/shaarli/Shaarli/issues?utf8=%E2%9C%93&q=label%3Aproxy+) issues.
## Robots and crawlers
Shaarli disallows indexing and crawling of your local documentation pages by search engines, using `<meta name="robots">` HTML tags.
Your Shaarli instance and other pages you host may still be indexed by various robots on the public Internet.
You may want to setup a robots.txt file or other crawler control mechanism on your server.
See [[1]](https://en.wikipedia.org/wiki/Robots_exclusion_standard), [[2]](https://support.google.com/webmasters/answer/6062608?hl=en) and [[3]](https://developers.google.com/search/reference/robots_meta_tag)
## See also ## See also