[转载]如何构建日均千万PV Web站点(二) 之~缓存为王~ - 姜明则 - 博客园

[转载]如何构建日均千万PV Web站点(二) 之~缓存为王~ – 姜明则 – 博客园.

随着网站业务的不断发展,用户的规模越来越大;介于中国无比蹩脚复杂的网路环境;南电信;北联通;中间竟然只用一条链路进行互联通信!有研究表明,网站访 问延迟和用户流失率正相关,网站访问速度越慢,用户越容易失去耐心而离开。为了提高更好的用户体验,留住用户,网站需要加速网站访问速度。如今主要的手段 只有使用CDN和反向代理了;此时网站的架构应该是这样的:

1、使用CDN和缓存服务器;CDN和反向代理的基本原理都是缓存数据,区别就在于CDN部署在网络提供商的机房,使用户在请求网站服务时,可 以从距离自己最近的网络提供商机房获取数据;CDN的网络环境很复杂;所谓的多重负载均衡架构模型;不过它们一般会使用DNS作为全局负载均衡器;高效, 并且能够根据客户端的源IP地址,来判断客户端的来源地区;将客户端的请求分配制本地负载均衡器;CDN架构图如下:从下图中可以看到,第一层GSLB和 第二层GSLB都有各自的域组,第一层GSLB通过区域设置,将整个服务池分为电信的服务池和联通的服务池,第二层GSLB通过区域设置,将电信的服务池 分为各省的服务池。这里的服务池就是提供相同业务的所有POP节点的组合,各省的服务池包含两个POP节点,POP节点也是GSLB在调度配置中所认识到 的虚拟服务器.GSLB通过负载均很策略最终返回一个POP节点地址,用户直接访问POP节点来获取网站缓存内容。

 

而反向代理则部署在网站的中心机房中,当用户请求到达中心机房中后,首先访问的服务器就是反向代理服务器,如果反向代理服务器ui中缓存着用户请求 的资源,就将其直接返回给用户。使用CDN和反向代理的目的都是尽早返回数据给用户,一方面加快用户的访问速度,另一方面也减轻了后端服务器的负载压力;

2、分布式数据库;分布式数据库是网站数据拆分的最后手段,只有在表单数据规模非常庞大的时候才使用;

3、服务器推送;将应用程序服务器;以及缓存服务器全部推送到运营商机房中;

4、NoSQL以及搜索引擎的引入;随着网站的业务越来越复杂,对数据的存储和检索需求也越来越复杂,这时网站就必须得引入一些非关系型数据库技术如NoSQL(MongoDB, 对于大数据量、髙并发、弱事务的互联网应用, MongoDB则是一个如瑞士军刀般的利器。尽管我不认同MongoDB会在所有场合完全取代MySQL,但我相信它完全可以满足Web 2.0和移动互联网应用的数据存储需求。MongoDB内置的水平扩展机制提供了从百万到十亿级别的数据量处理能力,其开箱即用的特性也大大降低了中小网 站的运维成本),以及非数据库查询技术如搜索引擎;NoSQL和搜索引擎对可伸缩的分布式特性具有更好的支持。因此,此时的架构模型就如下图所示:

Nginx 反向代理配置:

<span class="pln">user                              nobody nobody</span><span class="pun">;</span><span class="pln">
worker_processes                  </span><span class="lit">4</span><span class="pun">;</span><span class="pln">
worker_rlimit_nofile              </span><span class="lit">51200</span><span class="pun">;</span><span class="pln">
 
error_log                         logs</span><span class="pun">/</span><span class="pln">error</span><span class="pun">.</span><span class="pln">log  notice</span><span class="pun">;</span><span class="pln">
 
pid                               </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">run</span><span class="pun">/</span><span class="pln">nginx</span><span class="pun">.</span><span class="pln">pid</span><span class="pun">;</span><span class="pln">
 
events </span><span class="pun">{</span><span class="pln">
  </span><span class="kwd">use</span><span class="pln">                             epoll</span><span class="pun">;</span><span class="pln">
  worker_connections              </span><span class="lit">51200</span><span class="pun">;</span><span class="pln">
</span><span class="pun">}</span><span class="pln">
 
http </span><span class="pun">{</span><span class="pln">
  server_tokens                   off</span><span class="pun">;</span><span class="pln">
  include                         mime</span><span class="pun">.</span><span class="pln">types</span><span class="pun">;</span><span class="pln">
 
  proxy_redirect                off</span><span class="pun">;</span><span class="pln">
  proxy_set_header              </span><span class="typ">Host</span><span class="pln"> $host</span><span class="pun">;</span><span class="pln">
  proxy_set_header              X</span><span class="pun">-</span><span class="typ">Real</span><span class="pun">-</span><span class="pln">IP $remote_addr</span><span class="pun">;</span><span class="pln">
  proxy_set_header              X</span><span class="pun">-</span><span class="typ">Forwarded</span><span class="pun">-</span><span class="typ">For</span><span class="pln"> $proxy_add_x_forwarded_for</span><span class="pun">;</span><span class="pln">
  client_max_body_size          </span><span class="lit">20m</span><span class="pun">;</span><span class="pln">
  client_body_buffer_size       </span><span class="lit">256k</span><span class="pun">;</span><span class="pln">
  proxy_connect_timeout         </span><span class="lit">90</span><span class="pun">;</span><span class="pln">
  proxy_send_timeout            </span><span class="lit">90</span><span class="pun">;</span><span class="pln">
  proxy_read_timeout            </span><span class="lit">90</span><span class="pun">;</span><span class="pln">
  proxy_buffer_size             </span><span class="lit">128k</span><span class="pun">;</span><span class="pln">
  proxy_buffers                 </span><span class="lit">4</span><span class="pln"> </span><span class="lit">64k</span><span class="pun">;</span><span class="pln">
  proxy_busy_buffers_size       </span><span class="lit">128k</span><span class="pun">;</span><span class="pln">
  proxy_temp_file_write_size    </span><span class="lit">128k</span><span class="pun">;</span><span class="pln">
 
  default_type                    application</span><span class="pun">/</span><span class="pln">octet</span><span class="pun">-</span><span class="pln">stream</span><span class="pun">;</span><span class="pln">
  charset                         utf</span><span class="pun">-</span><span class="lit">8</span><span class="pun">;</span><span class="pln">
   
  client_body_temp_path           </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">tmp</span><span class="pun">/</span><span class="pln">client_body_temp </span><span class="lit">1</span><span class="pln"> </span><span class="lit">2</span><span class="pun">;</span><span class="pln">
  proxy_temp_path                 </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">tmp</span><span class="pun">/</span><span class="pln">proxy_temp </span><span class="lit">1</span><span class="pln"> </span><span class="lit">2</span><span class="pun">;</span><span class="pln">
  fastcgi_temp_path               </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">tmp</span><span class="pun">/</span><span class="pln">fastcgi_temp </span><span class="lit">1</span><span class="pln"> </span><span class="lit">2</span><span class="pun">;</span><span class="pln">
  uwsgi_temp_path                 </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">tmp</span><span class="pun">/</span><span class="pln">uwsgi_temp </span><span class="lit">1</span><span class="pln"> </span><span class="lit">2</span><span class="pun">;</span><span class="pln">
  scgi_temp_path                  </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">tmp</span><span class="pun">/</span><span class="pln">scgi_temp </span><span class="lit">1</span><span class="pln"> </span><span class="lit">2</span><span class="pun">;</span><span class="pln">
 
  ignore_invalid_headers          on</span><span class="pun">;</span><span class="pln">
  server_names_hash_max_size      </span><span class="lit">256</span><span class="pun">;</span><span class="pln">
  server_names_hash_bucket_size   </span><span class="lit">64</span><span class="pun">;</span><span class="pln">
  client_header_buffer_size       </span><span class="lit">8k</span><span class="pun">;</span><span class="pln">
  large_client_header_buffers     </span><span class="lit">4</span><span class="pln"> </span><span class="lit">32k</span><span class="pun">;</span><span class="pln">
  connection_pool_size            </span><span class="lit">256</span><span class="pun">;</span><span class="pln">
  request_pool_size               </span><span class="lit">64k</span><span class="pun">;</span><span class="pln">
 
  output_buffers                  </span><span class="lit">2</span><span class="pln"> </span><span class="lit">128k</span><span class="pun">;</span><span class="pln">
  postpone_output                 </span><span class="lit">1460</span><span class="pun">;</span><span class="pln">
 
  client_header_timeout           </span><span class="lit">1m</span><span class="pun">;</span><span class="pln">
  client_body_timeout             </span><span class="lit">3m</span><span class="pun">;</span><span class="pln">
  send_timeout                    </span><span class="lit">3m</span><span class="pun">;</span><span class="pln">
 
 
  log_format main                 </span><span class="str">'$server_addr $remote_addr [$time_local] $msec+$connection '</span><span class="pln">
                                  </span><span class="str">'"$request" $status $connection $request_time $body_bytes_sent "$http_referer" '</span><span class="pln">
                                  </span><span class="str">'"$http_user_agent" "$http_x_forwarded_for"'</span><span class="pun">;</span><span class="pln">
 
  open_log_file_cache               max</span><span class="pun">=</span><span class="lit">1000</span><span class="pln"> inactive</span><span class="pun">=</span><span class="lit">20s</span><span class="pln"> min_uses</span><span class="pun">=</span><span class="lit">1</span><span class="pln"> valid</span><span class="pun">=</span><span class="lit">1m</span><span class="pun">;</span><span class="pln">
 
  access_log                      logs</span><span class="pun">/</span><span class="pln">access</span><span class="pun">.</span><span class="pln">log      main</span><span class="pun">;</span><span class="pln">
  log_not_found                   on</span><span class="pun">;</span><span class="pln">
 
 
  sendfile                        on</span><span class="pun">;</span><span class="pln">
  tcp_nodelay                     on</span><span class="pun">;</span><span class="pln">
  tcp_nopush                      off</span><span class="pun">;</span><span class="pln">
 
  reset_timedout_connection       on</span><span class="pun">;</span><span class="pln">
  keepalive_timeout               </span><span class="lit">10</span><span class="pln"> </span><span class="lit">5</span><span class="pun">;</span><span class="pln">
  keepalive_requests              </span><span class="lit">100</span><span class="pun">;</span><span class="pln">
 
 
  gzip                            on</span><span class="pun">;</span><span class="pln">
  gzip_http_version               </span><span class="lit">1.1</span><span class="pun">;</span><span class="pln">
  gzip_vary                       on</span><span class="pun">;</span><span class="pln">
  gzip_proxied                    any</span><span class="pun">;</span><span class="pln">
  gzip_min_length                 </span><span class="lit">1024</span><span class="pun">;</span><span class="pln">
  gzip_comp_level                 </span><span class="lit">6</span><span class="pun">;</span><span class="pln">
  gzip_buffers                    </span><span class="lit">16</span><span class="pln"> </span><span class="lit">8k</span><span class="pun">;</span><span class="pln">
  gzip_proxied                    expired </span><span class="kwd">no</span><span class="pun">-</span><span class="pln">cache </span><span class="kwd">no</span><span class="pun">-</span><span class="pln">store </span><span class="kwd">private</span><span class="pln"> auth no_last_modified no_etag</span><span class="pun">;</span><span class="pln">
  gzip_types                      text</span><span class="pun">/</span><span class="pln">plain application</span><span class="pun">/</span><span class="pln">x</span><span class="pun">-</span><span class="pln">javascript text</span><span class="pun">/</span><span class="pln">css application</span><span class="pun">/</span><span class="pln">xml application</span><span class="pun">/</span><span class="pln">json</span><span class="pun">;</span><span class="pln">
  gzip_disable                    </span><span class="str">"MSIE [1-6]\.(?!.*SV1)"</span><span class="pun">;</span><span class="pln">
 
 
  upstream varnish  </span><span class="lit">81</span><span class="pln"> </span><span class="pun">{</span><span class="pln">
    least_conn
  
    server                        </span><span class="lit">172.16</span><span class="pun">.</span><span class="lit">100.103</span><span class="pun">:</span><span class="lit">81</span><span class="pln"> weight</span><span class="pun">=</span><span class="lit">1</span><span class="pln"> max_fails</span><span class="pun">=</span><span class="lit">2</span><span class="pun">;</span><span class="pln">
    server                        </span><span class="lit">172.16</span><span class="pun">.</span><span class="lit">100.104</span><span class="pun">:</span><span class="lit">81</span><span class="pln"> weight</span><span class="pun">=</span><span class="lit">1</span><span class="pln"> max_fails</span><span class="pun">=</span><span class="lit">2</span><span class="pun">;</span><span class="pln">
    server                        </span><span class="lit">172.16</span><span class="pun">.</span><span class="lit">100.105</span><span class="pun">:</span><span class="lit">81</span><span class="pln"> weight</span><span class="pun">=</span><span class="lit">1</span><span class="pln"> max_fails</span><span class="pun">=</span><span class="lit">2</span><span class="pun">;</span><span class="pln">
  
  </span><span class="pun">}</span><span class="pln">
 
  server </span><span class="pun">{</span><span class="pln">
    listen                        </span><span class="lit">80</span><span class="pun">;</span><span class="pln">
    server_name                   www</span><span class="pun">.</span><span class="pln">firefox</span><span class="pun">.</span><span class="pln">com</span><span class="pun">;</span><span class="pln">
    </span><span class="com"># config_apps_begin</span><span class="pln">
    root                          </span><span class="pun">/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">webapps</span><span class="pun">/</span><span class="pln">htdocs</span><span class="pun">;</span><span class="pln">
    access_log                    </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">logs</span><span class="pun">/</span><span class="pln">webapp</span><span class="pun">.</span><span class="pln">access</span><span class="pun">.</span><span class="pln">log     main</span><span class="pun">;</span><span class="pln">
    error_log                     </span><span class="pun">/</span><span class="kwd">var</span><span class="pun">/</span><span class="pln">logs</span><span class="pun">/</span><span class="pln">webapp</span><span class="pun">.</span><span class="pln">error</span><span class="pun">.</span><span class="pln">log      notice</span><span class="pun">;</span><span class="pln">
 
    location </span><span class="pun">/</span><span class="pln"> </span><span class="pun">{</span><span class="pln">
     
      location </span><span class="pun">~*</span><span class="pln"> </span><span class="pun">^.*/</span><span class="pln">favicon</span><span class="pun">.</span><span class="pln">ico$ </span><span class="pun">{</span><span class="pln">
        root                      </span><span class="pun">/</span><span class="pln">data</span><span class="pun">/</span><span class="pln">webapps</span><span class="pun">;</span><span class="pln">
        expires                   </span><span class="lit">180d</span><span class="pun">;</span><span class="pln">
        </span><span class="kwd">break</span><span class="pun">;</span><span class="pln">
      </span><span class="pun">}</span><span class="pln">
     
      </span><span class="kwd">if</span><span class="pln"> </span><span class="pun">(</span><span class="pln"> </span><span class="pun">!-</span><span class="pln">f $request_filename </span><span class="pun">)</span><span class="pln"> </span><span class="pun">{</span><span class="pln">
        proxy_pass                http</span><span class="pun">:</span><span class="com">//tomcat8080;</span><span class="pln">
        </span><span class="kwd">break</span><span class="pun">;</span><span class="pln">
      </span><span class="pun">}</span><span class="pln">
    </span><span class="pun">}</span><span class="pln">
 
    error_page                    </span><span class="lit">500</span><span class="pln"> </span><span class="lit">502</span><span class="pln"> </span><span class="lit">503</span><span class="pln"> </span><span class="lit">504</span><span class="pln">  </span><span class="pun">/</span><span class="lit">50x</span><span class="pun">.</span><span class="pln">html</span><span class="pun">;</span><span class="pln">
      location </span><span class="pun">=</span><span class="pln"> </span><span class="str">/50x.html {
      root                        html;
    }
  }
 
  server {
    listen                        8088;
    server_name                   nginx_status;
 
      location /</span><span class="pln"> </span><span class="pun">{</span><span class="pln">
          access_log                  off</span><span class="pun">;</span><span class="pln">
          deny                        all</span><span class="pun">;</span><span class="pln">
          </span><span class="kwd">return</span><span class="pln">                      </span><span class="lit">503</span><span class="pun">;</span><span class="pln">
      </span><span class="pun">}</span><span class="pln">
 
      location </span><span class="pun">/</span><span class="pln">status </span><span class="pun">{</span><span class="pln">
          stub_status                 on</span><span class="pun">;</span><span class="pln">
          access_log                  off</span><span class="pun">;</span><span class="pln">
          allow                       </span><span class="lit">127.0</span><span class="pun">.</span><span class="lit">0.1</span><span class="pun">;</span><span class="pln">
          allow                       </span><span class="lit">172.16</span><span class="pun">.</span><span class="lit">100.71</span><span class="pun">;</span><span class="pln">
          deny                        all</span><span class="pun">;</span><span class="pln">
      </span><span class="pun">}</span><span class="pln">
  </span><span class="pun">}</span><span class="pln">
 
</span><span class="pun">}</span>
赞(0) 打赏
分享到: 更多 (0)

觉得文章有用就打赏一下文章作者

支付宝扫一扫打赏

微信扫一扫打赏

登录

注册